These docs are for Cribl Stream 4.9 and are no longer actively maintained.
See the latest version (4.13).
Configure System Proxy
This document outlines how to configure Cribl Stream to use a proxy server for outbound HTTP/S requests.
There are two ways to define proxy variables in Cribl Stream:
- Using environment variables: This method is independent of the systemd service manager and can be used regardless of how Cribl Stream is installed or started.
- Using systemd override files: This method is specific to systemd and requires that Cribl Stream is started as a systemd service. Choose the method that best suits your deployment environment and preferences.
Prevent Proxy, Config, and Case Conflicts
Before you deploy Cribl Stream and configure proxies, Cribl recommends running the printenv
shell command to check for any existing proxies in your environment. If you find existing proxies, be sure to include their environment variables in the systemd override file.
If Cribl Stream is already running on any Nodes you are proxying, restart the Cribl server after you initially configure – or make any changes to – the variables outlined on this page.
The environment variables’ names can be either uppercase or lowercase. However, if you set duplicate versions of the same name, the lowercase version takes precedence. For example, if you’ve set both HTTPS_PROXY
and https_proxy
, the IP address specified in https_proxy
will take effect.
Set Up Proxy Environment Variables (Using Environment Variables)
You can direct all outbound HTTP/S requests to go through proxy servers. You do so by setting a few environment variables before starting Cribl Stream, as follows:
Configure the HTTP_PROXY
and HTTPS_PROXY
environment variables, either with your proxy’s IP address, or with an FQDN that resolves to that IP address.
Optionally, follow either convention with a colon and the port number to which you want to send queries. Some HTTP_PROXY
examples:
$ export HTTP_PROXY=http://10.15.20.25:1234
$ export HTTP_PROXY=http://proxy.example.com:1234
HTTPS_PROXY
examples:
$ export HTTPS_PROXY=http://10.15.20.25:5678
$ export HTTPS_PROXY=http://proxy.example.com:5678
In the above examples, note that an HTTPS_PROXY
environment variable’s referenced URL should generally be in http://
format.
To set up Leader/Worker communication, use the CRIBL_DIST_WORKER_PROXY environment variable.
HTTP and/or HTTPS?
Several Cribl Stream endpoints rely on the HTTPS protocol. These include the Cribl telemetry endpoint (which must be accessed under some license types and all Cribl.Cloud plans), and the CDN used to propagate application updates and certain documentation features (API Reference and docs PDFs).
You might configure certain other Cribl Stream features (such as REST API Collectors) that require access to HTTP endpoints. For maximum flexibility, consider setting environment variables to handle both the HTTPS and HTTP protocols.
Set Up Proxy Configuration with systemd
If you are proxying outbound traffic and starting Cribl Stream using systemd, add your proxy environment variables to the systemd override file (see Persisting Overrides). Add statements of this form:
[Service]
Environment=http_proxy=<yourproxy>
Environment=https_proxy=<yourproxy>
Environment=no_proxy=<no_proxy_list>
This will prevent Cribl Stream from throwing “failed to send anonymized telemetry metadata” errors.
Using a Literal %
Character
To use a literal %
character, make sure to escape it by using a double %%
.
For example, suppose that you want to persist the environment variable CRIBL_DIST_WORKER_PROXY
in a systemd unit file, and the value includes a password that contains the special character @
: special@password
. You must encode the @
as %40
. However, because %
is a special character that is used for specifiers in systemd unit files, you must also escape the %
itself by doubling it.
In this example, %%40
represents the literal @
character in the password:
Environment=CRIBL_DIST_WORKER_PROXY=socks5://proxyuser:special%%40password@mydomain.com:1080
Authenticating on Proxies
You can use HTTP Basic authentication on HTTP or HTTPS proxies. Specify the username and password in the proxy URL. For example:
$ export HTTP_PROXY=http://username:password@proxy.example.com:1234
$ export HTTPS_PROXY=http://username:password@proxy.example.com:5678
If your
username
orpassword
contains special characters, Cribl Stream will try to use these fields as the proxy address. As a workaround, URL‑encode these fields.
Bypassing Proxies with NO_PROXY
If you’ve set the above environment variables, you can negate them for specified (or all) hosts. Set the NO_PROXY
environment variable to identify URLs that should bypass the proxy server, to instead be sent as direct requests. Use the following format:
$ export NO_PROXY="<list of hosts/domains>"
Cribl recommends including the Leader Node’s host name in the NO_PROXY
list.
Client Configuration and Leader Reference
The way a client references the Leader Node depends on its configuration, which originates from the Distributed Settings for that particular Node. These settings reside in the $CRIBL_HOME/local/_system/instance.yml
file.
When deploying a configuration update, the Leader Node informs all connected clients (Workers and Edge Nodes) within its Group or Fleet that a new configuration bundle is available. The Leader will reference itself using the same method (IP address or FQDN) that clients already use to connect to it in the distributed environment. This ensures a seamless download process for the clients, regardless of their initial connection method.
If a Worker is configured to use the Leader’s IP for communication, the URL will also contain that specific IP. Consequently, the NO_PROXY
variable on the client must match the IP address used in the client configuration and Leader URL for optimal operation.
Cribl v.4.6.0 introduces direct S3 downloads for configuration bundles, bypassing the Leader Node and scaling deployments with many Worker Nodes. However, if your S3 bucket is firewalled or requires access controls, a proxy might be needed. In such scenarios, you need to exclude the S3 bucket address from the “NO_PROXY” list for successful downloads.
NO_PROXY
Usage
Within the NO_PROXY
list, separate the host/domain names with commas or spaces.
Optionally, you can follow each host/domain entry with a port. If not specified, the protocol’s default port is assumed.
To match subdomains, you must either list them all in full (for example, NO_PROXY="foo.example.com,bar.example.com"
),
or apply a wildcard by prefixing the domain name with a period or *.
: No_Proxy=".example.com"
or No_Proxy="*.example.com"
.
To match the whole domain including its subdomains, add it both with and without wildcard to the list:
No_Proxy="example.com,.example.com"
.
To disable all proxies, use the *
wildcard: NO_PROXY="*"
.
NO_PROXY
with an empty list disables no proxies.
Cloud NO_PROXY
Usage
You must include any cloud metadata endpoints (such as the AWS Instance Metadata Service) in the NO_PROXY
list:
AWS EC2 and Azure VM instances must include
169.254.169.254
in the list. If using IPv6 on AWS EC2, addfd00:ec2::254
to the list.AWS ECS Fargate tasks must include
169.254.170.2
.GCP (Google Cloud Platform) VM instances must include
metadata.google.internal
and169.254.169.254
.
Where Proxies Apply
Proxy configuration is relevant to the following Cribl Stream components that make outbound HTTP/S requests:
Destinations
- Amazon CloudWatch Logs
- Amazon Kinesis Data Streams
- Amazon S3 Compatible Stores
- Amazon SQS
- Azure Blob Storage
- Azure Monitor Logs
- Cribl HTTP
- CrowdStrike Falcon LogScale
- Datadog
- Data Lake > S3
- Elasticsearch
- Google Chronicle
- Google Pub/Sub
- Grafana Cloud
- Honeycomb
- Loki
- New Relic Ingest: Logs & Metrics
- New Relic Ingest: Events
- OpenTelemetry
- Prometheus
- SentinelOne DataSet
- SignalFX
- Splunk HEC
- Sumo Logic
- Wavefront
- Webhook
Sources
- Amazon SQS
- Amazon S3
- Azure Blob Storage
- CrowdStrike FDR
- Office 365 Services
- Office 365 Activity
- Office 365 Message Trace
- Prometheus Scraper
- Prometheus Edge Scraper
- Wiz
Collectors
- Amazon S3 Collector
- Azure Blob Storage Collector
- Google Cloud Storage Collector
- Health Check Collector
- REST/API Collector
Notification Targets
Testing Proxies
To initially test your proxy configuration, consider setting up a simple, free proxy server like mitmproxy (https://mitmproxy.org), and then monitoring traffic through that server. Verify that you can trace proxied requests from your Cribl Stream instance, and can validate that outgoing requests (to Destinations) are working properly.
Proxying Multiple Cribl Stream Instances in One Browser
Cribl Stream stores authentication tokens based on each http header’s URI scheme, host, and port information. Within a given browser, Cribl Stream enforces a same-origin policy to isolate instances.
This means that if you want to run multiple proxied Cribl Stream instances in one browser session, you must assign them different URI schemes, hosts, and/or ports. Otherwise, logging into an extra Cribl Stream instance will expire the prior instance’s session and log it out.
For example, assume that you’ve set up this pair of Apache proxy forward rules:
- https://web/cribla forwards to
cribl_hosta:8001/cribla
. - https://web/criblb forwards to
cribl_hostb:8001/criblb
.
These two proxied addresses cannot be run simultaneously in the same browser session. However, this pair – which lead with separate URI schemes – could:
- https://web/cribla forwards to
cribl_hosta:8001/cribla
. - https://web2/criblb forwards to
cribl_hostb:8001/criblb
.
Where separate instances must share URI formats, a workaround is to open the second instance in an incognito/private browsing window, or in a completely different browser.