On This Page

Home / Stream/ Sources/ Internal·Cribl HTTP

Cribl HTTP

The internal Cribl HTTP Source is available only in distributed deployments. It is provided to facilitate sending data between Worker Nodes that are connected to the same Leader. You’ll find this Source especially valuable in a hybrid Cloud deployment.

Type: System and Internal | TLS Support: YES | Event Breaker Support: No

You might choose this Source over the Cribl TCP Source in certain circumstances, such as when a firewall or proxy blocks raw TCP egress. In single‑instance mode or for testing, you can substitute the Raw HTTP/S Source. (However, this substitution will not provide the single-billing benefits described in the next section.)

How It Works
​

You can use the Cribl HTTP Source to transfer data between Workers. If the Cribl HTTP Source receives data from its Cribl HTTP Destination counterpart, you’re billed for ingress only once – when Cribl first receives the data. All data subsequently relayed to other Workers via a Cribl HTTP Destination/Source pair is not charged.

This use case is common in hybrid Cribl.Cloud deployments, where a customer-managed (on-prem) Worker Node sends data to a Worker in Cribl.Cloud for additional processing and routing to Destinations. However, the Cribl HTTP Destination/Source pair can similarly reduce your metered data ingress in other scenarios, such as on-prem Edge to on-prem Stream.

As one usage example, assume that you want to send data from one Worker Nodes deployed on-prem, to another that is deployed in Cribl.Cloud. You could do the following:

  • Create an on-prem File System Collector (or whatever Collector or Source is suitable) for the data you want to send to Cribl.Cloud.
  • Create an on-prem Cribl HTTP Destination.
  • Create a Cribl HTTP Source, on the target Worker Group in Cribl.Cloud.
  • For an on-prem Worker Node configure a File System Collector to send data to the Cribl HTTP Destination, and from there to the Cribl HTTP Source in Cribl.Cloud.
  • On Cribl-managed Cribl.Cloud Worker Nodes, make sure that TLS is either disabled on both the Cribl HTTP Destination and the Cribl HTTP Source it’s sending data to, or enabled on both. Otherwise, no data will flow. On Cribl.Cloud instances, the Cribl HTTP Source ships with TLS enabled by default.

Configuration Requirements
​

The key points about configuring this architecture are:

  • The Cribl HTTP Destination must be on a Worker Node that is connected to the same Leader as the Cribl HTTP Source.

  • You must specify the same Leader Address on the Worker Nodes that host both the Destination and Source. Otherwise, token verification will fail – breaking the connection and preventing data flow.

  • To get the Leader Address specifically for Cribl.Cloud hybrid Workers, see Hybrid Cribl HTTP/​Cribl TCP Configuration.

  • To configure the Leader Address via the UI, log directly into each Worker Node’s UI. Then select Settings > Global Settings > Distributed Settings > Leader Settings > Address.

  • To configure the Leader Address via the instance.yml file, the host values on the connecting Worker Nodes must be identical. In this example, both Worker Nodes must point to cribl-leader:

    distributed:
  mode: master
  master:
    host: cribl-leader
    port: 4200

  • When you configure the Cribl HTTP Destination, its Cribl endpoint field must point to the Address and Port you’ve configured on the Cribl HTTP Source.

  • Cribl 3.5.4 was a breakpoint in Cribl HTTP Leader/Worker communications. Worker Nodes running the Cribl HTTP Source on Cribl Stream 3.5.4 and later can send data only to Worker Nodes running v.3.5.4 and later. Worker Nodes running the Cribl HTTP Source on Cribl Stream 3.5.3 and earlier can send data only to Worker Nodes running v.3.5.3 and earlier.

  • Finally, it’s important to understand the special way the Cribl HTTP Source handles internal fields.

Configuring the Cribl HTTP Source
​

From the top nav, click Manage, then select a Worker Group to configure. Next, you have two options:

To configure via the graphical QuickConnect UI, click Routing > QuickConnect (Stream) or Collect (Edge). Next, click Add Source at left. From the resulting drawer’s tiles, select System and Internal > Cribl HTTP. Next, click either Add Destination or (if displayed) Select Existing. The resulting drawer will provide the options below.

Or, to configure via the Routing UI, click Data > Sources (Stream) or More > Sources (Edge). From the resulting page’s tile or left nav, select [System and Internal >] Cribl HTTP. Next, click New Source to open a New Source modal that provides the options below.

General Settings
​

Input ID: Enter a unique name to identify this Cribl HTTP Source definition.

Address: Enter the address to bind on. Defaults to 0.0.0.0 (all addresses).

Port: Enter the port number to listen on, e.g., 10200.

Optional Settings
​

Tags: Optionally, add tags that you can use for filtering and grouping in the Cribl Stream UI. Use a tab or hard return between (arbitrary) tag names. These tags aren’t added to processed events.

TLS Settings (Server Side)
​

Enabled Defaults to No. When toggled to Yes, exposes this section’s remaining fields.

Certificate name: Select a predefined certificate from the drop-down. A Create button is available to create a new certificate.

Private key path: Server path containing the private key (in PEM format) to use. Path can reference $ENV_VARS.

Passphrase: Passphrase to use to decrypt private key.

Certificate path: Server path containing certificates (in PEM format) to use. Path can reference $ENV_VARS.

CA certificate path: Server path containing CA certificates (in PEM format) to use. Path can reference $ENV_VARS.

Authenticate client (mutual auth): Require clients to present their certificates. Used to perform mutual authentication using SSL certs. Defaults to No. When toggled to Yes, exposes these two additional fields:

  • Validate client certs: Reject certificates that are not authorized by a CA in the CA certificate path, or by another trusted CA (e.g., the system’s CA). Defaults to No.
  • Common name: Regex matching subject common names in peer certificates allowed to connect. Defaults to .*. Matches on the substring after CN=. As needed, escape regex tokens to match literal characters. E.g., to match the subject CN=worker.cribl.local, you would enter: worker\.cribl\.local.

Minimum TLS version: Optionally, select the minimum TLS version to accept from connections.

Maximum TLS version: Optionally, select the maximum TLS version to accept from connections.

Persistent Queue Settings
​

Enable Persistent Queue defaults to No. When toggled to Yes:

Mode: Choose a mode from the drop-down:

  • With Smart mode, PQ will write events to the filesystem only when it detects backpressure from the processing engine.
  • With Always On mode, PQ will always write events directly to the queue before forwarding them to the processing engine.

Max buffer size: Maximum number of events to hold in-memory before dumping them to disk.

Commit frequency: Number of events to send before committing that Stream has read them.

Max file size: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, etc. Defaults to 1 MB.

Max queue size: The maximum amount of disk space the queue is allowed to consume. Once this limit is reached, Cribl Stream stops queueing and applies the fallback Queue‑full behavior. Enter a numeral with units of KB, MB, etc.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. To this value, Cribl Stream will append /<worker‑id>/<output‑id>.

Compression: Codec to use to compress the persisted data, once a file is closed. Defaults to None; Gzip is also available.

Processing Settings
​

Fields
​

In this section, you can add Fields to each event, using Eval-like functionality.

Name: Field name.

Value: JavaScript expression to compute field’s value, enclosed in quotes or backticks. (Can evaluate to a constant.)

Pre–Processing
​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.

Advanced Settings
​

Enable proxy protocol: Toggle to Yes if the connection is proxied by a device that supports Proxy Protocol v1 or v2. This setting affects how the Source handles the __srcIpPort field.

Capture request headers: Toggle this to Yes to add request headers to events, in the __headers field.

Max active requests: Maximum number of active requests allowed for this Source, per Worker Process. Defaults to 256. Enter 0 for unlimited.

Activity log sample rate: Determines how often request activity is logged at the info level. The default 100 value logs every 100th value; a 1 value would log every request; a 10 value would log every 10th request; etc.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Request timeout (seconds): How long to wait for an incoming request to complete before aborting it. The default 0 value means wait indefinitely.

Keep-alive timeout (seconds): After the last response is sent, Cribl Stream will wait this long for additional data before closing the socket connection. Defaults to 5 seconds; minimum is 1 second; maximum is 600 seconds (10 minutes).

The longer the Keep‑alive timeout, the more Cribl Stream will reuse connections. The shorter the timeout, the closer Cribl Stream gets to creating a new connection for every request. When request frequency is high, you can use longer timeouts to reduce the number of connections created, which mitigates the associated cost.

Connected Destinations
​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

Internal Fields
​

Cribl Stream uses a set of internal fields to assist in handling of data. These “meta” fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

The Cribl HTTP Source (and the Cribl TCP Source) treat internal fields differently than other Sources do. That’s because of the difference in the way that incoming data originates.

Other Sources ingest data that’s not coming from Cribl Edge or Stream, meaning that no Cribl internal fields can be present in that data when it arrives at the Source, and the Source is free to add internal fields without clobbering (overwriting) anything that existed already.

By contrast, the Cribl HTTP Source and the Cribl TCP Source ingest data that’s coming from a Cribl HTTP or Cribl TCP Destination. That data can contain internal fields when it arrives at the Source. This means that if the Source adds internal fields, those could potentially clobber what existed before.

To avoid this problem, the Cribl HTTP Source and the Cribl TCP Source add a unique __forwardedAttrs (i.e., “forwarded attributes”) field. The nested structure of the __forwardedAttrs field contains any of the following fields that are present in the arriving data:

Internal Fields
__headers – Added only when Advanced Settings > Capture request headers is set to Yes.
__inputId
__outputId
__srcIpPort – See details below.
Other Fields
cribl_breaker
cribl_pipe

These fields are copied into__forwardedAttrs, not moved there. As the data (apart from __forwardedAttrs) moves through the Source and any Pipelines, the values of these fields can be overwritten. But the copies of these fields in __forwardedAttrs remain unchanged, so you can retrieve them as necessary.

Overriding __srcIpPort with Client IP/Port
​

The __srcIpPort field’s value contains the IP address and (optionally) port of the HTTP client sending data to this Source.

When any proxies (including load balancers) lie between the HTTP client and the Source, the last proxy adds an X‑Forwarded‑For header whose value is the IP/port of the original client. With multiple proxies, this header’s value will be an array, whose first item is the original client IP/port.

If X‑Forwarded‑For is present, and Advanced Settings > Enable proxy protocol is set to No, the original client IP/port in this header will override the value of __srcIpPort.

If Enable proxy protocol is set to Yes, the X‑Forwarded‑For header’s contents will not override the __srcIpPort value. (Here, the upstream proxy can convey the client IP/port without using this header.)