On This Page

Home / Stream/ Integrations/ Sources/ Internal/Cribl TCP Source

Cribl TCP Source

The Cribl TCP Source receives data from a Cribl TCP Destination. Relaying data between a Cribl TCP Destination and Cribl TCP Source pair prevents double-billing: You won’t incur any additional charges or credit usage after initial ingest to the first Worker Group (depending on your Leader setup – see Transfer Data Between Workspaces or Environments for more details).

It’s common to send data from Cribl Edge to a Cribl Stream Worker Group using this kind of pairing.

Type: Internal | TLS Support: YES | Event Breaker Support: No

For guidance on when to choose this Source versus the Cribl HTTP Source, see Interoperation Protocols.

Within Cribl Stream, the Cribl TCP Source is available only in Distributed deployments. In single‑instance mode, or for testing, you can use the TCP JSON Source instead. However, this substitution will not facilitate sending all internal fields, as described below.

How It Works
​

You can use the Cribl TCP Source to transfer data between Worker Nodes and Workers. If the Cribl TCP Source receives data from its Cribl TCP Destination counterpart on another Worker, you’re billed for ingress only once – when Cribl first receives the data.

Using a Cribl HTTP Destination/Source pair, you’re not charged for subsequent data sent to:

  • Workers that share a Leader.
  • Workers that do not share a Leader, but that do have the exact same on-prem license installed, or two Cribl.Cloud Workspaces set up in the same Organization.

This use case is common in hybrid Cribl.Cloud deployments, where a customer-managed (hybrid) Worker Node sends data to a Worker in Cribl.Cloud for additional processing and routing to Destinations. However, the Cribl TCP Destination/Source pair can similarly reduce your metered data ingress in other scenarios, such as on-prem Cribl Edge to on-prem Cribl Stream.

As one usage example, assume that you want to send data from one Node deployed on-prem, to another that is deployed in Cribl.Cloud. You could do the following:

  • Create an on-prem File System Collector (or whatever Collector or Source is suitable) for the data you want to send to Cribl.Cloud.
  • Create an on-prem Cribl TCP Destination.
  • Create a Cribl TCP Source, on the target Cribl Stream Worker Group or Cribl Edge Fleet in Cribl.Cloud.
  • For an on-prem Node, configure a File System Collector to send data to the Cribl TCP Destination, and from there to the Cribl TCP Source in Cribl.Cloud.
  • On Cribl-managed Nodes in Cribl.Cloud, make sure that TLS is either disabled on both the Cribl TCP Destination and the Cribl TCP Source it’s sending data to, or enabled on both. Otherwise, no data will flow. On Cribl.Cloud instances, the Cribl TCP Source ships with TLS enabled by default.

Configuration Requirements
​

The key points about configuring this architecture are:

  • When you configure the Cribl TCP Destination, its Address and Port values must point to the Address and Port you’ve configured on the Cribl TCP Source.
  • Cribl 3.5.4 was a breakpoint in Cribl TCP Leader/Worker communications. Nodes running the Cribl TCP Source on Cribl 3.5.4 and newer can send data only to Nodes running v.3.5.4 and newer. Nodes running the Cribl TCP Source on Cribl 3.5.3 and older can send data only to Nodes running v.3.5.3 and older.

Configure the Cribl TCP Source
​

  1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
    • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Source and select the Source you want from the list, choosing either Select Existing or Add New.
    • To configure via the Routes, select Data > Sources (Stream) or More > Sources (Edge). Select the Source you want. Next, select Add Source.
  2. In the New Source modal, configure the following under General Settings:
    • Enabled: Toggle on to enable the Source.
    • Input ID: Enter a unique name to identify this Cribl TCP Source definition. If you clone this Source, Cribl Stream will add -CLONE to the original Input ID.
    • Description: Optionally, enter a description.
    • Address: Enter hostname/IP to listen for TCP JSON data. For example, localhost or 0.0.0.0.
    • Port: Enter the port number to listen on, for example, 10300.
  3. Next, you can configure the following Optional Settings:
    • Tags: Optionally, add tags that you can use for filtering and grouping in the Cribl Stream UI. Use a tab or hard return between (arbitrary) tag names. These tags aren’t added to processed events.
  4. Optionally, configure any TLS settings, Persistent Queue, Processing, and Advanced settings, or Connected Destinations outlined in the sections below.
  5. Select Save, then Commit & Deploy.

TLS Settings (Server Side)
​

Enabled: Defaults to toggled off. When toggled on:

Certificate name: Name of the predefined certificate.

Private key path: Server path containing the private key (in PEM format) to use. Path can reference $ENV_VARS.

Passphrase: Passphrase to use to decrypt private key.

Certificate path: Server path containing certificates (in PEM format) to use. Path can reference $ENV_VARS.

CA certificate path: Server path containing CA certificates (in PEM format) to use. Path can reference $ENV_VARS.

Authenticate client (mutual auth): Require clients to present their certificates. Used to perform mutual authentication using SSL certs. Default is toggled off. When toggled on:

  • Validate client certs: Reject certificates that are not authorized by a CA in the CA certificate path, or by another trusted CA (for example, the system’s CA). Default is toggled off.

  • Common name: Regex matching subject common names in peer certificates allowed to connect. Defaults to .*. Matches on the substring after CN=. As needed, escape regex tokens to match literal characters. For example, to match the subject CN=worker.cribl.local, you would enter: worker\.cribl\.local.

Minimum TLS version: Optionally, select the minimum TLS version to accept from connections.

Maximum TLS version: Optionally, select the maximum TLS version to accept from connections.

Persistent Queue Settings
​

Enable persistent queue: Defaults to toggled off. When toggled on:

Mode: Choose a mode from the drop-down:

  • With Smart mode, PQ will write events to the filesystem only when it detects backpressure from the processing engine.
  • With Always On mode, PQ will always write events directly to the queue before forwarding them to the processing engine.

Buffer size limit: Maximum number of events to hold in memory before dumping them to disk. Defaults to 1000.

Commit frequency: Number of events to send before committing that Cribl Stream has read them. Defaults to 42.

File size limit: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, and so on. Defaults to 1 MB.

Max queue size: The maximum amount of disk space the queue is allowed to consume. Once this limit is reached, Cribl Stream stops queueing and applies the fallback Queue‑full behavior. Enter a numeral with units of KB, MB, and so on.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. To this value, Cribl Stream will append /<worker‑id>/<output‑id>.

Compression: Codec to use to compress the persisted data, once a file is closed. Defaults to None; Gzip is also available.

Processing Settings
​

Fields
​

In this section, you can define new fields or modify existing ones using JavaScript expressions, similar to the Eval function.

  • The Field Name can either be a new field (unique within the event) or an existing field name to modify its value.
  • The Value is a JavaScript expression (enclosed in quotes or backticks) to compute the field’s value (can be a constant). Select this field’s advanced mode icon (far right) if you’d like to open a modal where you can work with sample data and iterate on results.

This flexibility means you can:

  • Add new fields to enrich the event.
  • Modify existing fields by overwriting their values.
  • Compute logic or transformations using JavaScript expressions.

Pre–Processing
​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline or Pack to process data from this input before the data is sent through the Routes.

Advanced Settings
​

Enable proxy protocol: Toggle on if the connection is proxied by a device that supports proxy protocol v1 or v2.

Enable load balancing (Cribl Stream only): Toggle on to distribute the data from incoming TCP connections across multiple Worker Processes. This improves performance compared to the default of having all data from an incoming connection processed on a single Worker Process.

  • This spins up an extra Worker Process (named wLB in Settings > Processes) on the Worker Node, which handles splitting data from incoming TCP connections across all other Worker Processes.
  • When enabled, the option to include common fields in the JSON header is not supported. This means fields specified under fields in the JSON header are not automatically added to all events.

Load balancing is available only in Cribl Stream Distributed deployments.

Active connection limit: Maximum number of active connections allowed per Worker Process. Defaults to 1000. Set a lower value if connection storms are causing the Source to hang. Set to 0 for unlimited connections.

Socket idle timeout (seconds): The duration that Cribl Stream will wait for activity on an idle TCP socket before closing the connection. Disabled when set to 0, the default.

Forced socket termination timeout (seconds): The extra time the server waits before forcibly closing a socket that has been idle (TCP socket idle timeout) or exceeded its maximum lifespan (TCP socket max lifespan) but has not yet properly closed. This prevents resource leaks caused by unresponsive clients or network issues. Configure based on network latency and client behavior. Default: 30 seconds. Set to 0 to disable.

Socket max lifespan (seconds): The duration that a socket is allowed to remain open, regardless of activity. This setting prevents resource exhaustion (such as TCP pinning) by limiting the lifespan of connections. Configure based on expected connection durations and resource availability. Disabled when set to 0, the default.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the configuration will be enabled everywhere.

Connected Destinations
​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

Internal Fields
​

Cribl Stream uses a set of internal fields to assist in handling of data. These “meta” fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

The Cribl TCP Source (and the Cribl HTTP Source) treat internal fields differently than other Sources do. That’s because of the difference in the way that incoming data originates.

Other Sources ingest data that’s not coming from Cribl Edge or Cribl Stream, meaning that no Cribl internal fields can be present in that data when it arrives at the Source, and the Source is free to add internal fields without clobbering (overwriting) anything that existed already.

By contrast, the Cribl TCP Source and the Cribl HTTP Source ingest data that’s coming from a Cribl TCP or Cribl HTTP Destination. That data can contain internal fields when it arrives at the Source. This means that if the Source adds internal fields, those could potentially clobber what existed before.

To avoid this problem, the Cribl TCP Source and the Cribl HTTP Source add a unique __forwardedAttrs (that is, “forwarded attributes”) field. The nested structure of the __forwardedAttrs field contains any of the following fields that are present in the arriving data:

Internal Fields
__srcIpPort
__inputId
__outputId
fleet / group
Other Fields
cribl_breaker
cribl_pipe

These fields are copied into __forwardedAttrs, not moved there. As the data (apart from __forwardedAttrs) moves through the Source and any Pipelines, the values of these fields can be overwritten. But the copies of these fields in __forwardedAttrs remain unchanged, so you can retrieve them as necessary.

Periodic Logging
​

Cribl Stream logs metrics about incoming requests and ingested events once per minute.

These logs are stored in the metrics.log file. To view them in the UI, open the Source’s Logs tab and choose Worker Process X Metrics from the drop-down, where X is the desired Worker Process.

This kind of periodic logging helps you determine whether a Source is in fact still healthy even when no data is coming in.