On This Page

Home / Stream/Sources

Sources

Cribl Stream can receive continuous data input from various Sources, including Splunk, HTTP, Elastic Beats, Kinesis, Kafka, TCP JSON, and many others. Sources can receive data from either IPv4 or IPv6 addresses.

Push and Pull Sources
Push and Pull Sources

COLLECTOR Sources
​

Collectors, the top group of Sources in Cribl Stream’s UI, are designed to ingest data intermittently – in on-demand bursts (“ad hoc collection”), or on preset schedules, or by “replaying” data from local or remote stores.

These Sources can ingest data only if a Leader is active:

For background and instructions on using Collectors, see:

PUSH Sources
​

Supported data Sources that send to Cribl Stream.

In the absence of an active Leader, these Sources can generally continue data ingestion. However, some Sources may experience functional limitations during prolonged Leader unavailability. See the respective Source documentation for details.

Data from these Sources is normally sent to a set of Cribl Stream Workers through a load balancer. Some Sources, such as Splunk forwarders, have native load-balancing capabilities, so you should point these directly at Cribl Stream.

PULL Sources
​

Supported Sources that Cribl Stream fetches data from.

System and Internal Sources
​

Sources that supply information generated by Cribl Stream about itself, or from files that it monitors; or, that move data among Workers within your Cribl deployment.

Configuring and Managing Sources
​

For each Source type, you can create multiple definitions, depending on your requirements.

From the top nav, click Manage, then select a Worker Group to configure. Next, you have two options:

To configure via the graphical QuickConnect UI, click Routing > QuickConnect (Stream) or Collect (Edge). Next, click Add Source at left. From the resulting drawer’s tiles, select the desired Source. Next, click either Add Destination or (if displayed) Select Existing.

Or, to configure via the Routing UI, click Data > Sources (Stream) or More > Sources (Edge). From the resulting page’s tiles or left nav, select the desired Source. Next, click New Source to open a New Source modal.

To edit any Source’s definition in a JSON text editor, click Manage as JSON at the bottom of the New Source modal, or on the Configure tab when editing an existing Source. You can directly edit multiple values, and you can use the Import and Export buttons to copy and modify existing Source configurations as .json files.

Capturing Source Data
​

To capture data from a single enabled Source, you can bypass the Preview pane, and instead capture directly from a Manage Sources page. Just click the Live button beside the Source you want to capture.

Source > Live button
Source > Live button

You can also start an immediate capture from within an enabled Source’s config modal, by clicking the modal’s Live Data tab.

Source modal > Live Data tab
Source modal > Live Data tab

Preconfigured Sources
​

To accelerate your setup, Cribl Stream ships with several common Sources configured for typical listening ports, but not switched on. Open, clone (if desired), modify, and enable any of these preconfigured Sources to get started quickly:

  • Syslog – TCP Port 9514, UDP Port 9514
  • Splunk TCP – Port 9997
  • Splunk HEC – Port 8088
  • TCP JSON – Port 10070
  • TCP – Port 10060
  • HTTP – Port 10080
  • Elasticsearch API – Port 9200
  • SNMP Trap – Port 9162
  • Cribl Internal > CriblLogs – Internal
  • Cribl Internal > CriblMetrics – Internal

Backpressure Behavior and Persistent Queues
​

By default, a Cribl Stream Source will respond to a backpressure situation – a situation where its in-memory buffer is overwhelmed with data, and/or downstream Destinations/receivers are unavailable – by blocking incoming data. The Source will refuse to accept new data until it can flush its buffer.

This will propagate block signals back to the sender, if it supports backpressure. Note that UDP senders (including SNMP Traps and some syslog senders) do not provide this support. So here, new events will simply be dropped (discarded) until the Source can process them.

Persistent Queues
​

Push Sources’ config modals provide a Persistent Queue Settings option to minimize loss of inbound streaming data. Here, the Source will write data to disk until its in-memory buffer recovers. Then, it will drain the disk-queued data in FIFO (first in, first out) order.

When you enable Source PQ, you can choose between two trigger conditions: Smart Mode will engage PQ upon backpressure from Destinations, whereas Always On Mode will use PQ as a buffer for all events.

For details about the PQ option and these modes, see Persistent Queues.

Persistent Queues, when engaged, slow down data throughput somewhat. It is redundant to enable PQ on a Source whose upstream sender is configured to safeguard events in its own disk buffer.

Other Backpressure Options
​

The S3 Source provides a configurable Advanced Settings > Socket timeout option, to prevent data loss (partial downloading of logs) during backpressure delays.

Diagnosing Backpressure Errors
​

When backpressure affects HTTP Sources (Splunk HEC, HTTP/S, Raw HTTP/S, and Kinesis Firehose), Cribl Stream internal logs will show a 503 error code.