Cribl LogStream can receive data from various Sources, including Splunk, HTTP, Elastic Beats, Kinesis, Kafka, TCP JSON, and many others.
Supported data Sources that send to Cribl LogStream:
- Splunk TCP
- Splunk HEC
- Elasticsearch API
- TCP JSON
- TCP Raw
- HTTP/ Raw
- Kinesis Firehose
- SNMP Traps
Data from these Sources is normally sent to a set of LogStream Workers through a loadbalancer. Some Sources, such as Splunk forwarders, have native loadbalancing capabilities, so you should point these directly at LogStream.
Supported Sources that Cribl LogStream fetches data from:
Sources that are internal to Cribl LogStream:
For each Source type, you can create multiple definitions, depending on your requirements.
To configure Sources, select Data > Sources, select the desired type from the tiles or the left menu, and then click + Add New.
On the Destination side, you can configure how each LogStream output will respond to a backpressure situation – a situation where its in-memory queue is overwhelmed with data.
All Destinations default to Block mode, in which they will refuse to accept new data until the downstream receiver is ready. Here, LogStream will back-propagate block signals through the Source, all the way back to the sender (if it supports backpressure, too).
All Destinations also support Drop mode, which will simply discard new events until the receiver is ready.
Several Destinations also support a Persistent Queue option to minimize data loss. Here, the Destination will write data to disk until the receiver is ready. Then it will drain the disk-buffered data in FIFO (first in, first out) order. See Persistent Queues for details about all three modes, and about Persistent Queue support.
The S3 Source provides a configurable Advanced Settings > Socket timeout option, to prevent data loss (partial downloading of logs) during backpressure delays.
Updated 16 days ago