Cribl LogStream – Docs

Getting started with Cribl LogStream

Questions? We'd love to help you! Meet us in #cribl (sign up)
Download manual as PDF - v2.2.0

    Docs Home

Destinations

Cribl LogStream can send data to various Destinations, including Splunk, Kafka, Kinesis, InfluxDB, Snowflake, Databricks, TCP JSON, and many others.

Streaming Destinations

Destinations that accept events in real time are referred to as streaming Destinations:

Non-Streaming Destinations

Destinations that accept events in groups or batches are referred to as non-streaming Destinations:

How Does Non-Streaming Delivery Work

Cribl LogStream uses a staging directory in the local filesystem to format and write outputted events before sending them to configured Destinations. After a set of conditions is met – typically file size and number of files, further details below – data is compressed and then moved or copied to the final Destination.

An inventory of open, or in-progress, files is kept in the staging directory's root, to avoid having to walk that directory at startup. This can get expensive if staging is also the final directory. At startup, Cribl LogStream will check for any leftover files in progress from prior sessions, and will ensure that they're moved/copied to their final Destination. The process of moving to the final Destination is delayed after startup (default delay: 30 seconds). Processing of these files is paced at one file per service period (which defaults to 1 second).

Batching Conditions

Several conditions govern when files are closed and rolled out:

  1. File reaches its configured maximum size.

  2. File reaches its configured maximum open time.

  3. File reaches its configured maximum idle time.

If a new file needs to be open, Cribl LogStream will enforce the maximum number of open files, by closing files in the order in which they were opened.

Data Delivery


Data is delivered to all Destinations on an at-least-once basis. When a Destination is unreachable, there are three possible behaviors:

  • Block - Cribl LogStream will block incoming events.
  • Drop - Cribl LogStream will drop events addressed to that Destination.
  • Queue - Cribl LogStream will Persistent-Queue events to that Destination.

You can configure the desired behavior through a Destination's Backpressure Behavior option. If this option is not present, Cribl LogStream's default behavior is to Block.

Configuring Destinations


For each destination type, users can create multiple definitions, depending on their requirements. To configure Destinations, select Data > Destinations, select the desired type from the resulting tiles or the left menu, then click + Add New.

Updated 24 days ago

Destinations


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.