Cribl - Docs

Getting started with Cribl LogStream

Questions? We'd love to help you! Meet us in #cribl (sign up)

Changelog    Guides

Destinations

You can send data processed through Cribl to other various destinations.
In Cribl, each pipeline can be independently configured with a destination definition. See the "Destinations" block on right below for a list of destination types.

How does it work


When non-streaming destination definitions are associated with a pipeline, Cribl will use a staging directory in the local filesystem to format and write outputted events. After a set of conditions (below) is met, typically file size and number of files, data is then compressed and then moved or copied to the final destination. An inventory of open, or in-progress files is kept in the root of staging directory in order to avoid having to walk that directory at startup. This can get expensive if staging is the final directory. At startup, Cribl will check for any left over files in progress from prior sessions and ensure they're moved/copied to final destination. The process of moving to final destination is delayed after startup (default 30 sec) and (b) processing of these files is paced at one per service period (default 1 second).

There are a number of conditions that govern when files are closed and rolled out:

  1. File reaches its configured max size
  2. File reaches its configured max open time
  3. File reaches its configured max idle time

If a new file needs to be open, Cribl will enforce the number of max open files, by closing them in the order in which they were opened.

Delivery Policies


There is a always at least one destination configured in Cribl. This is referred to as the default destination. In this version of Cribl, while each pipeline can be associated with any destination definition, in the event that that destination is unreachable, Cribl will send data to default. In the event that default is unavailable, the data will be dropped.

Destination Types


Streaming

Destinations that accept events in real-time and support back-pressure are referred to as streaming destinations. Supported destinations:

Splunk
Splunk HEC
AWS Kinesis Streams
Elasticsearch
Honeycomb
Kafka
Syslog
TCP JSON

Non-Streaming

Destinations that accept events in groups or batches are referred to as non-streaming destinations. Supported destinations:

S3 Compatible Stores
Filesystem/NFS

Configuring Destinations


For each destination type users can create multiple definitions depending on their requirements.
To configure destinations, click on Destinations, select the desired type from the left vertical menu then click Add New.