On This Page

Home / Edge/Destinations

Destinations

Cribl Edge can send data to various Destinations, including Cribl HTTP, Cribl TCP, Kafka, Kinesis, InfluxDB, Grafana Cloud, TCP JSON, Splunk, and others. Destinations can write data to either IPv4 or IPv6 addresses.

Destinations in the Edge ecosystem
Destinations in the Edge ecosystem

Streaming Destinations
​

Destinations that accept events in real time are referred to as streaming Destinations:

Non-Streaming Destinations
​

Destinations that accept events in groups or batches are referred to as non-streaming Destinations:

The Amazon S3 Compatible Stores Destination can be adapted to send data to downstream services like Databricks and Snowflake, for which Cribl Edge currently has no preconfigured Destination. For details, please contact Cribl Support.

Internal and Special-Purpose Destinations
​

These special-purpose Destinations route data within your Cribl Edge deployment, or among Workers across distributed or hybrid Cloud deployments:

  • Default: Specify a default output from among your configured Destinations.
  • Output Router: A “meta-Destination.” Configure rules that route data to multiple configured Destinations.
  • DevNull: Simply drops events. Preconfigured and active when you install Cribl Edge, so it requires no configuration. Useful for testing.
  • Cribl HTTP: Send data among peer Edge Nodes over HTTP.
  • Cribl TCP: Send data among peer Edge Nodes over TCP.
  • Cribl Stream (Deprecated): Use either Cribl HTTP or Cribl TCP instead.
  • SpaceOut: This experimental Destination is undocumented. Be careful!

How Does Non-Streaming Delivery Work
​

Cribl Edge uses a staging directory in the local filesystem to format and write outputted events before sending them to configured Destinations. After a set of conditions is met – typically file size and number of files, further details below – data is compressed and then moved to the final Destination.

An inventory of open, or in-progress, files is kept in the staging directory’s root, to avoid having to walk that directory at startup. This can get expensive if staging is also the final directory. At startup, Cribl Edge will check for any leftover files in progress from prior sessions, and will ensure that they’re moved to their final Destination. The process of moving to the final Destination is delayed after startup (default delay: 30 seconds). Processing of these files is paced at one file per service period (which defaults to 1 second).

In Cribl.Cloud, using a staging directory is only available on hybrid, customer-managed Edge Nodes.

Batching Conditions
​

Several conditions govern when files are closed and rolled out:

  1. File reaches its configured maximum size.

  2. File reaches its configured maximum open time.

  3. File reaches its configured maximum idle time.

If a new file needs to be open, Cribl Edge will enforce the maximum number of open files, by closing files in the order in which they were opened.

Data Delivery and Persistent Queues
​

Cribl Stream attempts to deliver data to all Destinations on an at-least-once basis. When a Destination is unreachable, there are three possible behaviors:

  • Block - Cribl Stream will block incoming events.
  • Drop - Cribl Stream will drop events addressed to that Destination.
  • Queue - To prevent data loss, Cribl Stream will write events to a Persistent Queue disk buffer, then forward them when a Destination becomes available. (Available on several streaming Destinations.)

For further information about backpressure, see Destination Backpressure Triggers.

You can configure your desired behavior through a Destination’s Backpressure Behavior drop-down. Where other options are not displayed, Cribl Stream’s default behavior is Block. For details about all the above behaviors and options, see Persistent Queues.

Configuring Destinations
​

For each Destination type, you can create multiple definitions, depending on your requirements.

To configure Destinations, from the top nav of a distributed deployment, first click Manage, then select a Fleet to configure and choose one of the options below. For a single-instance deployment proceed to the options below:

  • To access the graphical QuickConnect UI, click Collect. Next, select the desired type, and then click either Add New or (if displayed) Select Existing.
  • To access the Routing UI, click More > Destinations. On the resulting Manage Destinations page’s tiles, select the desired type, then click New Destination.

Capturing Outgoing Data
​

To capture data from a single enabled Destination, you can bypass the Preview pane, and instead capture directly from a Manage Destinations page. Just click the Live button beside the Destination you want to capture.

Destination > Live button
Destination > Live button

You can also start an immediate capture from within an enabled Destination’s config modal, by clicking the modal’s Live Data tab.

Destination modal > Live Data tab
Destination modal > Live Data tab

Monitoring Destination Status
​

Each Destinations’s configuration modal offers two tabs for monitoring: Status and Charts.

Status Tab
​

The Status tab provides details about the Edge Nodes in the Fleet and their status. An icon shows whether the Edge Node is operating normally.

You can click each Edge Node’s row to see specific information, for example, to identify issues when the Destination displays an error. The specific set of information provided depends on the Destination type. The data represents only process 0 for each Edge Node.

The content of the Status tab is loaded live when you open it and only displayed when all the data is ready. With a lot of busy Edge Nodes in a group, or nodes located far from the Leader, there may be a delay before you see any information.

The statistics presented are reset when the Edge Node restarts.

Charts Tab
​

The Charts tab presents a visualization of the recent activity on the Destination. The following data is available:

  • Events in
  • Thruput in (events per second)
  • Bytes in
  • Thruput in (bytes per second)
  • Blocked status

This data (in contrast with the status tab) is read almost instantly and does not reset when restarting an Edge Node.