Destinations
Cribl Stream can send transformed data to various Destinations, including Cribl HTTP, Cribl TCP, Cribl Lake, Elasticsearch, Amazon Kinesis, Amazon S3 and other object stores, Prometheus and compatible services, InfluxDB, Splunk, Snowflake, Databricks, TCP JSON, and many others.
Destinations can write data to either IPv4 or IPv6 addresses.
Destination are grouped into categories that define how they handle unreachable outputs (backpressure events) and their load-balancing capabilities.
Destination Categories
Destinations can be divided into streaming and non-streaming: those that accept events in real time, and those that batch them from a staging directory.
Other than that, each Destination can belong to one of the following (non-exclusive) categories:
Streaming and Non-Streaming Destinations
Streaming and non-streaming Destination differ in the way they receive events:
- Streaming Destinations accept events in real time.
- Non-streaming Destinations receive events in batches from a staging directory.
Non-Streaming Destinations
Non-streaming Destinations make use of the staging directory, and have specific behavior regarding batching events into files.
Staging Directory
With non-streaming Destinations, Cribl Stream uses a staging directory in the local filesystem to format and write outputted events before sending them to configured Destinations. After a set of conditions is met, data is compressed and then moved to the final Destination.
To reduce costs when the staging directory is also the final directory, Cribl Stream avoids iterating through all the files within a directory by keeping an inventory of open (in progress) files in the staging directory’s root. At startup, Cribl Stream will check for any leftover files in progress from prior sessions, and will ensure that they’re moved to their final Destination. The process of moving to the final Destination is delayed after startup (default delay: 30 seconds). Processing of these files is paced at one file per service period (which defaults to 1 second).
In Cribl.Cloud, using a staging directory is only available on customer-managed hybrid Workers.
Batching Conditions
In non-streaming delivery, a file is closed and rolled out when it reaches its configured maximum:
- Size
- Open time
- Idle time
If a new file needs to be open, Cribl Stream will enforce the maximum number of open files by closing files in the order in which they were opened.
Filesystem-based Destinations
Some Destinations are Filesystem-based, which means they receive files on disk from a staging directory and batch them in a queue. When a batch of events is ready for transmission, Cribl Stream closes the file, optionally compresses it, and transmits the file to the downstream service. Filesystem-based Destinations do not support persistent queues.
Load-balanced Destinations
Certain Destinations offer built-in load-balancing capabilities.
Available Destinations
Cribl Stream supports the following Destinations. You can configure proxy servers for all HTTP-based Destinations.
| Destination | Protocol | Streaming | Filesystem-Based | Load-Balanced |
|---|---|---|---|---|
| Amazon S3 Compatible Stores | HTTP/S | Non-streaming | ✓ | |
| Amazon CloudWatch Logs | HTTP/S | Streaming | ||
| Data Lakes > Cribl Lake | HTTP/S | Non-Streaming | ||
| Data Lakes > Amazon S3 | HTTPS only | Non-Streaming | ✓ | |
| Data Lakes > Amazon Security Lake | HTTP/S | Non-Streaming | ✓ | |
| Amazon Kinesis Data Streams | HTTP/S | Streaming | ||
| Amazon MSK | TCP | Streaming | ||
| Amazon SQS | HTTP/S | Streaming | ||
| Azure Blob Storage | HTTPS only | Non-Streaming | ✓ | |
| Azure Data Explorer | HTTPS only | Streaming or non-streaming | ||
| Azure Event Hubs | TCP | Streaming | ||
| Azure Monitor Logs | HTTPS only | Streaming | ||
| Microsoft Sentinel | HTTP/S | Streaming | ||
| ClickHouse | HTTP/S | Streaming | ✓ | |
| Confluent Cloud | TCP | Streaming | ||
| CrowdStrike Falcon LogScale | HTTPS only | Streaming | ||
| CrowdStrike Falcon Next-Gen SIEM | HTTPS only | Streaming | ||
| Datadog | HTTPS only | Streaming | ||
| Dynatrace HTTP | HTTP/S | Streaming | ✓ | |
| Dynatrace OTLP | HTTP/S | Streaming | ||
| Elasticsearch | HTTP/S | Streaming | ✓ | |
| Elastic Cloud | HTTPS only | Streaming | ✓ | |
| Exabeam | HTTP/S | Non-Streaming | ✓ | |
| Filesystem/NFS | Non-Streaming | ✓ | ||
| Google SecOps | HTTPS only | Streaming | ||
| Google Cloud Logging | HTTPS only | Streaming | ||
| Google Cloud Pub/Sub | HTTPS only | Streaming | ||
| Google Cloud Storage | HTTPS only | Non-Streaming | ✓ | |
| Grafana Cloud | HTTP/S | Streaming | ||
| Graphite | TCP or UDP | Streaming | ||
| Honeycomb | HTTPS only | Streaming | ||
| InfluxDB | HTTP/S | Streaming | ||
| Kafka | TCP | Streaming | ||
| Loki | HTTP/S | Streaming | ||
| MinIO | HTTP/S | Non-Streaming | ✓ | |
| NetFlow | UDP | Streaming | ||
| New Relic Events | HTTPS only | Streaming | ||
| New Relic Logs & Metrics | HTTPS only | Streaming | ||
| OpenTelemetry (OTel) | gRPC or HTTP/S | Streaming | ||
| Prometheus | HTTP/S | Streaming | ||
| SentinelOne DataSet | HTTPS only | Streaming | ||
| ServiceNow Cloud Observability | gRPC or HTTP/S | Streaming | ||
| SignalFx | HTTPS only | Streaming | ||
| SNMP Trap | UDP | Streaming | ||
| Splunk HEC | HTTP/S | Streaming | ✓ | |
| Splunk Load Balanced | TCP | Streaming | ✓ | |
| Splunk Single Instance | TCP | Streaming | ||
| StatsD | TCP or UDP | Streaming | ||
| StatsD Extended | TCP or UDP | Streaming | ||
| Sumo Logic | HTTP/S | Streaming | ||
| Syslog | TCP or UDP | Streaming | ✓ (TCP only) | |
| TCP JSON | TCP | Streaming | ✓ | |
| Wavefront | HTTPS only | Streaming | ||
| Webhook | HTTP/S | Streaming | ✓ |
You can adapt the Amazon S3 Compatible Stores Destination to send data to downstream services like Databricks and Snowflake, for which Cribl Stream currently has no preconfigured Destination. For details, please contact Cribl Support.
Internal Destinations
Internal Destinations are special-purpose Destinations that route data within your Cribl Stream deployment, or among Workers across distributed or hybrid Cribl.Cloud deployments. The following internal Destinations are available:
- Default: Specify a default output from among your configured Destinations.
- Output Router: A “meta-Destination.” Configure rules that route data to multiple configured Destinations.
- DevNull: Simply drops events. Preconfigured and active when you install Cribl Stream, so it requires no configuration. Useful for testing.
- Cribl HTTP: Send data among peer Worker Nodes over HTTP. Streaming and load-balanced.
- Cribl TCP: Send data among peer Worker Nodes over TCP. Streaming and load-balanced.
- SpaceOut: This experimental Destination is undocumented. Be careful!
Data Delivery to Unreachable Destinations
Cribl Stream attempts to deliver data to all Destinations that are configured to receive it at least once. When a Destination is unreachable, there are three possible behaviors:
- Block - Cribl Stream will block incoming events.
- Drop - Cribl Stream will drop events addressed to that Destination.
- Queue - To prevent data loss, Cribl Stream will write events to a persistent queue disk buffer, then forward them when a Destination becomes available. (Available on several streaming Destinations.)
For further information about backpressure (a situation when a Destination receives more data than it can send), see Destination Backpressure Triggers.
You can configure your desired behavior through a Destination’s Backpressure Behavior drop-down. Where other options are not displayed, Cribl Stream’s default behavior is Block. For details about all the above behaviors and options, see Persistent Queues.