Cribl Edge to Cribl Stream
Cribl Edge automatically discovers logs, metrics, application data, etc. – in real time – from your configured endpoints, and delivers them to Cribl Stream or any supported destination. Meanwhile, Cribl Stream can help collect, reduce, enrich, transform, and route data from Cribl Edge to any destination. And using a Cribl TCP Source, you can collect and route data from Edge Nodes to Stream Worker Nodes connected to the same Leader, without incurring additional cost.
This guide outlines how to route data from an Edge Node (or an entire Fleet) to an existing Stream Worker Group for additional processing. We will walk you through the following:
- Configure Cribl TCP Source on Cribl Stream to receive data from the Edge Node.
- Configure the Exec Source on Cribl Edge to collect data on the Edge Node.
- Configure the Cribl TCP Destination on Cribl Edge to send data to Cribl Stream.
- Configure a Route to Send the Data.
And finally, we will confirm the data flow.
While this use case connects Edge Nodes to Workers through the Cribl TCP Source and Destination, you can also use the Cribl HTTP Source and Destination in certain circumstances – such as when a firewall or proxy blocks raw TCP egress.
Configure the Cribl TCP Source on Cribl Stream
In Cribl Stream, start by configuring and enabling a Cribl TCP Source. The key requirement here is to set the Port to listen on. By default, the Cribl TCP Destinations listen on Port 10300. To simplify our scenario, we will set the Cribl TCP Source to listen on the same Port. (Optionally, you can also configure TLS, Event Breakers, metadata fields, and/or a pre-processing Pipeline.)
When done, Commit and Deploy your changes. Before moving on to the next step, confirm that your Source is healthy.
On Cribl-managed Cribl.Cloud Worker/Edge Nodes, make sure that TLS is either disabled on both the Cribl TCP Source and the Cribl TCP Destination it’s receiving data from, or enabled on both. Otherwise, no data will flow. In the Source, TLS is enabled by default.
Configure the Exec Source on Cribl Edge
Next, we’ll configure the Exec Source on your Edge Node. This Source will break the incoming streams of data into discrete events, and send them to Cribl Stream.
In this step, you can swap out the Exec Source by instead configuring a
System Metrics or
The Exec Source enables you to periodically execute a command and collect its stdout output. In the Exec Source’s configuration modal, specify:
- Which command to execute.
- The number of times to attempt running the command.
- The interval between attempts.
In our example, we are running the ps command to list and retrieve running processes every 10 seconds.
If we don’t configure an Event Breaker, then with each capture we run on the dataset, each process will be ingested as its own event, without the header information. So to structure the data, we’ll add an Event Breaker.
On the Exec Source configuration modal’s left tab, select Event Breaker. In the Event Breaker rulesets drop-down, select Cribl – Do Not Break Ruleset.
Next, preview your data on the modal’s Live Data tab.
Configure the Cribl TCP Destination on Cribl Edge
To get the data flowing, we’ll configure the Cribl TCP Destination on your Edge Node. A few things to note when configuring this Destination:
- Set the Port to listen on. For this example, we’ll use the default
10300. If you configure a different Port, make sure the Source points to the same Address and Port. - If you don’t have a load balancer in front of your Workers, you can configure load balancing directly on this Destination.
- Optionally, define your Compression, Throttling, and Backpressure behavior requirements.
- On Cribl-managed Cribl.Cloud Worker/Edge Nodes, make sure that TLS is either disabled on both the Cribl TCP Destination and the Cribl TCP Source it’s sending data to, or enabled on both. Otherwise, no data will flow. In the Destination, TLS is disabled by default.
Once you’ve configured your Destination, test it to verify that your Edge Node can communicate with the Stream Worker Group.
Configure a Route to Send the Data
Finally, configure a Route to send your data to Cribl Stream. In this example, we are using the passthru Pipeline.
Confirm the Data Flow
To confirm that your data is flowing, navigate back to Cribl Stream’s Cribl TCP Source. Run a Live Data capture on the Source.
You can also check the Monitoring page’s Data submenu, to isolate the throughput on your Source.
