Process and route data collected by Cribl Edge.

You can route data collected in Cribl Edge in several ways:

• To Cribl Stream: For further processing and scalable distribution. See Send Data from Cribl Edge to Cribl Stream ​.
• To Cribl Search: For analysis or visualization. Use the Cribl Search Destination or see Collect and Visualize Nvidia GPU Metrics for an example use case.
• Directly to Destinations: Such as cheap storage, SIEMs, or other security and IT tools.

Cribl Edge is not intended for extensive data manipulation. Limit Pipelines on Edge Nodes to basic Cribl Functions. Avoid complex operations, to prevent bottlenecks and ensure optimal performance.

A typical workflow offloads major processing from Cribl Edge Nodes, for example by:

1. Collecting data locally.
2. Filtering out unwanted data, using the Drop Function.
3. Routing the remaining data to Cribl Stream for further processing and delivery to your desired Destinations.

You can check the Pipeline Profiler to access statistics on each Pipeline’s performance.

QuickConnect vs. Routes in Cribl Edge ​

QuickConnect and Data Routes are both interfaces for managing data flow in Cribl Edge, but they serve distinct purposes, based on the complexity of your routing needs.

QuickConnect is ideal for simple, straightforward scenarios with a given Source sending the same set of data through an optional Pipeline to one or more Destinations. It offers ease of use and speedy configuration.

Data Routes, which are the default option, are better suited for complex data routing involving conditional processing. They provide flexibility, advanced logic, and scalability for handling intricate data flows.

Choose QuickConnect for rapid, minimal configuration setups, and opt for Routes when your data requires more sophisticated routing and processing.

Processing Cribl Edge Data with Pipelines ​

Once you have collected your data, you have options to transform it before sending it to your Destinations.

You can transform data within pre-processing, processing, or post-processing Pipelines. Some examples of common operations performed in Pipelines in Cribl Edge include dropping events and enriching them with metadata.

Drop Events to Lower Ingest Cost ​

By using the Drop Function, you can effectively eliminate data from your ingestion pipeline without incurring additional costs.

There are few ways you can take advantage of this Function:

• Use pre-processing Pipelines to filter and drop specific events, based on criteria like event codes.
• Use processing Pipelines to drop data before cloning events to send them to multiple Destinations. This reduces data volume.
• Safely experiment with configurations, without incurring costs, by routing data to the DevNull Destination.

The number of dropped events that lower the ingestion cost is calculated based on average event size. This means it may not represent the exact number of bytes you drop.

Enrich Events by Adding Metadata ​

To enrich and refine your data within Cribl Edge, effectively use the Eval Function’s Add Field option. This feature enables you to dynamically create new fields based on existing data elements - such as tags from mapping rules, or event metadata. By converting tags and metadata to fields, you can enhance data analysis and manipulation capabilities.

For instance, a File Monitor might be collecting IIS logs from a Windows server. To ensure that these events are stored correctly in your final analysis system, you could specify a value of index = 'IIS'. This creates a new field named index with the value IIS, providing essential context for your data.