The Cribl Edge and Cribl Stream overlay uses Cribl Edge to collect and shape data at the Source, maximizing efficiency and resiliency before forwarding to a Cribl Stream Worker Group.

Common Patterns

​

The primary use cases and examples of this overlay are:

• Highly distributed environments: For highly distributed environments with wide geographical spread, use this overlay to establish local data collection and control. This involves grouping Edge Nodes (on sites like branch offices or endpoints) into Fleets, which then forward their data to one or more Stream Worker Groups using Cribl HTTP.

• Local data needs: Use this overlay at the Source to enable local data buffering, pre-transmission filtering/sampling, or enrichment with host-specific context.

• Standardized collection: Implement this overlay to achieve standardized data collection across diverse workloads. This allows you to replace multiple third-party agents with a single, unified, and centrally managed Cribl Edge solution.

Benefits

​

• Improved resilience: This overlay uses two layers of queues, local queues on the Edge Node and persistent queues (PQs) on the Stream Worker Groups. This combination significantly reduces data loss during failures.

• Bandwidth reduction: Edge Nodes filter, sample, or summarize data at the Source, sending only the necessary, curated events upstream to reduce bandwidth and ingestion costs, particularly across expensive WAN links.

• Operational consistency: Achieve consistent configuration and management across a high volume of remote nodes using the centralized control provided by Fleets and Subfleets. This ensures that the same configurations can be delivered via Packs everywhere.

• Simplified ingest: The overlay simplifies data collection by preserving a single logical flow where Cribl Edge feeds data to Cribl Stream for processing, which then sends it onward to the Destinations.

Design Notes

​

• Edge alignment: Cribl Edge Fleets should be paired with their nearest Regional Worker Group (if applicable) to keep traffic local and minimize cross-region costs. Use Cribl HTTP with TLS as the default secure protocol for Cribl Edge to Cribl Stream flows.

• Lightweight processing: Keep Cribl Edge processing focused on collection, tagging, simple parsing, and basic PII scrubbing. Offload heavy enrichment, large lookup tables, or complex routing logic to the central Stream Worker Groups to maximize Cribl Edge performance.

• Topology planning: Plan the Fleet design (such as one Fleet per OS family or region) to align with the overall Worker Group placement strategy. For details, see Fleet Design.

• Connectivity: Where direct connectivity isn’t possible, consider intermediary components like VPNs, outposts, or similar patterns. These methods ensure that the data still follows the required Edge-to-Stream sequence while using a secure channel for transmission.