Common Patterns (Overlays)
Overlays are patterns that are layered onto a Distributed: Multi-Worker Group/Fleet. They serve to define logical partitioning, enforce governance, and meet strategic goals.
- Functional split: Partitions Worker Groups by workload type (push, pull, replay). This allows Worker Groups to scale independently and run more efficiently, and it isolates failures so they do not interrupt the live data streams.
- Regional or geo split: Divides Worker Groups by physical location (region, data center, or sovereignty boundary). This helps meet regulatory compliance (like GDPR) and reduces cross-region egress costs.
- Worker Group to Worker Group bridging: Provides a secure mechanism using Cribl HTTP/TCP to route data across trust boundaries. It also enforces schema governance and compliance before data transfer.
- Cribl Edge and Stream: Uses Edge Nodes near the Source to handle local collection, filtering, and buffering. This maximizes efficiency and resiliency before forwarding curated data to Cribl Stream Worker Groups.
- Hub-and-spoke with Core Worker Group: Uses spoke Worker Groups to ingest data locally, while a central Core Worker Group performs global normalization, enrichment, and controlled routing to all final Destinations.
- Replay-first: Prioritizes long-term durability and cost control by initially writing all raw data to low-cost object storage. It then feeds downstream platforms with data using real-time subsets and flexible on-demand replay.