As detailed in Worker Group and Fleet Placement, Worker Groups should be positioned as close as possible (both logically and physically) to their data Sources. Usually, this means one Worker Group per data center or cloud region.

Strategic placement reduces network latency, simplifies routing, and helps control cross-region egress costs. This section focuses specifically on how placement, data transfers between Worker Groups across WAN links, and choices around persistent queues (PQ), batching, and object-store Destinations affect end-to-end latency.

Latency Trade-offs and Resilience

​

For low-latency use cases (such as security alerting or SRE incident triage), architecture decisions must make trade-offs explicit:

• Persistent Queues (PQ): These provide resilience but introduce disk I/O and buffering delays. For details, see When to Use Persistent Queues.
• Batching and object stores: Destinations like S3-compatible storage typically favor throughput and cost over real-time delivery. Large batch sizes or long “age” thresholds will delay data visibility, regardless of how fast the upstream processing is. For details, see Destination Architecture.

On-Prem Latency and Topology

​

For on-prem environments, follow these patterns to ensure predictable performance:

• Zone affinity: Design Worker Groups per data center or major site. Place each Group in the same network zone as its key data Sources (for example, core logging tiers, security tools, and observability platforms).
• Avoid circular data routing: Do not route traffic from a site to a central Worker Group only to send it back to a Destination at the original site. Avoid configurations that require the same data to cross WAN links multiple times. These inefficient paths create unnecessary latency, increase data egress costs, and expand the “blast radius” (meaning a single link failure or Destination outage can disrupt data flows across multiple locations).
• Avoid unnecessary cross-region traffic: If data must be passed between locations (for example, to a regional collection point or a long-term storage tier), limit this to a single, clearly defined processing tier. Each additional transfer between data centers or cloud regions increases the risk of delays and connection failures.

Two-tier Pattern for Remote Sites

​

Where multiple transfers between Worker Groups are unavoidable (such as remote sites with limited infrastructure), use this model:

1. Local Edge or Worker Group: Deploy a lightweight Cribl Edge Fleet or small Worker Group for collection, normalization, deduplication, and reduction near the Source.
2. Regional Worker Group: Forward reduced data to a regional hub for heavier processing, enrichment, and cross-site fan-out.

Cribl.Cloud Latency and Topology

​

In Cribl.Cloud deployments, Worker Groups are provisioned in specific regions and managed as part of the hosted control and data planes.

• Place Worker Groups close to Sources: Establish Worker Groups in the same or nearest cloud region as the majority of Sources (for example, cloud-native logs, SaaS connectors, or data sent from on-prem via HTTP/TCP).
• Align with primary Destinations: Keep Worker Groups and key Destinations (such as SaaS SIEM, analytics platforms, regional S3 buckets) in the same cloud region to avoid cross-region transfer fees.
  «««< HEAD
• Avoid unnecessary cross-region traffic: Reserves cross-region routing for explicit multi-region aggregation or Disaster Recovery (DR) scenarios.
• Consolidate regional data: If you need to standardize data from multiple regions, use a dedicated aggregation Worker Group instead of linking several production Worker Groups together. This approach simplifies the architecture and ensures that an issue in one processing stage does not trigger a cascading failure that disrupts your entire production environment. =======
• Avoid unnecessary cross-region traffic: Reserve cross-region routing for explicit multi-region aggregation or Disaster Recovery (DR) scenarios.
• Consolidate regional data: If you need to standardize data from multiple regions, use a dedicated aggregation Worker Group instead of linking several production Worker Groups together. This approach simplifies the architecture and ensures that an issue in one processing stage doesn’t trigger a “domino effect” that disrupts your entire production environment.

05590ed0d366a35148d5471d8ba49d70d3d354bf

• Network transit and critical workflows: Only route time-sensitive data through Cribl.Cloud if your Service Level Agreements (SLAs) can accommodate the variable latency of Internet or VPN transit. For mission-critical, ultra-low-latency workflows (such as automated threat blocking or real-time incident response) it is best practice to keep processing on-prem, physically close to both the data Source and the local Destination. Use Cribl.Cloud primarily for asynchronous workflows, such as regional data aggregation, long-term archival storage, or offline security analytics.

Hybrid Latency Patterns

​

Hybrid architectures combine the localized control of on-premises deployments with the elasticity and SaaS-proximity of Cribl.Cloud. A standard high-performance pattern follows a clear, linear path:

1. On-prem (Source): Use Cribl Edge or local Worker Groups for initial collection, normalization, and volume reduction near the data source.
2. Cribl.Cloud (data processing and delivery): Forward the refined data to Cribl.Cloud for complex transformation, enrichment, and route to SaaS analytics platforms located in the same cloud region.

To keep hybrid latency predictable:

• Minimize transfers: Keep the number of transfers (such as from an Edge Fleet to a Worker Group) as low as possible for real-time data flows.
• Define a latency budget: Set a specific performance goal for every major data stream (such as “under 30 seconds from Source to SIEM”). Be sure to distinguish between the expectations for real-time operational flows (data ingest to processing to indexing) and asynchronous workflows (like historical replay or scheduled batch uploads), as their delivery timelines will naturally differ.
• Monitor and tune: Verify that the number of transfers (such as from a Cribl Edge Fleet to a Worker Group, or between multiple Worker Groups) is as small as possible for real-time delivery.
  • Keep Pipelines simple in the real-time operational flows; defer expensive enrichment to asynchronous or batch-oriented flows.
  • Monitor Persistent Queue metrics and disk usage as leading indicators of downstream issues or backpressure.