High-availability Collectors is a resiliency feature for Collector jobs in a Worker Group. When the Leader is unavailable, scheduled and other Leader-orchestrated collection jobs can keep running instead of stopping outright. Many common collection paths, including the REST Collector and other Collector Sources, depend on those jobs to pull data on a schedule. High-availability Collectors reduces interruptions to that ingestion when the Leader fails, is upgraded, or is temporarily unreachable over the network.

Leader High Availability/Failover keeps the Leader role available for the full control plane, including Collector scheduling, usually with standby Leaders and a shared failover volume (often NFS). High-availability Collectors is an optional complement with a narrower focus: orchestration of Collector jobs by Worker Nodes during Leader loss, without NFS or another external coordinator.

In Cribl.Cloud, High-availability Collectors is automatically enabled for Cribl-Managed Worker Nodes, and the setting is hidden. All Worker Nodes in the Worker Group must be on Cribl version 4.19.0 or newer.

In on-prem and hybrid deployments, High-availability Collectors is off by default. To turn it on globally, select Enable Collector HA in the Worker Group’s General Settings.

For eligibility, supported environments, and known behavior, see Requirements and Limitations.

High-Availability Collectors Orchestration ​

High-availability Collectors keeps scheduled Collectors running during Leader outages by letting a Worker Group elect a temporary Captain to orchestrate jobs.

While the Leader is reachable, it remains the normal orchestrator for Collector jobs in the Worker Group.

When the Worker Group detects that the Leader is unreachable, Worker Nodes in that group can elect one peer as Captain. The Captain temporarily runs the jobs framework locally, scheduling and dispatching Collector jobs. Other Worker Nodes pull tasks from the Captain and report task and state updates back to the Captain.

The handoff is not instantaneous. Detecting that the Leader is unreachable and electing a Captain takes a short time. Scheduled collection can pause briefly before the Captain takes over, and again when the Leader resumes control.

When the Leader is reachable again, Worker Nodes adopt the Leader, and the Leader reclaims orchestration from the Captain.

State Continuity ​

High-availability Collectors synchronizes scheduling and collection state across the Worker Group to limit the job history and collection state lost during a failover. Non-Captain Worker Nodes periodically pull the latest scheduler and collection state from the Captain, so this synchronization happens at intervals rather than instantly. When the Leader recovers, the Captain hands its accumulated state back to the Leader before it steps down. As a result, scheduled Collectors usually resume from a recent checkpoint instead of starting over. Because state synchronizes at intervals, a job can still repeat or overlap a run around a failover, as described in Requirements and Limitations.

Requirements and Limitations ​

• Distributed deployments only: High-availability Collectors requires a Distributed deployment with a Leader and Worker Nodes organized into Worker Groups. It does not apply to single-instance (standalone) Cribl Stream.

• Cribl Stream only: High-availability Collectors applies only to Cribl Stream Worker Groups. It is not available for Cribl Edge Fleets, Cribl Search, or Cribl Lake.

• Inter-node connectivity: High-availability Collectors requires Worker Nodes in the Worker Group to reach each other over HTTP so they can coordinate election, task dispatch, and status updates during Captain-led periods.

• Version and configuration parity: Every healthy Worker Node in the Worker Group must run Cribl version 4.19.0 or newer and report the same committed configuration version. A single Worker Node on an older version makes the entire Worker Group ineligible to elect a Captain. The same applies when the Worker Group’s configuration has not converged across all Worker Nodes, such as after a recent change that has not been deployed everywhere. Eligibility returns once the Worker Nodes converge.

• Cribl.Cloud Government: High-availability Collectors is not available in Cribl.Cloud Government (FedRAMP) environments.

• Possible duplicate events: State synchronizes periodically rather than instantly, so a change of Captain can still cause some scheduled runs to be retried or to overlap. High-availability Collectors favors running a scheduled job again over silently skipping it. As a result, Destinations can receive duplicate events around a failover, which you can review in Job Inspector and related job views. Adjust Routes, Pipelines, and downstream consumers as needed to drop or deduplicate on a stable key, or configure Destinations so that sending duplicate events does not corrupt stored totals or state.

• Job history visibility: Each orchestrator stores job run details on its own node, so any change of Captain can leave gaps in the Job Inspector. After the Leader hands off to a Captain, after one Captain replaces another, or after the Leader resumes control, the Job Inspector might not show execution details for runs that a previous Leader or Captain coordinated.

• Quorum: Captain election requires a majority (quorum) of eligible Worker Nodes to be healthy and reachable. A Worker Group with a single eligible Worker Node is supported: that Worker Node becomes Captain without an election. A Captain also needs to keep a majority of reachable Worker Nodes: if it loses that majority during an outage, it steps down and the Worker Group tries to elect a new Captain, which can pause collection again. If the Leader and a majority of Worker Nodes become unavailable at the same time, the remaining Worker Nodes might not reach quorum, and High-availability Collectors is unavailable until enough healthy, compatible Worker Nodes rejoin.

Troubleshooting High-Availability Collectors ​

To observe Captain election and failover behavior, set the logging level for the Captain logging channel to debug. Worker Nodes then log when they adopt the Leader as Captain, open or join an election, win or yield Captaincy, and hand orchestration back to the Leader on recovery.

To confirm that scheduled Collector jobs keep running during a Leader outage, use the Job Inspector to review running and completed jobs.

If Worker Nodes do not elect a Captain while the Leader is unreachable, check the following:

• Worker Node version: every healthy Worker Node in the Worker Group must run Cribl version 4.19.0 or newer. A single older Worker Node makes the whole Worker Group ineligible.
• Configuration parity: every healthy Worker Node must report the same committed configuration version. Commit and deploy any pending changes so the Worker Group converges.
• Quorum: a majority of eligible Worker Nodes must be healthy and able to reach each other over HTTP. Without quorum, no Captain is elected.
• Setting enabled: in on-prem and hybrid Worker Groups, confirm that Enable Collector HA is turned on in General Settings. In Cribl.Cloud, High-availability Collectors is enabled automatically for Cribl-Managed Worker Nodes.

If Worker Nodes elect a new Captain while the Leader is still unreachable, the Job Inspector might not show execution details for runs that were coordinated on a previous Captain. Around each change of Captain, also expect some retried or overlapping Collector job runs and duplicate events at Destinations.

Using High-Availability Collectors With Leader HA ​

High-availability Collectors complements Leader High Availability/Failover and broader control plane HA as described in High Availability Architecture. You can use both High-availability Collectors and Leader HA, Leader HA alone, High-availability Collectors alone, or neither if you accept brief scheduling pauses during Leader maintenance.

If you only need scheduled Collector jobs to run during a short Leader outage, High-availability Collectors is sufficient. If you need the Leader UI, REST API, configuration commits and deploys, or other Leader-only control plane features while the primary Leader is down, use Leader HA.

Leader HA typically pairs standby Leaders with a shared failover volume so the same Leader-hosted configuration and state remain available when the active Leader changes. High-availability Collectors does not use that Leader replication model. Instead, Captain election and Collector job coordination happen only among Worker Nodes over HTTP in the Worker Group. Use both Leader HA and High-availability Collectors when you need replicated Leader availability and continuous Collector scheduling while Worker Nodes cannot reach a Leader.

High-availability Collectors does not replace Leader HA. With High-availability Collectors, a Worker Node Captain temporarily handles only Collector job scheduling and task coordination among Worker Nodes in the Worker Group. The Leader keeps every other control plane responsibility. Until the Leader returns or a standby Leader becomes active, expect gaps or outages for Leader-dependent features, including:

• The Leader UI and REST API (no in-product administration on the Leader).
• Configuration changes that require a Leader (including commit and deploy).
• Ad hoc Collector runs and other operator actions initiated through the Leader.
• Licensing distribution to nodes (restarts during the outage can still be problematic without Leader HA, as described in High Availability Architecture).
• Notifications and Leader-centric metrics collection.