Home / Stream/ Setting Up Cribl Stream/ On-Prem Deployment/ Distributed Deployment/Set Up Leader and Worker Nodes

Set Up Leader and Worker Nodes

In a Distributed deployment, each Cribl Stream instance runs in a defined mode: either as a Leader Node, governing the whole deployment, or as a Worker Node managed by the Leader.

Before setting up your Leader and Worker Nodes, be sure to review Deployment Planning.

You can select the mode in which any instance operates, using:

Internally, the mode for a Leader Node is called master, which corresponds to an earlier, outdated name for the mode. That is why you will see master in commands, configuration keys and environment variable names.

Set Mode with Environment Variables

You can configure the mode for Leader and Worker Nodes via environment variables.

CRIBL_DIST_MODE defines the mode in which the current environment is working: worker or leader. CRIBL_DIST_LEADER_URL defines the Leader Node URL for the deployment.

For a Worker Node, set CRIBL_DIST_LEADER_URL in the following way:

export CRIBL_DIST_LEADER_URL=tcp://${CRIBL_DIST_TOKEN:-criblmaster}@<masterHostname>:4200 ./cribl start

When configuring a Leader Node, use 0.0.0.0 (or another suitable binding address) for host:

export CRIBL_DIST_LEADER_URL=tcp://${CRIBL_DIST_TOKEN:-criblmaster}@0.0.0.0:4200 ./cribl start

If CRIBL_DIST_LEADER_URL is set, you can omit setting CRIBL_DIST_MODE, and the mode for the instance will default to worker.

Environment variables take priority over other ways of setting up Leader and Worker Nodes. If any CRIBL_DIST_* variable is defined in the system, you won’t be able to configure Distributed mode settings via UI.

Set Mode with UI Settings

You can configure the mode for each instance in a Distributed deployment via UI.

  1. In the sidebar, select Settings, then Global.

  2. Select Distributed Settings, then General Settings.

  3. In Mode, choose the mode you want the instance to run as:

    • Leader.
    • Stream: Managed Worker (managed by Leader).
  4. Select Leader Settings, and configure the required Leader settings:

    • Address and Port for Leader Node.

    • Address (for example: criblleader.mycompany.com) for Worker.

    • Forward to Leader API: Toggle on (default) to enable API requests on the distributed port and provide backward compatibility.

      If you need to customize the Leader’s port (from its default 4200), you can do so via the CLI’s mode‑worker command.

  5. Optionally, customize the remaining settings.

If you configure Leader High Availability/Failover, further Distributed Settings changes will be locked for each Leader Node. You can still update distributed settings by modifying configuration files, as outlined on the page linked above.

Enable UI Access to Workers (Teleporting)

Remote UI access to Workers (also known as teleporting) enables you to click through from the Leader’s Workers page to an authenticated view of each Worker’s UI.

When you make a change directly on a Worker Node using remote access (teleport), Cribl Stream does not propagate those changes to the Leader Node. The next commit and deploy from the Leader will overwrite the changes. The only exception is adding tags. Additionally, the Preview view in teleport will not reflect any local changes done using remote access (teleport).

To enable (remote) UI access to Workers from the Leader Node:

  1. In the sidebar, select Worker Groups.
  2. In the row of the Worker Group you want to enable teleporting for, enable UI Access.
Worker UI access setting
Worker UI access setting

On Cribl.Cloud, the Worker Groups page adds extra columns. You’ll see each Group’s Worker type (customer-managed / hybrid versus Cribl-managed / Cribl.Cloud) and, for Cloud Groups, the anticipated ingress rate and Provisioned status. For details, see Managing Cribl.Cloud Worker Groups.

To access a Worker Node via teleporting:

  1. In the sidebar, select Workers.

  2. Select the link for any Worker Node you want to inspect.

    A purple border that appears around the UI indicates that you are viewing the Worker Node remotely.

    Select Restart Stream on the border to restart the Worker, or select the X close button to return to the Workers page.

Authenticated view of a Worker
Authenticated view of a Worker

An alternative way of allowing UI access to Workers is enabling the workerRemoteAccess configuration key in groups.yml.

Add Tags to Workers in Teleport View

You can add tags to a Worker Node you’ve accessed by teleporting. Tags help the Leader Node map a Worker Node to a Worker Group. Note that changing the tags on a Worker can cause the Leader to map the Worker into a new Group.

  1. Teleport into a Worker Node.
  2. On the Workers submenu, select Worker Settings, then Distributed Settings.
  3. Enter tags for this Worker Node and select Save.

In Cribl.Cloud, you can only add tags to hybrid Worker Nodes. You must accept the confirmation modal, as changing the tags on a Worker can affect the Worker Group that the Leader maps it into.

Set Mode with CLI

You can configure the mode for an instance in a Distributed Deployment by using commands: mode-master and mode-worker.

For example, to configure a Worker Node, use command of this form:

./cribl mode-worker -H <master-hostname-or-IP> -p <port> [options] [args]

The command requires the -H and -p parameters. For other options, see the CLI Reference for mode-worker.

Here is an example command:

./cribl mode-worker -H 192.0.2.1 -p 4200 -u myAuthToken

Cribl Stream will need to restart after this command is issued.

Set Mode in instance.yml

You can configure each instance to be either Leader or Worker in $CRIBL_HOME/local/_system/instance.yml, under the distributed section, by setting the mode value:

LeaderWorker
$CRIBL_HOME/local/_system/instance.yml
distributed:
  mode: master
  master:
    host: <IP or 0.0.0.0>
    port: 4200
    forwardToLeaderApi: true
    tls:
      disabled: true
    ipWhitelistRegex: /.&ast;/
    authToken: <auth token>
    enabledWorkerRemoteAccess: false
    compression: none
    connectionTimeout: 5000
    writeTimeout: 10000
$CRIBL_HOME/local/_system/instance.yml
distributed:
  mode: worker
  envRegex: /^CRIBL_/
  master:
    host: <master address>
    port: 4200
    authToken: <token here>
    compression: none
    tls:
      disabled: true
    connectionTimeout: 5000
    writeTimeout: 10000
  tags:
       - tag1
       - tag2
       - tag42
  group: teamsters

Leader/Worker Mode and Auth Tokens

When you configure an instance to work as the Leader, switching from a Single-instance to a Distributed mode, Cribl Stream generates a random auth token.

See How to Secure the Auth Token for the Leader Node for information on changing your auth token and allowed characters.

Troubleshooting Resources

Cribl University offers a Troubleshooting Criblet on Attaching Workers to Leaders. (To follow the direct course link, first log into your Cribl University account - it’s free!)