Boot fully provisioned Workers

Cribl Stream Workers can completely provision themselves, directly from the Leader, upon initial boot. This means that a group of any number of Workers can launch and be fully functional within the cluster, in seconds.

How Does It Work? ​

A Cribl Stream Leader Node provides a bootstrap API endpoint, at /init/install-worker.sh, which returns a shell script. You can run this shell script on any supported machine (see Restrictions below), without Cribl Stream installed. This fully provisions the machine as a Worker Node.

Although you can specify the download URL when you execute the initial curl command, the Cribl Stream package is not downloaded until you generate the script via the API, and then execute it.

For additional information on using environment variables to set the hostname, see Setting the Hostname.

Requirements ​

All Worker Nodes’ hosts must enable ongoing outbound communication to the Leader’s port 4200, to enable the Leader to manage the Workers. While the bootstrap script runs, firewalls on each Worker’s host must also allow outbound communication on the following ports:

• Port 443 to https://cdn.cribl.io.
• Port 443 to a Cribl.Cloud Leader.
• Port 9000 to an on-prem Leader.

If any of this traffic must go through a proxy, see System Proxy Configuration for configuration details. To anticipate and resolve edge cases not mentioned here, see Restrictions below.

Troubleshoot Root Access or SSL Errors ​

The script installs Cribl Stream into /opt/cribl, creates a user and group named cribl to install, own, and run Cribl Stream/Edge, and makes other system-level changes. These actions require root privileges on every Linux distribution, not only on systems like Ubuntu that restrict direct root access. If you run the script as a user that lacks root privileges, it fails.

If your user can’t act as root directly, run the script with the sudo command. Where you place sudo depends on how you run the script:

• When you pipe the bootstrap command into a shell, put sudo immediately before the shell, between the pipe and bash, so that the installation script (not just curl) runs with elevated privileges. The line ends with | sudo bash -. For example: curl '<bootstrap URL>' | sudo bash -
• When you run a script that you’ve already downloaded or pasted, start the command with sudo. For example: sudo bash install-worker.sh

If you encounter errors of this form: ssl certificate problem: self signed certificate in certificate chain …add the -k flag to disable certificate validation.

UI Access ​

Cribl Stream admins can use the UI to concatenate and copy/paste the bootstrap script, automating several steps below. You can use an adjacent option to grab a script that updates a Worker’s Group assignment.

To use these options, you must have the admin Role on a Distributed deployment’s Leader Node, with a certain license tier. You must also create the Worker Groups before using the following instructions to add or reassign Workers to them.

Add New Worker Node ​

1. In the Cribl Stream sidebar, select Workers.

2. On the Workers page, select Add/Update Worker Node.

3. Select Linux > Add from the menu.

4. In the Add Linux Worker modal, the Install package location defaults to Cribl CDN. If desired, change this to Download URL. (For details about this option, see Add Download URL.)

5. In Leader or Outpost hostname/IP, verify or update the Leader URI. If you want the Worker Node to connect through an Outpost,provide the Outpost hostname and port in this field.

6. As needed, correct the target Group, User, and User Group.

7. Copy the resulting script to your clipboard and select Done to close the modal.

8. Paste the script onto your Worker Node’s command line and execute it.

   • As needed (see Troubleshoot Root Access or SSL Errors above), insert the sudo command immediately before bash at the end of the generated Script, so the line ends with | sudo bash -, and/or append the -k flag.

Update Existing Worker Node ​

You can also auto-generate a script that will update an existing Worker’s Group assignment, and/or assign the Worker to a different Leader:

1. In the Cribl Stream sidebar, select Workers.

2. On the Workers page, select Add/Update Worker Node.

3. Select Linux > Update from the menu.

4. In the Update Configuration on Linux Workers modal, select the new target Group for this Worker Node.

5. As needed, correct or change the Leader or Outpost hostname/IP and/or Leader port number.

6. The Script type drop-down defaults to the Environment variable option for generating the script’s command. If you’re not relying on environment variables, change this to CLI.

7. Copy the resulting script to your clipboard and select Done to close the modal.

8. Paste the script onto your Worker Node’s command line and execute it.

   • As needed (see Troubleshoot Root Access or SSL Errors above), insert the sudo command immediately before bash at the end of the generated Script, so the line ends with | sudo bash -, and/or append the -k flag.

API Spec ​

Request Format ​

GET http://<leader hostname or IP>:9000/init/install-worker.sh

Query Strings ​

Example HTTP Request ​

GET http://<leader hostname or IP>:9000/init/install-worker.sh?token=79364d6e-dead-beef-4c6e-554445664867

As of version 3.0, Cribl Stream’s former “master” application components are renamed “leader.” While some legacy terminology remains within CLI commands/​options, configuration keys/values, and environment variables, this document will reflect that.

Example Response ​

#!/bin/sh

### START CRIBL LEADER TEMPLATE SETTINGS ###

CRIBL_MASTER_HOST="<Master FQDN/IP>"
CRIBL_AUTH_TOKEN="<Auth token string>"
CRIBL_VERSION="<Version>"
CRIBL_GROUP="<Default group preference>"
CRIBL_MASTER_PORT="<Master heartbeat port>"
CRIBL_DOWNLOAD_URL="<download url>"

### END CRIBL MASTER TEMPLATE SETTINGS ###

# Set defaults
checkrun() { $1 --help >/dev/null 2>/dev/null; }
faildep() { [ $? -eq 127 ] && echo "$1 not found" && exit 1; }
[ -z "${CRIBL_MASTER_HOST}" ] && echo "CRIBL_MASTER_HOST not set" && exit 1
CRIBL_INSTALL_DIR="${CRIBL_INSTALL_DIR:-/opt/cribl}"
CRIBL_MASTER_PORT="${CRIBL_MASTER_PORT:-4200}"
CRIBL_AUTH_TOKEN="${CRIBL_AUTH_TOKEN:-criblmaster}"
CRIBL_GROUP="${CRIBL_GROUP:-default}"
if [ -z "${CRIBL_DOWNLOAD_URL}" ]; then
    FILE="cribl-${CRIBL_VERSION}-linux-x64.tgz"
    CRIBL_DOWNLOAD_URL="https://cdn.cribl.io/dl/$(echo ${CRIBL_VERSION} | cut -d '-' -f 1)/${FILE}"
fi
UBUNTU=0
CENTOS=0
AMAZON=0

echo "Checking dependencies"
checkrun curl && faildep curl
checkrun adduser && faildep adduser
checkrun usermod && faildep usermod
BOOTSTART=1
SYSTEMCTL=1
checkrun systemctl && [ $? -eq 127 ] && BOOTSTART=0
checkrun update-rc.d && [ $? -eq 127 ] && BOOTSTART=0

echo "Checking OS version"
lsb_release -d 2>/dev/null | grep -i ubuntu && [ $? -eq  0 ] && UBUNTU=1
cat /etc/system-release 2>/dev/null | grep -i amazon && [ $? -eq 0 ] && AMAZON=1

echo "Creating cribl user"
if [ $UBUNTU -eq 1 ]; then
    adduser cribl --home /home/cribl --gecos "Cribl Stream User" --disabled-password
fi
if  [ $CENTOS -eq 1 ] || [ $AMAZON -eq 1 ]; then
    adduser cribl -d /home/cribl -c "Cribl Stream User" -m
    usermod -aG wheel cribl
fi

echo "Installing Cribl Stream"
mkdir -p ${CRIBL_INSTALL_DIR}
curl -Lso ./cribl.tar.gz "${CRIBL_DOWNLOAD_URL}"
tar xzf ./cribl.tar.gz -C ${CRIBL_INSTALL_DIR} --strip-components=1
rm -f ./cribl.tar.gz
chown -R cribl:cribl ${CRIBL_INSTALL_DIR}

if [ $BOOTSTART -eq 1 ]; then
    echo "Setting Cribl Stream to start on boot"
    ${CRIBL_INSTALL_DIR}/bin/cribl boot-start enable -u cribl
fi

mkdir -p ${CRIBL_INSTALL_DIR}/local/_system
cat <<-EOF > ${CRIBL_INSTALL_DIR}/local/_system/instance.yml
distributed:
  mode: worker
  master:
    host: ${CRIBL_MASTER_HOST}
    port: ${CRIBL_MASTER_PORT}
    authToken: ${CRIBL_AUTH_TOKEN}
    tls:
      disabled: true
  group: ${CRIBL_GROUP}
EOF

chown -R cribl:cribl ${CRIBL_INSTALL_DIR}
if [ $BOOTSTART -eq 1 ]; then
  service cribl start
else
  ${CRIBL_INSTALL_DIR}/bin/cribl start
fi

curl Option ​

An easy way of wrapping HTTP methods is to use the curl command. Here is an example, which uses a GET operation by default, with the same URL used in the above HTTP example:

curl http://<leader hostname or IP>:9000/init/install-worker.sh?token=79364d6e-dead-beef-4c6e-554445664867

Check Requirements above to avoid/resolve port or ownership issues.

Chain Script Execution ​

The GET and curl procedures above will only output the contents of the script that needs executing - the script will still need to be manually executed.

However, you can automate that part, too, using a command like the one shown below. This passes the script’s contents to the bash shell to immediately execute.

curl http://<leader hostname or IP>:9000/init/install-worker.sh?token=79364d6e-dead-beef-4c6e-554445664867 | bash -

Unless you run as root, you need to run the installation with elevated privileges (see Troubleshoot Root Access or SSL Errors above). To do this, insert the sudo command immediately before bash, so the line ends with | sudo bash -:

curl http://<leader hostname or IP>:9000/init/install-worker.sh?token=79364d6e-dead-beef-4c6e-554445664867 | sudo bash -

If you don’t have a bash shell available, you can pipe the command to | sh - (or | sudo sh -) instead.

Add Download URL ​

By default, the bootstrap script downloads the Cribl Stream package from the Cribl CDN. To use a different location, add the download_url query parameter when you fetch the script from the Leader. The Leader accepts only http:// or https:// URLs for this parameter.

To pass download_url on the command line, enclose the full Leader URL in double quotes. Otherwise, the shell treats & as a background operator.

Point download_url at the package file itself. This example uses an internal web server:

curl "http://<leader hostname or IP>:9000/init/install-worker.sh?token=79364d6e-dead-beef-4c6e-554445664867&download_url=https://<your_internal_webserver>/cribl-2.2.0-4589617e-linux-x64.tgz" | sh -

Offline Bootstrap Option ​

Use this workflow when Worker Nodes cannot reach the Cribl CDN. Publish the package on a reachable web server, then set download_url as in Add Download URL. Local file paths and file:// URLs are not supported.

1. Download the installation package (.tgz) and its SHA256 or MD5 checksum file (.sha256 or .md5) from the Cribl download page.

2. Publish both files on a web server the Worker Node can reach, such as an internal mirror, artifact repository, or another host on the same network.

   The bootstrap script requires the checksum file (.sha256 or .md5) to be published on that web server in the same directory as the package (the same folder your package URL and checksum URL serve from). When Cribl Stream is running in FIPS mode, MD5 is not an acceptable hashing algorithm.

3. Fetch the bootstrap script from the Leader and set download_url to the full URL of the package file on that server. Follow Add Download URL for quoting and command examples.

Tag to Assign Workers to Worker Groups ​

Cribl Stream uses Mapping Rulesets to map Workers to Worker Groups. When you create a Worker from a bootstrap script, you can take advantage of Mapping Rulesets to specify which Worker Group you want the newly-created Worker to join. This is done by adding tags to the download URL, in the form &tag=tag1&tag=tag2.

Basic Example

Suppose you have a Worker Group, Group420, that you want bootstrapped Workers to join.

In the active Mapping Ruleset for Group420, create a new rule that maps any Worker with the tag awseast1 to the Group420 Worker Group. You can do this with a filter:

cribl.tags.include('awseast1')

Then, in the bootstrap script, add a download URL with a tag that matches the filter you just created. For example:

curl "http://<logstream_leader>:9000/init/install-worker.sh?tag=awseast1&token=<token>" | bash -

When you use the script to bootstrap a new Worker, the Worker will be assigned to Group420.

Advanced Example

Suppose you have four Worker Groups distributed between two regions and two platforms:

Using two tags (one for region and one for platform) you can represent all four possible combinations, and thus all four Worker Groups:

• tag=aws&tag=region1 maps to Group01.
• tag=azure&tag=region1 maps to Group02.
• tag=aws&tag=region2 maps to Group03.
• tag=azure&tag=region2 maps to Group04.

For each Worker Group, you’ll create a Mapping Rule with a filter for the appropriate tag combination to match. For example, this filter would match Group04:

cribl.tags.includes('azure') && cribl.tags.includes('region2')

Then you can use the tag combination in the download URL of a bootstrap script. For example, the tags in this download URL map to Group04:

http://<logstream_leader>:9000/init/install-worker.sh?tag=azure&tag=region2&token=<token>

Any Worker created by the script with the above download URL will be assigned to Group04.

Status Codes ​

Restrictions ​

Keep the following in mind when using bootstrap scripts:

• Each Worker must normally reach the Cribl CDN to download the installation package. To use another location, set download_url to an http:// or https:// URL (Add Download URL). If Workers cannot reach the Cribl CDN, see Offline Bootstrap Option.
• TLS is not enabled by default. If enabled and configured, access to this feature will be over https instead of http.
• Red Hat, Ubuntu, CentOS, and Amazon Linux are the only supported Worker platforms.

User Data ​

For public-cloud customers, an easy way to use bootstrap scripts is in an instance’s user data. First, be sure to set the Leader Node to mode = 'leader'. Then use the following script (changing the command as needed. based on the information above). Upon launch, the Worker Node will reach out to the Leader, download the script, download the Cribl Stream package from the specified location, and then install and configure Cribl Stream:

#!/bin/bash
curl http://<leader-node-ip/host-address>:9000/init/install-worker.sh?token=<auth-token> | sh -