Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF – v.3.1.2

Upgrading

This page outlines how to upgrade a Cribl LogStream single-instance or distributed deployment along one of the following supported upgrade paths:

  • v2.x ==> v2.x || v3.x
  • v1.7.x/v2.0.x ==> v2.x.x || v3.x
  • v1.6.x or below ==> v1.7.x ==> v2.x.x || v3.x

🚧

LogStream does not support direct upgrades from a Beta to a GA version. To get the GA version running, you must perform a new install.

See notes on Upgrading from LogStream 2.2 or Prior Versions below.

Standalone/Single-Instance

This path requires upgrading only the single/standalone node:

  1. Stop LogStream.

  2. Uncompress the new version on top of the old one.

    On some Linux systems, tar might complain with: cribl/bin/cribl: Cannot open: File exists. In this case, please remove the cribl/bin/cribl directory if it's empty, and untar again. If you have custom functions in cribl/bin/cribl, please move them under $CRIBL_HOME/local/cribl/functions/ before untarring again.

  3. Restart LogStream.

Distributed Deployment

For a distributed deployment, the order of upgrade is: Upgrade first the Leader Node, then upgrade the Worker Nodes, then commit and deploy the changes on the Leader.

Upgrade the Leader Node

  1. Commit and deploy your desired last version. (This will be your most recent checkpoint.)

    • Optionally, git push to your configured remote repo.
  2. Stop Cribl LogStream.

    • Optional but recommended: Back up the entire $CRIBL_HOME directory.

    • Optional: Check that the Worker Nodes are still functioning as expected. In the absence of the Leader Node, they should continue to work with their last deployed configurations.

  3. Uncompress the new LogStream version on top of the old one.

  4. Restart LogStream and log back in.

  5. Wait for all the Worker Nodes to report to the Leader, and ensure that they are correctly reporting the last committed configuration version.

📘

Workers' UI will not be available until the Worker version has been upgraded to match the version on the Leader. Errors like those below will appear until the Worker nodes are upgraded.

Worker Node version mismatchWorker Node version mismatch

Worker Node version mismatch

Upgrade the Worker Nodes

These are the same basic steps as when upgrading a Single Instance, above:

  1. Stop Cribl LogStream on each Worker Node.

  2. Uncompress the new version on top of the old one.

  3. Restart LogStream.

Commit and Deploy Changes from the Leader Node

  1. Ensure that newly upgraded Worker Nodes report to the Leader with their new software version.

  2. Commit and deploy the newly updated configuration only after all Workers have upgraded.

Post-2.1.4 upgrade to 2.2Post-2.1.4 upgrade to 2.2

Post-2.1.4 upgrade to 2.2

Upgrade and Rollback via the UI

LogStream v.2.4.4 and higher provide streamlined options to upgrade the Leader Node (or single instance), as well as Worker Nodes, directly through the UI. In LogStream 3.0 or higher, go to global ⚙️ Settings (lower left) > System > Upgrade. These streamlined controls perform the whole above sequence of stopping the LogStream server, updating the installed package, and restarting LogStream.

LogStream 3.0 (or higher) also enables you to manage automatic backup and rollback in case an upgrade fails.

📘

These options will work only if all LogStream instances (including Worker Processes) start at v.2.4.4 or higher.

Be aware that the Checking for upgrade status message, and its accompanying spinner, can take up to several minutes to resolve. Also, after you initiate an upgrade, it can take up to several minutes before the View button (described below) is displayed.

The following controls are available:

Package source: The default CDN button downloads a package directly from Cribl's content delivery network. Selecting the alternative Path button exposes these additional controls:

  • Package location: Enter either a URL (HTTP) or a local path to the upgrade package.
  • Package hash location: Enter either a URL (HTTP) or a local path to the hash that validates the package. Supports sha256 and md5 formats. (You can simply append .sha256 to the contents of the Package location field.)
  • Save/Cancel buttons: Click Save to store the specified locations. Clicking Cancel restores the CDN package-source selection.

Upgrade/Upgrade Leader

In a Single-instance deployment, the Upgrade button is the only other control provided. In a Distributed deployment, the Upgrade button is displayed on an Upgrade Leader tab, and clicking it upgrades the Leader Node. (As with manual upgrades, always upgrade the Leader before upgrading the Workers.)

Upgrade Worker Groups

This second tab, displayed only in distributed deployments, shows each Worker Group's status.

Upgrade Worker Groups tabUpgrade Worker Groups tab

Upgrade Worker Groups tab

📘

Upgrading Workers from the Leader requires a LogStream Standard or Enterprise license.

Click any row's Upgrade button to upgrade that group. The resulting Upgrade Group dialog offers two states: Basic Upgrade and Advanced Upgrade.

Basic Upgrade Configuration

In this default Upgrade Group dialog, you can simply upgrade the whole Group, by clicking the dialog's Upgrade button to confirm.

LogStream will check to ensure that Workers are upgraded no higher than the Leader's version. Upgrades are performed as the user that was running LogStream on each machine.

Advanced Upgrade Configuration

Click Advanced configuration to expose these additional options:

Quantity %: Specify what percentage of the Group's Workers to upgrade in this operation. If you enter a value less than the default 100%, LogStream will perform a partial upgrade, keeping the remaining Workers active to process data.

Rolling upgrade: Toggle this slider on to upgrade Workers one at a time. Enabling the slider also enables the dialog's two remaining controls:

Retry delay (ms): How many milliseconds to wait between upgrade attempts. Defaults to 1000 ms (1 second).

Retry count: How many times to retry a failed upgrade. Defaults to 5.

After you click the Upgrade confirmation button, the Upgrade Worker Groups tab will display an additional button on this Group's row:

View: Click to display the upgrade task's status in the Job Inspector modal – select that modal's System tab to access details.

📘

When you initiate an upgrade via the UI, the new package is untarred to $CRIBL_HOME/unpack.<random‑hash>.tmp. This location inherits the permissions you've already assigned to $CRIBL_HOME.

Backup and Rollback

By default, LogStream will automatically roll back to a stored backup package if an upgrade (initiated through the UI) fails. You can adjust this behavior at global ⚙️ Settings (lower left) > System > General Settings > Upgrade & Share Settings, using the following controls.

🚧

LogStream can perform rollbacks only on Worker Nodes/instances that started on at least LogStream v. 3.0.0, before the attempted upgrade.

Enable automatic rollback: LogStream will automatically roll back an upgrade if the LogStream server fails to start, or if the Worker Node fails to connect to the Leader. (Toggle to No to defeat this behavior.)

Rollback timeout (ms): Time to wait, after an upgrade, before checking each Node's health to determine whether to roll back. Defaults to 30000 milliseconds, i.e., 30 seconds.

Rollback condition retries: Number of times to retry the health check before performing a rollback. Defaults to 5 attempts.

Check interval (ms): Time to wait between health-check retries. Defaults to 1000 milliseconds, i.e., 1 second.

Backups directory: Specify where to store backups. Defaults to $CRIBL_HOME/state/backups.

Backup persistence: A relative time expression specifying how long to keep backups after each upgrade. Defaults to 24h.

Upgrading from LogStream 2.2 or Prior Versions

As of version 2.3, LogStream Free and One licenses are permanent, but they enforce certain restrictions that especially affect distributed deployments:

  • Even if you have more than one Worker Group defined, only one Worker Group will be visible and usable.

    • This will be the first Group listed in $CRIBL_HOME/local/cribl/groups.yml – typically, the default Group. You can edit groups.yml to move the desired Group to the top.
  • Your cluster will be limited to 10 Worker Processes across all Worker Nodes.

    • LogStream will balance (or rebalance) these Processes as evenly as possible across the Worker Nodes.
  • Authentication will fall back to local authorization. You will not be able to authenticate via Splunk, LDAP, or SSO/OpenID.

  • Git Push to remote repos will not be supported through the product.

🚧

If you are upgrading LogStream Free or LogStream One from version 2.2.x or lower, these changes might require you to adjust your existing configuration and/or workflows.

See Licensing for details on all current license options.

As of LogStream 2.3, licenses no longer need to be deployed directly to Worker Groups. The Leader will push license information down to Worker Groups as part of the heartbeat.

Splunk App Package Upgrade Steps

🚧

See Deprecation note for v.2.1.

Follow these steps to upgrade from v.1.7, or higher, of the Cribl App for Splunk:

  1. Stop Splunk.

  2. Untar/unzip the new app version on top of the old one.

    On some Linux systems, tar might complain with: cribl/bin/cribl: Cannot open: File exists. In this case, please remove the cribl/bin/cribl directory if it's empty, and untar again. If you have custom functions in cribl/bin/cribl, please move them under $CRIBL_HOME/local/cribl/functions/ before untarring again.

  3. Restart Splunk.

Upgrading from Splunk App v.1.6 (or Lower)

As of v.1.7, contrary to prior versions, Cribl's Splunk App package defaults to Search Head Mode. If you have v.1.6 or earlier deployed as a Heavy Forwarder app, upgrading requires an extra step to restore this setting:

  1. Stop Splunk.

  2. Untar/unzip the new app version on top of the old one.

  3. Convert to HF mode by running: $SPLUNK_HOME/etc/apps/cribl/bin/cribld mode-hwf

  4. Restart Splunk.

Updated 17 days ago

Upgrading


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.