Diagnosing Issues

To help diagnose LogStream problems, you can share a diagnostic bundle with Cribl Support. The bundle contains a snapshot of configuration files and logs at the time the bundle was created, and gives troubleshooters insights into how LogStream was configured and operating at that time.

What’s in the Diagnostic Bundle

The following directories (and their contents) off of $CRIBL_HOME are included:

  • /default/*
  • /local/*
  • /log/*
  • /groups/*
  • /state/jobs/* – will return all jobs if left empty.

Creating and Exporting a Diagnostic Bundle

Users can create and securely share bundles with Cribl Support either from the UI or from the CLI.

In either case, you’ll need outbound internet access to https://diag‑upload.cribl.io and a valid support case number. That site only works when using the cribl diag command or uploading using the LogStream UI (i.e. connecting directly to it with your web browser will fail).

Using the UI

To create a bundle, go to global ⚙️ Settings (lower left) > Diagnostics > Diagnostic Bundle and click Create Diagnostic Bundle.

  • To download the bundle locally to your machine, click Export.
  • To share the bundle with Cribl Support, toggle Send to Cribl Support to Yes, enter your case number, and then click Export.

You can create a bundle from individual workers if you have the Worker UI access setting enabled. Go to Workers > <worker-name> > Settings (top right) > Diagnostics > Diagnostic Bundle, and click ** Create Diagnostic Bundle**.

Previously created bundles are stored in $CRIBL_HOME/diag. They’re also listed in the UI, where you can re-download them or share them with Cribl Support.

Using the CLI

To create a bundle using the CLI, use the diag command.

Open your support case before sending a diagnostic bundle to Cribl Support. Then, when creating the bundle, be sure to use the case number provided. Case numbers are 8 digits in length and you must include any leading zeros. For example, 00001234.

diag command CLI
# $CRIBL_HOME/bin/cribl diag
Usage: [sub-command] [options] [args]

Commands:
get    - List existing Cribl LogStream diagnostic bundles
create - Creates diagnostic bundle for Cribl LogStream
send   - Send LogStream diagnostic bundle to Cribl Support, args:
   -c <caseNumber> - Cribl Case Number
  [-p <path>]      - Diagnostic bundle path (if empty, then new bundle will be created)


## Creating a diagnostic bundle
# $CRIBL_HOME/bin/cribl diag create
Created Cribl LogStream diagnostic bundle at /opt/cribl/diag/cribl-logstream-<hostname>-<datetime>.tar.gz.

## Creating and sending a diagnostic bundle
# $CRIBL_HOME/bin/cribl diag send -c <caseNumber>
Sent LogStream diagnostic bundle to Cribl Support

## Sending a previously created diagnostic bundle
# $CRIBL_HOME/bin/cribl diag send -p /opt/cribl/diag/cribl-logstream-<hostname>-<datetime>.tar.gz -c <caseNumber>
Sent LogStream diagnostic bundle to Cribl Support

Including CPU Profiles

If Cribl Support asks you to grab CPU profiles of Worker Processes, follow these steps:

  1. Use top or htop on the Worker Node to identify Worker PIDs consuming a lot of CPU.
  2. See Sizing & Scaling > CPU Profiling for instructions on accessing the UI’s Profile options (for your deployment type), and generating and saving profiles.
  3. Find the Worker Processes matching the PIDs you identified above.
  4. Click Profile on each. Start with the default 10-second Duration.
  5. Once the profile is displayed, save it to a JSON file. (See details at the above link.)
  6. Repeat steps 3–6 for other CPU-intensive Worker Processes.
  7. Upload the profile JSON files to Cribl Support.

On an already CPU-starved Worker Node, profiling might fail with an error message, or just hang. In this case, you might need a few retries to get a successful profile.