Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF – v.3.1.1

Monitoring

To get an operational view of a Cribl LogStream deployment, you can consult the following resources.

Monitoring Resources

Monitoring Page

Select Monitoring from the left nav (distributed deployments) or top nav (single-instance deployments). The resulting Monitoring page displays information about traffic in and out of the system, as well as collection jobs and tasks. It tracks events, bytes, splits by data fields over time, and broader system metrics.

The initial view (below) shows aggregate data for all Groups and all Workers. You can use the drop-downs at the upper right to isolate individual Groups, or individual Workers. Here, you can also change the display's granularity from the default 15 min. Coverage is limited to the previous 24 hours (this maximum is not configurable).

The displayed CPU Load Average is an average per Worker Process, updated at 1‑minute granularity. (It is not an average for the Worker Node as a whole.)

Byte-related charts show the uncompressed size of processed data. Bytes in/out are measured based on the size of _raw (meaning that metrics events will reflect 0 bytes processed, because they include no _raw field).

Monitoring pageMonitoring page

Monitoring page

Dense displays are condensed to sparklines for legibility. Hover over the right edge to display Maximize buttons that you can click to zoom these up to detailed graphs.

Sparklines and fly-outSparklines and fly-out

Sparklines and fly-out

You can hover over an expanded graph fly-out to display further details.

Throughput detailsThroughput details

Throughput details

Data Monitoring

From the Monitoring page's top nav, open the Data submenu to isolate throughput for any of the following:

  • Sources
  • Routes
  • Pipelines
  • Packs
  • Destinations
  • Data Fields
Monitoring > Data submenu (Pipelines selected)Monitoring > Data submenu (Pipelines selected)

Monitoring > Data submenu (Pipelines selected)

System Monitoring

From the Monitoring page's top nav, open the System submenu to isolate throughput for any of the following:

Monitoring > System submenu (Jobs in‑flight selected)Monitoring > System submenu (Jobs in‑flight selected)

Monitoring > System submenu (Jobs in‑flight selected)

Licensing

Select System > Licensing from the Monitoring page's top nav to check your licenses' expiration dates, daily data throughput quotas, and daily and 90-day trailing daily throughput.

Job Inspector

Select System > Job Inspector from the Monitoring page's top nav to view and manage pending, in-flight, and completed collection jobs and their tasks. For details about the resulting page, see Monitoring and Inspecting Collection Jobs.

Flows (Beta)

Select Flows from the Monitoring page's top nav or ••• overflow menu to see a graphical, left-to-right visualization of data flow through your LogStream deployment.

Internal Logs and Metrics

Select Logs from the Monitoring page's top nav. LogStream's internal logs and internal metrics provide comprehensive information about an instance's status/health, inputs, outputs, Pipelines, Routes, Functions, and traffic.

Health Endpoint

Query this endpoint on any instance to check the instance's health. (Details below.)

Types of Logs

LogStream provides the following log types, by originating process:

  • API Server Logs – These logs are emitted primarily by the API/main process. They correspond to the top-level cribl.log that shows up on the Diag page. These include telemetry/license-validation logs. Filesystem location: $CRIBL_HOME/log/cribl.log

  • Worker Process(es) Logs – These logs are emitted by all the Worker Processes, and are very common on single-instance deployments and Worker Nodes. Filesystem location: $CRIBL_HOME/log/worker/N/cribl.log

  • Worker Group Logs – These logs are emitted by all processes that help a Leader Node configure Worker Groups. Filesystem location: $CRIBL_HOME/log/group/GROUPNAME/cribl.log

LogStream rotates logs every 5 MB, keeping the most recent 5 logs. In a distributed deployment, all Workers forward their metrics to the Leader Node, which then consolidates them to provide a deployment-wide view.

Forward Logs and Metrics Externally

LogStream supports forwarding internal logs and metrics to your preferred external monitoring solution. To make internal data available to send out, go to Sources and enable the Cribl Internal Source.

This will send internal logs and metrics down through Routes and Pipelines, just like another data source. Both logs and metrics will have a field called source, set to the value cribl, which you can use in Routes' filters.

Note that the only logs supported here are Worker Process logs (see Types of Logs above). You can, however, use a Script Collector to listen for API Server or Worker Group events.

For recommendations about useful Cribl metrics to monitor, see Internal Metrics.

📘

CriblMetrics Override

The Disable field metrics setting – in global ⚙️ Settings (lower left) > System > General Settings > Limits ‑ applies only to metrics sent to the Leader Node. When the Cribl Internal Source is enabled, LogStream ignores this Disable field metrics setting, and full-fidelity data will flow down the Routes.

Search Internal Logs

LogStream exists because logs are great and wonderful things! Using its Monitoring > Logs page, you can search all LogStream's internal logs at once – from a single location, for both Leader and Worker Nodes. This enables you to query across all internal logs for strings of interest.

The labels on this screenshot highlight the key controls you can use (see the descriptions below):

Logs page (controls highlighted)Logs page (controls highlighted)

Logs page (controls highlighted)

  1. Log file selector: Choose the Node to view. In a Distributed Deployment, this list will be hierarchical, with Workers displayed inside their Leader.

  2. Fields selector: Click the Main | All | None toggles to quickly select or deselect multiple check boxes below.

  3. Fields: Select or deselect these check boxes to determine which columns are displayed in the Results pane at right. (The upper Main Fields group will contain data for every event; other fields might not display data for all events.)

  4. Time range selector: Select a standard or custom range of log data to display.

  5. Search box: To limit the displayed results, enter a JavaScript expression here. An expression must evaluate to truthy to return results. You can press Shift+Enter to insert a newline.

Typeahead assist is available for expression completion:

Click a field in any event to add it to a query:

Click other fields to append them to a query:

Shift+click to negate a field:

ℹ

To modify the depth of information that is originally input to the Logs page, see Logging Settings.

  1. Click the Search box's history arrow (right side) to retrieve recent queries:
  1. The Results pane displays most-recent events first. Each event's icon is color-coded to match the event's severity level.

Click individual log events to unwrap an expanded view of their fields:

Logging Settings

Through LogStream's global Settings, you can adjust the level (verbosity) of internal logging data processed, per logging channel. You can also redact fields in customized ways.

Change Logging Levels

Select global ⚙️ Settings (lower left) > System > Logging > Levels to open the Manage Logging Levels page. Here, you can:

  • Modify one channel by clicking its Level column. In the resulting drop-down, you can set a verbosity level ranging from error up to debug. (Top of composite screenshot below.)

  • Modify multiple channels by selecting their check boxes, then clicking the Change log level drop-down at the bottom of the page. (Bottom of composite screenshot below.) You can select all channels at once by clicking the top check box. You can search for channels at top right.

Manage Logging Levels pageManage Logging Levels page

Manage Logging Levels page

Change Logging Redactions

Select global ⚙️ Settings (lower left) > System > Logging > Redactions: to open the Redact Internal Log Fields page. Here, you can customize the redaction of sensitive, verbose, or just ugly data within LogStream's internal logs.

Redact Internal Log Fields pageRedact Internal Log Fields page

Redact Internal Log Fields page

It's easiest to understand this page's fields from bottom to top:

  • Default fields: LogStream always redacts these fields. You can't modify this list.
  • Additonal fields: Type or paste in the names of other fields you want to redact. Use a tab or hard return to confirm each entry.
  • Custom redact string: Unless this field is empty, it defines a literal string that will override LogStream's default redaction pattern, explained below.

Default Redact String

By default, LogStream transforms this page's selected fields by applying the following redaction pattern:

  • Echo the field value's first two characters.
  • Replace all intermediate characters with a literal ... ellipsis.
  • Echo the value's last two characters.

Anything you enter in the Custom redact string field will override this default ??...?? pattern.

Health Endpoint

Each LogStream instance exposes a health endpoint – typically used in conjunction with a Load Balancer – that you can use to make operational decisions.

Health Check Endpoint

Healthy Response

LogStream Version

curl http(s)://<host>:<port>/api/v1/health

{"status":"healthy"}

Through 2.4.3

curl http(s)://<host>:<port>/api/v1/health

{"status":"healthy","startTime":1617814717110} (see details below)

2.4.4 and later

Specifically, the health endpoint can return one of the following response codes:

200 – healthy.
400 – an auth token was provided, but does not match any provisioned token.
503 – server busy: too many concurrent connections (configurable).

Updated 7 days ago

Monitoring


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.