Cribl LogStream ā€“ Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF - v2.4.5

Monitoring

To get an operational view of a Cribl LogStream deployment, you can consult the following resources.

Monitoring Page

Select Monitoring from the top menu. This exposes information about traffic in and out of the system, as well as collection jobs and tasks. It tracks events, bytes, splits by data fields over time, and broader system metrics. Coverage is limited to the previous 24 hours. (Byte-related charts show the uncompressed size of processed data.)

Monitoring page

Dense displays are condensed to sparklines for legibility. Hover over the right edge to display Maximize buttons that you can click to zoom these up to detailed graphs.

Sparklines and fly-out

You can hover over an expanded graph fly-out to display further details.

Throughput details

Internal Logs and Metrics

Select Logs from the Monitoring submenu. LogStream's internal logs and internal metrics provide comprehensive information about an instance's status/health, inputs, outputs, Pipelines, Routes, Functions, and traffic.

Health Endpoint

Query this endpoint on any instance to check the instance's health. (Details below.)

Types of Logs

LogStream provides the following log types, by originating process:

  • API Server Logs ā€“ These logs are emitted primarily by the API/main process. TheyĀ correspond to the top-level cribl.log that shows up on the Diag page. These include telemetry/license-validation logs. Filesystem location: $CRIBL_HOME/log/cribl.log

  • Worker Process(es) Logs ā€“ These logs are emitted by all the Worker Processes, and are very common on single-instance deployments and Worker Nodes. Filesystem location: $CRIBL_HOME/log/worker/N/cribl.log

  • Worker Group Logs ā€“ These logs are emitted by all processes that help a Master Node configure Worker Groups. Filesystem location: $CRIBL_HOME/log/group/GROUPNAME/cribl.log

LogStream rotates logs every 5 MB, keeping the most recent 5 logs. In a distributed deployment, all Workers forward their metrics to the Master Node, which then consolidates them to provide a deployment-wide view.

Forward Logs and Metrics Externally

LogStream supports forwarding internal logs and metrics to your preferred external monitoring solution. To send out internal data, go to Data > Sources and enable the Cribl Internal Source.

This will send internal logs and metrics down through Routes and Pipelines, just like another data source. Both logs and metrics will have a field called source, set to the value cribl, which you can use in Routes' filters.

Note that the only logs supported here are Worker Process logs (see TypesĀ of Logs above). You can, however, use a ScriptĀ Collector to listen for APIĀ Server or WorkerĀ Group events.

For recommendations about useful Cribl metrics to monitor, see Internal Metrics.

šŸ“˜

CriblMetrics Override

The DisableĀ field metrics setting (in Settings > System > GeneralĀ Settings > Limits) applies only to metrics sent to the MasterĀ Node. When the CriblĀ Internal Source is enabled, LogStream ignores this DisableĀ field metrics setting, and full-fidelity data will flow down the Routes.

Search Internal Logs

LogStream exists because logs are great and wonderful things! Using its Monitoring > Logs page, you can search all LogStream's internal logs at once ā€“ from a single location, for both Master and Worker Nodes. This enables you to query across all internal logs for strings of interest.

The labels on this screenshot highlight the key controls you can use (see the descriptions below):

Logs page (controls highlighted)

  1. Log file selector: Choose the Node to view. In a Distributed Deployment, this list will be hierarchical, with Workers displayed inside their Master.

  2. Fields selector: Click the Main | All | None toggles to quickly select or deselect multiple check boxes below.

  3. Fields: Select or deselect these check boxes to determine which columns are displayed in the Results pane at right. (The upper Main Fields group will contain data for every event; other fields might not display data for all events.)

  4. Time range selector: Select a standard or custom range of log data to display.

  5. Search box: To limit the displayed results, enter a JavaScript expression here. An expression must evaluate to truthy to return results. You can press Shift+Enter to insert a newline.

Typeahead assist is available for expression completion:

Click a field in any event to add it to a query:

Click other fields to append them to a query:

Shift+click to negate a field:

ā„¹

To modify the depth of information that is originally input to the Logs page, see LoggingĀ Settings.

  1. Click the Search box's history arrow (right side) to retrieve recent queries:
  1. The Results pane displays most-recent events first. Each event's icon is color-coded to match the event's severity level.

Click individual log events to unwrap an expanded view of their fields:

Logging Settings

Through LogStream's System Settings, you can adjust the level (verbosity) of internal logging data processed, per logging channel. You can also redact fields in customized ways.

Change Logging Levels

Select Settings > System > Logging > Levels to open the Manage Logging Levels page. Here, you can:

  • Modify one channel by clicking its Level column. In the resulting drop-down, you can set a verbosity level ranging from error up to debug. (Top of composite screenshot below.)

  • Modify multiple channels by selecting their check boxes, then clicking the Change log level drop-down at the bottom of the page. (Bottom of composite screenshot below.) You can select all channels at once by clicking the top check box. You can search for channels at top right.

Manage Logging Levels page

Change Logging Redactions

Select Settings > System > Logging > Redactions: to open the Redact Internal Log Fields page. Here, you can customize the redaction of sensitive, verbose, or just ugly data within LogStream's internal logs.

Redact Internal Log Fields page

It's easiest to understand this page's fields from bottom to top:

  • Default fields: LogStream always redacts these fields. You can't modify this list.
  • Additonal fields: Type or paste in the names of other fields you want to redact. Use a tab or hard return to confirm each entry.
  • Custom redact string: Unless this field is empty, it defines a literal string that will override LogStream's default redaction pattern, explained below.

Default Redact String

By default, LogStream transforms this page's selected fields by applying the following redaction pattern:

  • Echo the field value's first two characters.
  • Replace all intermediate characters with a literal ... ellipsis.
  • Echo the value's last two characters.

Anything you enter in the Custom redact string field will override this default ??...?? pattern.

Health Endpoint

Each LogStream instance exposes a health endpoint ā€“ typically used in conjunction with a LoadĀ Balancer ā€“ that you can use to make operational decisions.

Health Check Endpoint

Healthy Response

LogStream Version

curl http(s)://<host>:<port>/api/v1/health

{"status":"healthy"}

Through 2.4.3

curl http(s)://<host>:<port>/api/v1/health

{"status":"healthy","startTime":1617814717110} (see details below)

2.4.4 and later

Specifically, the health endpoint can return one of the following response codes:

200 ā€“ healthy.
400 ā€“ an auth token was provided, but does not match any provisioned token.
503 ā€“ server busy: too many concurrent connections (configurable).

Updated 3 days ago

Monitoring


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.