To get an operational view of a Cribl LogStream deployment, you can consult the following resources.
Select Monitoring from the left nav (distributed deployments) or top nav (single-instance deployments). The resulting Monitoring page displays information about traffic in and out of the system, as well as collection jobs and tasks. It tracks events, bytes, splits by data fields over time, and broader system metrics.
The initial view (below) shows aggregate data for all Groups and all Workers. You can use the drop-downs at the upper right to isolate individual Groups, or individual Workers. Here, you can also change the display's granularity from the default
15 min. Coverage is limited to the previous 24 hours (this maximum is not configurable).
The displayed CPU Load Average is an average per Worker Process, updated at 1‑minute granularity. (It is not an average for the Worker Node as a whole.)
Byte-related charts show the uncompressed size of processed data. Bytes in/out are measured based on the size of
_raw (meaning that metrics events will reflect
0 bytes processed, because they include no
Dense displays are condensed to sparklines for legibility. Hover over the right edge to display Maximize buttons that you can click to zoom these up to detailed graphs.
You can hover over an expanded graph fly-out to display further details.
From the Monitoring page's top nav, open the Data submenu to isolate throughput for any of the following:
- Data Fields
From the Monitoring page's top nav, open the System submenu to isolate throughput for any of the following:
- Queues (see Persistent Queues)
- Jobs (and tasks in-flight, see Collector Sources)
- Job Inspector
Select System > Licensing from the Monitoring page's top nav to check your licenses' expiration dates, daily data throughput quotas, and daily and 90-day trailing daily throughput.
Select System > Job Inspector from the Monitoring page's top nav to view and manage pending, in-flight, and completed collection jobs and their tasks. For details about the resulting page, see Monitoring and Inspecting Collection Jobs.
Select Flows from the Monitoring page's top nav or ••• overflow menu to see a graphical, left-to-right visualization of data flow through your LogStream deployment.
Select Logs from the Monitoring page's top nav. LogStream's internal logs and internal metrics provide comprehensive information about an instance's status/health, inputs, outputs, Pipelines, Routes, Functions, and traffic.
Query this endpoint on any instance to check the instance's health. (Details below.)
LogStream provides the following log types, by originating process:
API Server Logs – These logs are emitted primarily by the
API/mainprocess. They correspond to the top-level
cribl.logthat shows up on the Diag page. These include telemetry/license-validation logs. Filesystem location:
Worker Process(es) Logs – These logs are emitted by all the Worker Processes, and are very common on single-instance deployments and Worker Nodes. Filesystem location:
Worker Group Logs – These logs are emitted by all processes that help a Leader Node configure Worker Groups. Filesystem location:
LogStream rotates logs every 5 MB, keeping the most recent 5 logs. In a distributed deployment, all Workers forward their metrics to the Leader Node, which then consolidates them to provide a deployment-wide view.
LogStream supports forwarding internal logs and metrics to your preferred external monitoring solution. To make internal data available to send out, go to Sources and enable the Cribl Internal Source.
This will send internal logs and metrics down through Routes and Pipelines, just like another data source. Both logs and metrics will have a field called
source, set to the value
cribl, which you can use in Routes' filters.
For recommendations about useful Cribl metrics to monitor, see Internal Metrics.
The Disable field metrics setting – in global ⚙️ Settings (lower left) > System > General Settings > Limits ‑ applies only to metrics sent to the Leader Node. When the Cribl Internal Source is enabled, LogStream ignores this Disable field metrics setting, and full-fidelity data will flow down the Routes.
LogStream exists because logs are great and wonderful things! Using its Monitoring > Logs page, you can search all LogStream's internal logs at once – from a single location, for both Leader and Worker Nodes. This enables you to query across all internal logs for strings of interest.
The labels on this screenshot highlight the key controls you can use (see the descriptions below):
Log file selector: Choose the Node to view. In a Distributed Deployment, this list will be hierarchical, with Workers displayed inside their Leader.
Fields selector: Click the Main | All | None toggles to quickly select or deselect multiple check boxes below.
Fields: Select or deselect these check boxes to determine which columns are displayed in the Results pane at right. (The upper Main Fields group will contain data for every event; other fields might not display data for all events.)
Time range selector: Select a standard or custom range of log data to display.
truthyto return results. You can press Shift+Enter to insert a newline.
Typeahead assist is available for expression completion:
Click a field in any event to add it to a query:
Click other fields to append them to a query:
Shift+click to negate a field:
To modify the depth of information that is originally input to the Logs page, see Logging Settings.
- Click the Search box's history arrow (right side) to retrieve recent queries:
- The Results pane displays most-recent events first. Each event's icon is color-coded to match the event's severity level.
Click individual log events to unwrap an expanded view of their fields:
Through LogStream's global Settings, you can adjust the level (verbosity) of internal logging data processed, per logging channel. You can also redact fields in customized ways.
Select global ⚙️ Settings (lower left) > System > Logging > Levels to open the Manage Logging Levels page. Here, you can:
Modify one channel by clicking its Level column. In the resulting drop-down, you can set a verbosity level ranging from error up to debug. (Top of composite screenshot below.)
Modify multiple channels by selecting their check boxes, then clicking the Change log level drop-down at the bottom of the page. (Bottom of composite screenshot below.) You can select all channels at once by clicking the top check box. You can search for channels at top right.
Select global ⚙️ Settings (lower left) > System > Logging > Redactions: to open the Redact Internal Log Fields page. Here, you can customize the redaction of sensitive, verbose, or just ugly data within LogStream's internal logs.
It's easiest to understand this page's fields from bottom to top:
- Default fields: LogStream always redacts these fields. You can't modify this list.
- Additonal fields: Type or paste in the names of other fields you want to redact. Use a tab or hard return to confirm each entry.
- Custom redact string: Unless this field is empty, it defines a literal string that will override LogStream's default redaction pattern, explained below.
By default, LogStream transforms this page's selected fields by applying the following redaction pattern:
- Echo the field value's first two characters.
- Replace all intermediate characters with a literal
- Echo the value's last two characters.
Anything you enter in the Custom redact string field will override this default
Each LogStream instance exposes a
health endpoint – typically used in conjunction with a Load Balancer – that you can use to make operational decisions.
Health Check Endpoint
2.4.4 and later
health endpoint can return one of the following response codes:
200 – healthy.
400 – an auth token was provided, but does not match any provisioned token.
503 – server busy: too many concurrent connections (configurable).
Updated 7 days ago