On This Page

Home / Stream/ Integrations/ Sources/ System/Journal Files Source

Journal Files Source

The Journal Files Source collects data from systemd’s centralized logging, which is called the journald service. This service is a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes messages from kernel, boot, syslog, or other services. This Source is currently configured to configured to collect data from the user journal, as system is privileged. This Source is disabled by default.

Type: System | TLS Support: N/A | Event Breaker Support: No

The Journal Files Source is available in Cribl.Cloud on customer-managed hybrid Workers, but not on Cribl-managed Workers in Cribl.Cloud.

Discovering and Filtering Journal Files to Monitor
​

To determine all the names of all journals to read, the Journal Files Source runs a discovery procedure at a configurable Polling interval. The Source then applies an Allowed list to filter the initial list down into its final form. Then, for each journal on the list, the Source compares current state with previously stored state. This comparison determines whether the Journal Files Source will actually stream a given journal for a given polling interval. The Source then runs the files through the Filter Rules which allows you to set additional criteria.

Cribl Stream ships with a single, disabled instance of the Journal Files Source with a default configuration. The search path is configured to point to $CRIBL_EDGE_FS_ROOT/var/log/journal/$MACHINE_ID and defaults to streaming the primary system.journal it finds there. $MACHINE_ID is an environment variable that you can insert into the Source’s search path. If you use the environment variable but don’t assign a value, the Source will look up the local host machine id from /etc/machine-id and use that value.

Currently, this Source does not support all of the compression options that the journald service supports. The parser this Source uses to read the files can handle LZ4 and ZSTD compression. When the parser encounters compressed fields it can’t unpack, an error message emits in the logs. Below is an example of the error message:

Error reading journald event, dropping offset=31678152 evt="Mar 02 21:29:54 goats-xps15 multipassd: undefined"

Also, this Source does not support the new COMPACT binary file format. The logs will contain an error message when the parser encounters this type of file.

Configure a Journal Files Source
​

  1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
    • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Source and select the Source you want from the list, choosing either Select Existing or Add New.
    • To configure via the Routes, select Data > Sources (Stream) or More > Sources (Edge). Select the Source you want. Next, select Add Source.
  2. In the New Source modal, configure the following under General Settings:
    • Enabled: Toggle on to enable the Source.
    • Input ID: Enter a unique name to identify this Source configuration. If you clone this Source, Cribl Stream will add -CLONE to the original Input ID.
    • Description: Optionally, enter a description.
    • Search path: Specify the directory to search journals on. Accepts environment variables. The default value for this field varies depending on the product:
    • In Cribl Edge, it defaults to $CRIBL_EDGE_FS_ROOT/var/log/journal/$MACHINE_ID.
    • In Cribl Stream, it defaults to /var/log/journal/$MACHINE_ID.
    • Journal allowlist: The full path of each discovered journal is matched against this wildcard list. Defaults to system.
  3. Next, you can configure the following Optional Settings:
    • Polling interval: How often, in seconds, to collect metrics. If not specified, defaults to 10 seconds.
    • Filter Rules: Optionally, specify which journal objects to allow. Events are generated if all the rules’ expressions evaluate to true, or if no rules are specified.
    • Filter expression: JavaScript expression to filter Journal objects. Returns true to include it. For details, see Filter Expressions below.
    • Description: Optional description of the rule.
    • Current boot only: Skip events that are not part of the current boot session.
    • Age duration limit: Optionally, specify a maximum age of journal objects to stream. This Source will filter events with timestamps newer than the configured duration. Enter a duration in a format like 30s, 4h, 3d, or 1w. Defaults to an empty field, which applies no age filters.
    • Tags: Optionally, add tags that you can use to filter and group Sources in Cribl Stream’s UI. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.
  4. Optionally, you can adjust the Processing and Advanced settings, or Connected Destinations outlined in the sections below.
  5. Select Save, then Commit & Deploy.

Filter Expressions
​

The default Filter expression is severity <= 4 with a Description of Allow events having 'emergency', 'alert', 'critical', 'error', or 'warning' priority.

The screenshot below shows other examples you can use:
Filter expressions
Filter expressions

Processing Settings
​

Fields
​

In this section, you can add Fields to each event, using Eval-like functionality.

Name: Field name.

Value: JavaScript expression to compute field’s value, enclosed in quotes or backticks. (Can evaluate to a constant.)

Pre-Processing
​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline or Pack to process data from this input before the data is sent through the Routes.

Advanced Settings {advanced-tab}
​

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Connected Destinations
​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

In Cribl Edge, this Source defaults to QuickConnect. While, in Cribl Stream, it defaults to Send to Routes.