Cribl Stream writes internal logs that record its own processes, API activity, configuration changes, and data handling. These logs provide valuable insights into system health, performance, and potential issues.

Distributed deployments write logs for the Leader, each Worker Node, and supporting services. Single-instance deployments write a smaller set of the same logs.

The sections on this page list where Cribl Stream writes an internal log. The following pages include an example event and field details for each log:

• cribl.log
• access.log
• audit.log
• metrics.log
• notifications.log
• ui-access.log

Several logs listed on this page are exposed only in customer-managed (on-prem) deployments. In Cribl.Cloud, Leaders support Cribl Stream Worker Node logs on hybrid Workers.

However, Members who have the Admin Permission on Cribl Search can use it to search the cribl_internal_logs Dataset for additional details about the Leader.

Leader Node Logs (Distributed) ​

The API/main process emits the following logs into the Leader Node’s $CRIBL_HOME/log/ directory.

The API/main process emits the following service logs into the Leader Node’s $CRIBL_HOME/log/service/ directory. Each service includes a cribl.log file that logs the service’s internal telemetry and an access.log file that logs which API calls the service has handled.

The Config Helper process for each Worker Group/Fleet emits the following log in $CRIBL_HOME/log/group/GROUPNAME.

Worker Node Logs (Distributed) ​

The API Process emits the following log in $CRIBL_HOME/log/.

Each Worker Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID.

For convenience, the UI aggregates the Worker/Edge Node Process logs as follows.

In Cribl Stream, the logs listed above are currently available only on customer-managed hybrid Workers. The single-instance logs listed below are not relevant to Cribl.Cloud.

TCP load balancing ​

If TCP load balancing is enabled in at least one supported Source (such as Syslog, TCP JSON, or Cribl TCP), the Worker also emits the following log to $CRIBL_HOME/log/worker/LB/.

Events in the load balancer copy use different channel values than the Worker Process cribl.log for the same Source. On the load balancer log, search with channel == "<source-id>_lb". On the Worker Process log, use channel == "input:<source-id>". For the built-in in_tcp_json Source, use channel == "in_tcp_json_lb" and channel == "input:in_tcp_json", respectively.

Single-Instance Logs ​

The API/main process emits the same logs as it does for a Distributed deployment, in $CRIBL_HOME/log/:

• cribl.log
• access.log
• audit.log
• notifications.log
• ui-access.log

Each Worker/Edge Node Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID.

Outpost Logs ​

The Outpost service emits the following logs into the $CRIBL_HOME/log/service/outpost/N/ directory, where N is the Outpost Node Process ID:

cribl_stderr and Per-Process Stderr Logs ​

In rare cases, the Node.js backend can encounter a fatal error, such as an out-of-memory condition, that prevents normal logging. To help with troubleshooting, Cribl Stream writes details about these failures to stderr log files, including:

• cribl_stderr.log at the Worker Node level.
• Per-process stderr log files, for example cribl_stderr_<PID>.log, for specific API or service processes.

Key points:

• These stderr logs are intended for Cribl Support and are not exposed in the UI.
• They follow the same size-based rotation policy as other internal logs. See Log Rotation and Retention.
• Some stderr logs include prepended UTC timestamps to make it easier to correlate failures with other system activity, such as events in cribl.log.

Log Rotation and Retention ​

Cribl Stream writes multiple internal log files for Leaders, Worker Nodes, and supporting services. To prevent any single log file from consuming excessive disk space, Cribl Stream automatically rotates these files based on size rather than time.

By default, Cribl Stream:

• Rotates each log file when it reaches 5 MB.
• Keeps the five most recent rotated files per log.

This rotation policy applies to all internal log files, including:

• Leader and Worker Node API logs in $CRIBL_HOME/log/, such as:
  • cribl.log
  • access.log
  • audit.log
  • notifications.log
  • ui-access.log
• Service logs in $CRIBL_HOME/log/service/....
• Worker Group logs in $CRIBL_HOME/log/group/<GROUPNAME>/.
• Worker Node logs in $CRIBL_HOME/log/worker/<N>/..., such as metrics.log.
• Stderr-based logs, such as cribl_stderr.log.

Verbose logging levels (for example, debug or silly) and high system activity can cause logs to reach the 5 MB threshold more quickly.

For guidance on forwarding internal logs and metrics to external observability platforms, see Forward Logs and Metrics Externally.

Use Logs and Datasets for Troubleshooting ​

Troubleshooting health, API, or data-handling problems often requires searching across internal log files. In Cribl Search, you can search built-in Datasets that include multiple logs instead of opening each file separately in the Cribl UI. See Map Log Files to Built-in Datasets to match common symptoms to a log file and built-in Dataset.

To display and export internal logs for a single Leader or Worker Node in the product UI, see Search Internal Logs. You can also forward logs externally.

Built-in Datasets do not include Collector job artifacts (job.log, task.log, and related files under state/jobs/). For those logs, see Collector Job Logs.

Map Log Files to Built-in Datasets ​

The following table identifies the internal logs to investigate for common issues and the built-in Datasets that include the log file. The table references these built-in Datasets:

• cribl_internal_logs: A federated search Dataset that includes internal .log files under $CRIBL_LOG_DIR on the Leader. The cribl_leader Dataset Provider reads these Leader-local files. It does not search log directories on remote nodes.
• cribl_logs: A Lake Dataset that includes audit and access logs from your Cribl.Cloud Leader. Query from Cribl Search.
• cribl_metrics: A Lake Dataset that includes internal metrics from your Cribl.Cloud Leader. Query from Cribl Search.

Query Internal Logs in Built-in Datasets ​

To find events from an internal log, search the built-in Dataset that includes it in Cribl Search.

Members must have the Admin Permission on Cribl Search to query cribl_internal_logs. To query cribl_logs or cribl_metrics, Members need the Read Only Permission on Cribl Lake Datasets in addition to the Admin Permission on Cribl Search.

The following search examples target internal log paths and event fields that are useful for troubleshooting. Adjust paths and filters for the log you are investigating. For log query syntax details, see Build a Log Search.

Find API requests that returned an error status:

dataset="cribl_internal_logs" source="*access.log" status>=400

Show Config Helper events for all Worker Groups:

dataset="cribl_internal_logs" source="*group/*/cribl.log"

Find commits and deploys:

dataset="cribl_internal_logs" source="*audit.log"
| where action in ("commit", "deploy")

Find events for one Notification rule:

dataset="cribl_internal_logs" source="*notifications.log"
| where channel == "<rule-id>"

Find API errors handled by a Leader service:

dataset="cribl_internal_logs" source="*service/*/access.log" status>=400

Find Cribl.Cloud Leader API requests that returned an error status:

dataset="cribl_logs" data_source="*access.log" status>=400