Cribl Edge generates internal application logs that monitor its own operations and health. They provide valuable insights into the system’s behavior, performance, and potential issues.

Distributed deployments emit a larger set of logs than single-instance deployments. We’ll describe the distributed set first.

You can display and export all internal logs by selecting Monitoring in the sidebar, then selecting Logs (Stream) or selecting Logs in the sidebar (Edge). Logs’ persistence depends on event volume, not time - for details, see Log Rotation and Retention.

Several logs listed on this page are exposed only in customer-managed (on-prem) deployments. In Cribl.Cloud, Leaders support Cribl Edge Worker Node logs on hybrid Workers.

However, Organization Members who have the Admin Permission on Cribl Search can use that product to search the cribl_internal_logs Dataset for additional details about the Leader and its Cribl-managed Workers.

Leader Node Logs (Distributed)

​

The API/main process emits the following logs into the Leader Node’s $CRIBL_HOME/log/ directory.

The API/main process emits the following service logs into the Leader Node’s $CRIBL_HOME/log/service/ directory. Each service includes a cribl.log file that logs the service’s internal telemetry and an access.log file that logs which API calls the service has handled.

The Config Helper process for each Worker Group/Fleet emits the following log in $CRIBL_HOME/log/group/GROUPNAME.

Outpost Logs

​

The Outpost service emits the following logs into the $CRIBL_HOME/log/service/outpost/N/ directory, where N is the Outpost Node Process ID:

Edge Node Logs (Distributed)

​

The API Process emits the following log in $CRIBL_HOME/log/.

Each Worker Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID. (The metrics.log file is written only when HTTP-based Destinations exist.)

If TCP load balancing is enabled in at least one supported Source (such as Syslog, TCP JSON, or Cribl TCP), the Worker Process also emits the following log to $CRIBL_HOME/log/worker/LB:

Load balancing logs use different channel names to indicate the Source they come from than Worker Process logs. To search by channel name, use channel == "<source-id>_lb" instead of channel == "input:<source-id>". For example, for the built-in TCP JSON Source, enter channel == "in_tcp_json_lb" in the search box instead of channel == "input:in_tcp_json".

For convenience, the UI aggregates the Worker/Edge Node Process logs as follows.

In Cribl Edge, the logs listed above are currently available only on customer-managed hybrid Workers. The single-instance logs listed below are not relevant to Cribl.Cloud.

Single-Instance Logs

​

The API/main process emits the same logs as it does for a Distributed deployment, in$CRIBL_HOME/log/:

• cribl.log
• access.log
• audit.log
• notifications.log
• ui-access.log

Each Worker/Edge Node Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID. (The metrics.log file is written only when HTTP-based Destinations exist.)

_raw stats Event Fields

​

Each Worker/Edge Node Process logs this information at a 1-minute frequency. So each event’s scope covers only that Worker/Edge Node Process, over a 1-minute time span defined by the startTime and endTime fields.

Sample Event

​

{"time":"2022-11-17T16:54:05.349Z","cid":"w0","channel":"server","level":"info","message":"_raw stats","inEvents":307965,"outEvents":495848,"inBytes":52756162,"outBytes":83028013,"starttime":1668703980,"endtime":1668704040,"activeCxn":0,"openCxn":0,"closeCxn":0,"rejectCxn":0,"abortCxn":0,"pqInEvents":62000,"pqOutEvents":114591,"pqInBytes":12163896,"pqOutBytes":22481509,"pqTotalBytes":480467058,"droppedEvents":0,"tasksStarted":6,"tasksCompleted":6,"activeEP":9,"blockedEP":0,"cpuPerc":101.09,"eluPerc":97.81,"mem":{"heap":277,"heapTotal":287,"ext":46,"rss":453,"buffers":0}}

Field Descriptions

​

Log Rotation and Retention

​

Cribl Edge writes multiple internal log files for Leaders, Edge Nodes, and supporting services. To prevent any single log file from consuming excessive disk space, Cribl Edge automatically rotates these files based on size.

By default, Cribl Edge:

• Rotates each log file when it reaches 5 MB.
• Keeps the five most recent rotated files per log.

This rotation policy applies to all internal log files, including:

• Leader/Node logs in $CRIBL_HOME/log/, such as:
  • cribl.log
  • access.log
  • audit.log
  • notifications.log
  • ui-access.log
• Service logs in $CRIBL_HOME/log/service/....
• Fleet logs in $CRIBL_HOME/log/group/<GROUPNAME>/cribl.log.
• Edge Node logs in $CRIBL_HOME/log/worker/<N>/..., such as metrics.log.
• Stderr-based logs such as cribl_stderr.log.

Verbose logging levels (for example, debug or silly) and high system activity can cause logs to reach the 5 MB threshold more quickly.

For guidance on forwarding internal logs and metrics to external observability platforms, see Forward Logs and Metrics Externally.

cribl_stderr and per-process stderr logs

​

In rare cases, Cribl Edge’s Node.js backend can encounter a fatal error (such as an out-of-memory condition) that prevents normal logging. To help with troubleshooting, Cribl Edge writes details about these failures to special stderr log files, including:

• cribl_stderr.log at the node level.
• Per-process stderr log files (for example, cribl_stderr_<PID>.log) written for specific processes such as API or service processes.

Key points:

• These stderr logs are intended for Cribl Support and are not exposed in the UI.
• They follow the same size-based rotation policy as other internal logs (rotate at 5 MB, keep five rotated copies per file).
• Some stderr logs include prepended UTC timestamps to make it easier to correlate failures with other system activity.