These docs are for Cribl Edge 4.12 and are no longer actively maintained.
See the latest version (4.13).
Linux System Metrics Details
Events generated by the System Metrics Source have metrics metadata to designate dimension and metric fields. The host field contains the hostname, and is included as a dimension in all of them. The collectors include:
In the Source’s configuration modal, you can set the level of detail for each type of metrics:
- Basic enables minimal metrics, averaged or aggregated.
- All enables full, detailed metrics, specified for individual CPUs, interfaces, and so on.
- Custom displays sub-menus and buttons from which you can choose a level of detail (Basic, All, Custom, or Disabled) for each type of event.
- Disabled means that no metrics will be generated.
Basic and Custom have different meanings depending on event type and will be described under each section below.
The tables outline the metrics emitted for each mode (Basic or Custom) and where applicable, the dimensions (to indicate where the metrics are coming from).
System
With System Metrics enabled, Cribl Edge captures CPU load averages, uptime, and count. The Custom option allows you to include process metrics that reflect the numbers of processes in various states.
Metrics for the overall system include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
node_uname_info | Labeled system information as provided by the uname system call. | Counter | release, sysname, version | Basic |
node_cpu_count | The number of CPU cores. | Gauge | release, sysname, version | Basic |
node_uptime_seconds | System uptime in seconds. | Counter | N/A | Basic |
node_boot_time_seconds | Node boot time in Unix time. | Counter | N/A | Basic |
node_time_seconds | System time in seconds. | Counter | N/A | Basic |
node_load1 | 1m load average. | Gauge | N/A | Basic |
node_load5 | 5m load average. | Gauge | N/A | Basic |
node_load15 | 15m load average. | Gauge | N/A | Basic |
node_open_fds | Open file descriptors | Counter | N/A | Basic |
node_processes_state_all | Total number of processes in different states. | Gauge | state | Basic |
node_processes_threads | Allocated threads in system. | Gauge | state | Basic |
node_procs_blocked | Number of processes blocked waiting for I/O to complete. | Gauge | state | Basic |
node_procs_running | Number of processes in runnable state. | Gauge | state | Basic |
node_processes_state | Number of processes in each state. | Gauge | state | Custom: Process metrics |
CPU
Cribl Edge captures active, user, system, idle, and iowait percentages over all CPUs, with options to add per-CPU metrics and raw time counters for each state.
Metrics for CPUs include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
node_cpu_percent_active_all | Percent all the CPUs spent in activity. | Gauge | N/A | Basic |
node_cpu_seconds_active_all_total | Seconds all the CPUs spent in activity (excluding idle and wait). | Counter | N/A | Custom: CPU time metrics |
node_cpu_seconds_active_total | Seconds each CPU spent in activity (excluding idle and wait). | Counter | cpu | Custom: CPU time metrics |
node_cpu_seconds_all_total | Seconds for all CPUs usage. | Counter | mode | Custom: CPU time metrics |
node_cpu_seconds_total | Seconds for each CPU’s usage. | Counter | cpu, mode | Custom: CPU time metrics |
node_cpu_percent_active | Percent each CPU spent in activity. | Gauge | cpu | Custom: Per CPU or Detailed metrics |
node_cpu_percent_all | Percent CPU usage for all. | Gauge | mode,user | Custom: Detailed metrics |
node_cpu_percent | Percent CPU usage for each CPU. | Gauge | cpu,mode,user | Custom: Per CPU or Detailed metrics |
- The
Per CPUoption adds metrics with thecpudimension. - The
Detailed metricsoption adds metrics withmodedimension set to:irq,softirq,steal,guest,guest_nice, andnice.
Memory
With System Metrics enabled, Cribl Edge captures memory metrics including total, used, available, swap_free, and swap_total, with the option to toggle all memory states.
Metrics for memory include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
node_memory_MemTotal_bytes | Memory information field MemTotal_bytes. | Gauge | N/A | Basic |
node_memory_Used_bytes | Used memory in bytes. | Gauge | N/A | Basic |
node_memory_Used_percent | Percent used memory. | Gauge | N/A | Basic |
node_memory_MemAvailable_bytes | Memory information field MemAvailable_bytes. | Gauge | N/A | Basic |
node_memory_MemAvailable_percent | Percent memory available. | Gauge | N/A | Basic |
node_memory_SwapFree_bytes | Memory information field SwapFree_bytes. | Gauge | N/A | Basic |
node_memory_SwapTotal_bytes | Memory information field SwapTotal_bytes. | Gauge | N/A | Basic |
node_vmstat_oom_kill | /proc/vmstat information field oom_kill. | Gauge | N/A | Basic |
node_vmstat_pgfault | /proc/vmstat information field pgfault. | Gauge | N/A | Basic |
node_vmstat_pgmajfault | /proc/vmstat information field pgmajfault. | Gauge | N/A | Basic |
node_vmstat_pgpgin | /proc/vmstat information field pgpgin. | Gauge | N/A | Basic |
node_vmstat_pgpgout | /proc/vmstat information field pgpgout. | Gauge | N/A | Basic |
node_vmstat_pswpin | /proc/vmstat information field pswpin. | Gauge | N/A | Basic |
node_vmstat_pswpout | /proc/vmstat information field pswpout. | Gauge | N/A | Basic |
node_memory_Active_bytes | Memory information field Active_bytes. | Gauge | N/A | Custom: Detailed metrics |
node_memory_Buffers_bytes | Memory information field Buffers_bytes. | Gauge | N/A | Custom: Detailed metrics |
node_memory_Cached_bytes | Memory information field Cached_bytes. | Gauge | N/A | Custom: Detailed metrics |
node_memory_MemFree_bytes | Memory information field MemFree_bytes. | Gauge | N/A | Custom: Detailed metrics |
node_memory_SwapCached_bytes | Memory information field SwapCached_bytes. | Gauge | N/A | Custom: Detailed metrics |
Network
With System Metrics enabled, Cribl Edge captures bytes, packets, errors, and network connections over all interfaces. The Custom option allows you to filter interfaces, and to decide whether to select per-interface metrics and generate protocol metrics.
Metrics for networks include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
node_network_receive_bytes_all_total | Network device statistic receive_bytes. | Counter | N/A | Basic |
node_network_receive_errs_all_total | Network device statistic receive_errs. | Counter | N/A | Basic |
node_network_receive_packets_all_total | Network device statistic receive_packets. | Counter | N/A | Basic |
node_network_transmit_bytes_all_total | Network device statistic transmit_bytes. | Counter | N/A | Basic |
node_network_transmit_errs_all_total | Network device statistic transmit_errs. | Counter | N/A | Basic |
node_network_transmit_packets_all_total | Network device statistic transmit_packets. | Counter | N/A | Basic |
node_socket_tcp_established_total | TCP established connections. | Counter | N/A | Basic |
node_network_receive_bytes_total | Network device statistic receive_bytes per interface. | Counter | device | Custom: Per Interface |
node_network_receive_errs_total | Network device statistic receive_errs per interface. | Counter | device | Custom: Per Interface |
node_network_receive_packets_total | Network device statistic receive_packets per interface. | Counter | device | Custom: Per Interface |
node_network_transmit_bytes_total | Network device statistic transmit_bytes per interface. | Counter | device | Custom: Per Interface |
node_network_transmit_errs_total | Network device statistic transmit_errs per interface. | Counter | device | Custom: Per Interface |
node_network_transmit_packets_total | Network device statistic transmit_packets per interface. | Counter | device | Custom: Per Interface |
node_network_receive_drop_all_total | Network device statistic receive_drop. | Counter | N/A | Custom: Detailed Metrics |
node_network_receive_drop_total | Network device statistic receive_drop per interface. | Counter | device | Custom: Detailed Metrics |
node_network_transmit_drop_all_total | Network device statistic transmit_drop. | Counter | N/A | Custom: Detailed Metrics |
node_network_transmit_drop_total | Network device statistic transmit_drop per interface. | Counter | device | Custom: Detailed Metrics |
node_socket_tcp_syn_sent_total | TCP sent packets total. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_syn_recv_total | TCP received packets total. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_fin_wait1_total | Total connections waiting for termination request from remote TCP. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_fin_wait2_total | Active TCP connections to be shut down. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_time_wait_total | Length of time to pass to be sure the remote TCP received the acknowledgement to terminate. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_close_total | Total TCP sockets with closed connections. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_last_ack_total | Total TCP sockets in state before the TCP connection is closed. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_listen_total | Total TCP sockets waiting for a connection request from any remote TCP/port. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_closing_total | Total TCP sockets waiting for connection termination request acknowledgement. | Counter | N/A | Custom: Detailed Metrics |
node_socket_tcp_none_total | Number of TCP sockets with no connections. | Counter | N/A | Custom: Detailed Metrics |
node_socket_udp_total | Number of UDP sockets in use. | Counter | N/A | Custom: Detailed Metrics |
Disk
With System Metrics enabled, Cribl Edge captures disk-used metrics – in percent, bytes read and written, and read and write operations – over all mounted disks. The Custom option allows you to filter devices, mountpoint, and filesystem type, and to decide whether to select per-device metrics and generate detailed metrics.
Metrics for Disk include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
node_disk_reads_completed_all_total | Total number of reads completed successfully. | Counter | N/A | Basic |
node_disk_read_bytes_all_total | Total number of bytes read successfully. | Counter | N/A | Basic |
node_disk_writes_completed_all_total | Network device statistic receive_packets. | Counter | N/A | Basic |
node_disk_written_bytes_all_total | Total number of bytes written successfully. | Counter | N/A | Basic |
node_filesystem_size_bytes_all | Filesystem size in bytes. | Gauge | N/A | Basic |
node_filesystem_avail_bytes_all | Filesystem space available to non-root users in bytes. | Gauge | N/A | Basic |
node_filesystem_used_bytes_all | Filesystem used space in bytes. | Gauge | N/A | Basic |
node_filesystem_used_percent_all | Percent filesystem used space. | Gauge | N/A | Basic |
node_filesystem_files_free_all | Filesystem free file nodes. | Gauge | N/A | Basic |
node_filesystem_files_used_all | Filesystem total used file nodes. | Gauge | N/A | Basic |
node_filesystem_files_used_percent_all | Percent Filesystem used in all disks. | Gauge | N/A | Basic |
node_filesystem_size_bytes_all | Filesystem size in bytes. | Gauge | N/A | Basic |
node_filesystem_used_bytes | Filesystem used space in bytes per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_used_percent | Percent Filesystem used per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_reads_completed_total | Total number of reads completed successfully per disk. | Counter | device | Custom: Per device metrics |
node_disk_read_bytes_total | Total number of bytes read successfully per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics |
node_disk_writes_completed_total | Network device statistic receive_packets per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics |
node_disk_written_bytes_total | Total number of bytes written successfully per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics |
node_disk_discards_completed_all_total | Total number of discards completed successfully. | Counter | N/A | Custom: Detailed Metrics |
node_disk_discards_completed_total | Total number of discards completed successfully per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_discards_merged_all_total | Total number of discards merged. | Counter | N/A | Custom:Detailed metrics |
node_disk_read_time_seconds_total | Total number of seconds spent by all reads per disk. | Counter | device, fstype, mountpoint | Custom:Detailed metrics |
node_disk_write_time_seconds_all_total | Total number of seconds spent by all writes. | Counter | N/A | Custom: Detailed Metrics |
node_disk_read_time_seconds_all_total | Total number of seconds spent by all reads. | Counter | N/A | Custom: Detailed Metrics |
node_disk_write_time_seconds_total | Total number of seconds spent by all writes per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_reads_merged_all_total | Total number of reads merged. | Counter | N/A | Custom: Detailed Metrics |
node_disk_writes_merged_all_total | Total number of writes merged. | Counter | N/A | Custom: Detailed metrics |
node_disk_reads_merged_total | Total number of reads merged per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_writes_merged_total | Total number of writes merged per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_discards_merged_total | Total number of discards merged per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_io_time_seconds_all_total | Total seconds spent doing I/Os. | Counter | N/A | Custom: Detailed Metrics |
node_disk_io_time_seconds_total | Total seconds spent doing I/Os per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_io_now_all | The number of I/Os currently in progress. | Gauge | N/A | Custom: Detailed Metrics |
node_disk_io_now | The number of I/Os currently in progress per disk. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_disk_io_time_weighted_seconds_all_total | Weighted number of seconds spent doing I/Os per device. | Counter | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_size_bytes | Filesystem size in bytes per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_avail_bytes | Filesystem space available to non-root users in bytes per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_files_free | Filesystem free file nodes per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_files_used_percent | Percent Filesystem used per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
node_filesystem_size_bytes | Filesystem size in bytes per disk. | Gauge | device, fstype, mountpoint | Custom: Per device metrics or Detailed metrics |
Container
With System Metrics enabled, Cribl Edge generates Docker information with CPU, memory, network, and disk metrics for running containers. Optionally, you can customize which containers to generate metrics from.
Metrics for Container include the following:
| Name | Description | Type | Dimensions | Mode |
|---|---|---|---|---|
container_start_time_seconds | UNIX time (seconds since the epoch) when the container was started. | Counter | N/A | Basic |
container_finish_time_seconds | UNIX time (seconds since the epoch) when the container was stopped. Only for non-running containers. | Counter | N/A | Basic |
container_fs_reads_bytes_all_total | Total bytes read for all disk devices. | Counter | N/A | Basic |
container_memory_usage_percent | Percent of available memory being used. | Gauge | N/A | Basic |
container_network_receive_bytes_all_total | Total bytes received for all network interfaces. | Counter | N/A | Basic |
container_network_receive_errors_all_total | Total number of errors received for all network interfaces. | Counter | N/A | Basic |
container_network_receive_packets_all_total | Total number of packets received for all network interfaces. | Counter | N/A | Basic |
container_network_transmit_bytes_all_total | Total bytes transmitted for all network interfaces. | Counter | N/A | Basic |
container_network_transmit_errors_all_total | Total number of errors transmitted for all network interfaces. | Counter | N/A | Basic |
container_network_transmit_packets_all_total | Total number of packets transmitted for all network interfaces. | Counter | N/A | Basic |
container_memory_total_bytes | Total number of memory bytes available. | Counter | N/A | Basic |
container_cpu_user_seconds_total | Number of seconds the container has been on the CPU running user code. | Counter | cpu | Custom: Per device metrics or Detailed metrics |
container_cpu_system_seconds_total | Number of seconds the container has been on the CPU running kernel code. | Counter | cpu | Custom: Per device metrics or Detailed metrics |
container_fs_reads_bytes_total | Total bytes read per device | Counter | device | Custom: Per device metrics or Detailed metrics |
container_fs_writes_bytes_all_total | Total bytes written for all disk devices | Counter | N/A | Custom: Detailed metrics |
container_fs_writes_bytes_total | Total bytes written per device | Counter | device | Custom: Per device metrics or Detailed metrics |
container_fs_reads_all_total | Total number of read operations for all disk devices. | Counter | N/A | Custom: Detailed metrics |
container_fs_writes_total | Total number write operations per device. | Counter | device | Custom: Per device metrics or Detailed metrics |
container_memory_mapped_file | Total bytes writted for all disk devices. | Counter | N/A | Custom: Detailed metrics |
container_memory_max_usage_bytes | Highest seen value of the container_memory_usage_bytes metric. | Counter | N/A | Custom: Detailed metrics |
container_memory_pgin | Total number of memory page-in events. | Counter | N/A | Custom: Detailed metrics |
container_mem.pgpgout | Total number of memory pages paged out. | Counter | N/A | Custom: Detailed metrics |
container_memory_pgfault | Total number of major page faults. | Counter | N/A | Custom: Detailed metrics |
container_memory_pgmajfault | Total number of minor page faults. | Counter | N/A | Custom: Detailed metrics |
container_memory_usage_bytes | Number of memory bytes used. | Counter | N/A | Custom: Detailed metrics |
container_network_receive_dropped_all_total | Total number of receives dropped for all network interfaces. | Counter | N/A | Custom: Detailed metrics |
container_network_transmit_dropped_all_total | Total number of transmits dropped for all network interfaces. | Counter | N/A | Custom: Detailed metrics |
Process Metrics
With Process Metrics enabled, Cribl Edge captures process-specific metrics from Linux servers and reports them as events. This allows you to monitor specific processes on Cribl.Cloud instances. You can generate events for any process object.
For information on how to configure the System Metrics Source to generate process-specific metrics, check out the Process Metrics section of the System Metrics page.
Process-specific metrics are not affected by the Host Metrics detail setting.
Process-specific metrics include the following:
| Name | Description | Type | Dimensions |
|---|---|---|---|
process_num_threads | The number of threads. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_open_filedesc | The number of file descriptors. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_write_bytes | The number of bytes which this process caused to be sent to the storage layer. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_read_bytes | The number of bytes this process actually fetched from the storage layer. This number is accurate for block-backed filesystems. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_major_page_faults | The number of major faults for this process that required loading a memory page from disk. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_minor_page_faults | The number of minor faults for this process that have not required loading a memory page from disk. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_voluntary_context_switches | The number of voluntary context switches. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_nonvoluntary_context_switches | The number of involuntary context switches. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_cpu_usage | The process’s CPU usage, expressed as a percentage of total CPU power. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_cpu_seconds | The process’s CPU usage, based on user time and system time. | Counter | process_cmdline, process_set, process_uid, process_gid, process_service |
process_resident_memory_bytes | The amount of memory used, in bytes. Includes the pages that count toward text, data, or stack space. Does not include pages that haven’t been demand-loaded in or are swapped out. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_virtual_memory_bytes | The process’s virtual memory size. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_swapped_memory_bytes | The process’s swapped memory size. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_memory_bytes | The total amount of memory used by the process, in bytes. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_memory_usage | The total amount of memory used by the process, as a percentage of total memory. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |
process_start_time | The time that the process started, derived by adding the start time to the boot time, making it relative to epoch. | Gauge | process_cmdline, process_set, process_uid, process_gid, process_service |