Collect Kubernetes data with Cribl Edge to forward them for further observability and analytics.

Kubernetes environments generate large volumes of observability data, including logs, events, and metrics, making data collection complex. Cribl Edge offers the Kubernetes Logs, Kubernetes Events, and Kubernetes Metrics Source, as well as Prometheus Edge Scraper to simplify this task.

The Kubernetes Logs, Kubernetes Metrics, and Prometheus Edge Scraper Sources support disk spooling, which allows Cribl Search to query their output.

Collect Kubernetes Logs ​

The Kubernetes Logs Source connects to the Kubernetes API and loads the lists of Pods on the node on a configurable polling interval.

For a full description and reference of the Source’s settings, see Kubernetes Logs Source.

Enable Preconfigured Kubernetes Logs Source ​

Cribl Edge has a preconfigured in_kube_logs Source that you can adapt to your needs. To enable log collection with the Source:

1. On the top bar, select Products, and then select Edge. Under Fleets, select a Fleet.
2. Select Collect.
3. Select the in_kube_logs Kubernetes Logs Source.
4. Select and drag the plus icon next to it to a Destination (while testing the procedure, you can choose DevNull).
5. In the Connection Configuration modal, select how you want the data to flow through to the Destination. Select Passthru to send all the data without processing, then select Save.
6. To start the Source, select Commit, enter a commit name, and select Commit & Deploy.

Now, ensure that data is being collected: hover over the Kubernetes Logs Source and select Capture. After a short time you should see sample data captured by the Source. Once you’re satisfied, you can switch from DevNull to your desired target Destination.

Finetune Kubernetes Log Collection ​

Now that you have the basic Kubernetes log flow configured, you can customize the Source settings.

Configure Log Polling Interval ​

Set the polling interval based on the expected start and stop intervals of Pods in your environment. Higher volume log sources may require a faster polling interval to avoid rotating logs faster than you can collect them.

For example, if the container emits 1 MBps and rotates logs at 10 MB, the polling interval must be less than 10s, to prevent hitting the 10 MB rotation threshold twice in 10 seconds.

For more information about configuring the polling interval, see Polling Interval and Log Rotation.

Configure Log Filter Rules ​

The Source determines which Pods to report on based on the rules defined in Filter Rules. The default !metadata.namespace.startsWith('kube-') rule means the Source ignores Pods in the kube-* namespace.

You can use any Kubernetes Pod object fields in the filter, such as metadata, spec, and status.

Below the Capture button, select the Advanced Settings button and then toggle Show Internal Fields on to explore the __metadata fields forwarded from the Cribl Edge Nodes. These metadata fields show information about the Pod, Node, and Cribl environment where they were collected.

See Kubernetes Pod Object definition for details.

Collect Kubernetes Events ​

Kubernetes events include state changes (such as Pod starts and jobs that fail to start) occurring on the Kubernetes cluster regarding workloads. To monitor such events, use the preconfigured Kubernetes Events Source called in_kube_events.

1. On the top bar, select Products, and then select Edge. Under Fleets, select a Fleet.
2. Select Collect.
3. Select the in_kube_events Kubernetes Events Source.
4. Select and drag the plus icon next to it to a Destination (while testing the procedure, you can choose DevNull).
5. In the Connection Configuration modal, select how you want the data to flow through to the Destination. Select Passthru to send all the data without processing, then select Save.
6. To start the Source, select Commit, enter a commit name, and select Commit & Deploy.

Now, ensure that data is being collected: hover over the Kubernetes Events Source and select Capture. After a short time you should see sample data captured by the Source. Once you’re satisfied, you can switch from DevNull to your desired target Destination.

For a full description and reference of the Source’s settings, see Kubernetes Events Source.

Configure Event Filter Rules ​

The Source determines which events to collect based on the rules defined in Filter Rules. No default rules are defined, which means all emitted events are collected.

As an example, you can filter out events based on their type with object.type.equals('Warning') or, based on the reason, with object.reason.equals('Failed').

Collect Kubernetes Metrics ​

The Kubernetes Metrics Source reports cluster, Node, Pod, and container status from the Kubernetes API. It does not include all Prometheus metrics exposed by the kubelet. To collect node and container-level metrics such as node_cpu_seconds_total and cAdvisor counters, configure Cribl Edge to scrape the kubelet’s /metrics and /metrics/cadvisor endpoints with the Prometheus Edge Scraper.

Collect Metrics with the Kubernetes Metrics Source ​

The Kubernetes Metrics Source lets you generate metrics periodically based on the status and configuration of a Kubernetes cluster and its Nodes, Pods, and containers.

You can enable Collect kubelet metrics and Collect cAdvisor metrics on the Kubernetes Metrics Source to scrape these endpoints from one Source.

Use Prometheus Edge Scraper when you need separate Sources, or custom job fields. Do not enable both Sources for the same endpoints to avoid duplicating scrape load.

See a full description and reference of the Source’s settings in Kubernetes Metrics Source, and more information about metrics collected in Kubernetes Metrics Details.

Collect Metrics with Prometheus Edge Scraper ​

You can configure the Prometheus Edge Scraper to collect metrics from Pods and Nodes (as well as other Prometheus targets).

To collect Kubernetes metrics, Cribl Edge uses the /metrics and /metrics/cadvisor endpoints on the kubelet. This information gives you a good idea of what’s going on on each Node and Pod.

Using the Prometheus Edge Scraper, you can discover and collect metrics in Kubernetes Clusters dynamically. Kubernetes-specific discovery types include:

• Kubernetes Node: When configured to scrape Kubernetes Nodes, each Cribl Edge Node (Pod) will scrape from the configured port on every deployed Kubernetes Node in the DaemonSet. Use this for kubelet and cAdvisor collection and to scrape other node-local interfaces.
• Kubernetes Pods: When configured for Pod scraping, a single Edge Node will be elected to scrape the metrics from the discovered endpoints.
• Kubernetes Service Monitor (v4.18+): Discovers and scrapes Prometheus endpoints by reading ServiceMonitor custom resources defined by the Prometheus Operator, resolving matching Kubernetes Services, and scraping endpoints those Services expose.

For a full description of how to configure the Source and a reference of its settings, see Prometheus Edge Scraper.

Deploy Cribl Edge as a DaemonSet with direct network access to each Node. Authorize the Source to list Pods. See RBAC for Prometheus Edge Scraper.

Configure Kubelet and cAdvisor Metrics ​

1. In the sidebar, select Fleets, then select a Fleet.
2. Select More > Sources, navigate to Prometheus Edge Scraper and select Add Source.
3. In Input ID, enter kubelet_metrics or cadvisor_metrics, depending on your use case.
4. Set Discovery type to Kubernetes Node.
5. Set Protocol to https.
6. In Port, enter 10250.
7. Leave Path at the default /metrics, or change to /metrics/cadvisor.
8. Select the Authentication tab and set Authentication method to Kubernetes.
9. Optionally, select the Fields tab, select Add Field, and add a field named job with value 'kubelet' or 'cadvisor'.
10. Select Save, then select Commit & Deploy.

To emit metrics in the same field format as System Metrics and Kubernetes Metrics, toggle on Use field per metric in each Source’s optional settings.

Verify collection by hovering over a Source and selecting Capture. After a short time, you should see kubelet or cAdvisor metrics in the captured data.