Collect Kubernetes data with Cribl Edge to forward them for further observability and analytics.

Kubernetes environments generate large volumes of observability data, including logs, events, and metrics, making data collection complex. Cribl Edge offers the Kubernetes Logs, Kubernetes Events, and Kubernetes Metrics Source, as well as Prometheus Edge Scraper to simplify this task.

The Kubernetes Logs, Kubernetes Metrics, and Prometheus Edge Scraper Sources support disk spooling, which allows Cribl Search to query their output.

Collect Kubernetes Logs

​

The Kubernetes Logs Source connects to the Kubernetes API and loads the lists of Pods on the node on a configurable polling interval.

For a full description and reference of the Source’s settings, see Kubernetes Logs Source.

Enable Preconfigured Kubernetes Logs Source

​

Cribl Edge has a preconfigured in_kube_logs Source that you can adapt to your needs. To enable log collection with the Source:

1. On the top bar, select Products, and then select Edge. Under Fleets, select a Fleet.
2. Select Collect.
3. Select the in_kube_logs Kubernetes Logs Source.
4. Select and drag the plus icon next to it to a Destination (while testing the procedure, you can choose DevNull).
5. In the Connection Configuration modal, select how you want the data to flow through to the Destination. Select Passthru to send all the data without processing, then select Save.
6. To start the Source, select Commit, enter a commit name, and select Commit & Deploy.

Now, ensure that data is being collected: hover over the Kubernetes Logs Source and select Capture. After a short time you should see sample data captured by the Source. Once you’re satisfied, you can switch from DevNull to your desired target Destination.

Finetune Kubernetes Log Collection

​

Now that you have the basic Kubernetes log flow configured, you can customize the Source settings.

Configure Log Polling Interval

​

Set the polling interval based on the expected start and stop intervals of Pods in your environment. Higher volume log sources may require a faster polling interval to avoid rotating logs faster than you can collect them.

For example, if the container emits 1 MBps and rotates logs at 10 MB, the polling interval must be less than 10s, to prevent hitting the 10 MB rotation threshold twice in 10 seconds.

For more information about configuring the polling interval, see Polling Interval and Log Rotation.

Configure Log Filter Rules

​

The Source determines which Pods to report on based on the rules defined in Filter Rules. The default !metadata.namespace.startsWith('kube-') rule means the Source ignores Pods in the kube-* namespace.

You can use any Kubernetes Pod object fields in the filter, such as metadata, spec, and status.

Below the Capture button, select the Advanced Settings button and then toggle Show Internal Fields on to explore the __metadata fields forwarded from the Cribl Edge Nodes. These metadata fields show information about the Pod, Node, and Cribl environment where they were collected.

See Kubernetes Pod Object definition for details.

Collect Kubernetes Events

​

Kubernetes events include state changes (such as Pod starts and jobs that fail to start) occurring on the Kubernetes cluster regarding workloads. To monitor such events, use the preconfigured Kubernetes Events Source called in_kube_events.

1. On the top bar, select Products, and then select Edge. Under Fleets, select a Fleet.
2. Select Collect.
3. Select the in_kube_events Kubernetes Events Source.
4. Select and drag the plus icon next to it to a Destination (while testing the procedure, you can choose DevNull).
5. In the Connection Configuration modal, select how you want the data to flow through to the Destination. Select Passthru to send all the data without processing, then select Save.
6. To start the Source, select Commit, enter a commit name, and select Commit & Deploy.

Now, ensure that data is being collected: hover over the Kubernetes Events Source and select Capture. After a short time you should see sample data captured by the Source. Once you’re satisfied, you can switch from DevNull to your desired target Destination.

For a full description and reference of the Source’s settings, see Kubernetes Events Source.

Configure Event Filter Rules

​

The Source determines which events to collect based on the rules defined in Filter Rules. No default rules are defined, which means all emitted events are collected.

As an example, you can filter out events based on their type with object.type.equals('Warning') or, based on the reason, with object.reason.equals('Failed').

Collect Kubernetes Metrics

​

The Kubernetes Metrics Source lets you generate metrics periodically based on the status and configuration of a Kubernetes cluster and its Nodes, Pods, and containers.

See a full description and reference of the Source’s settings in Kubernetes Logs Source, and more information about metrics collected in Kubernetes Metrics Details.

Collect Metrics with Prometheus Edge Scraper

​

You can also configure the Prometheus Edge Scraper to collect metrics from Pods and Nodes (as well as other Prometheus targets).

To collect Kubernetes metrics, Cribl Edge uses the /metrics and /metrics/cadvisor endpoints on the kubelet. This information gives you a good idea of what’s going on on each Node and Pod.

Using the Prometheus Edge Scraper, you can discover and collect metrics in Kubernetes Clusters dynamically. There are two Kubernetes-specific modes of operation:

• Kubernetes Node: When configured to scrape Nodes, each Cribl Edge Node (Pod) will scrape from the configured port on every deployed Node in the DaemonSet. Use this to scrape endpoints on the Kubelet and other node-local interfaces.
• Kubernetes Pods: When configured for Pod scraping, a single Edge Node will be elected to scrape the metrics from the discovered endpoints.

For a full description of how to configure the Source and a reference of its settings, see Prometheus Edge Scraper.