Deploy Cribl Edge in Kubernetes Pods to collect logs, metrics, and events.

Cribl Edge connects to the Kubernetes Cluster API and the kubelet API on each node to read logs, metrics, and events.

To correctly collect logs, metrics, and events using its dedicated Sources, the Cribl Edge agent must be deployed as a DaemonSet in your Kubernetes cluster.

Use Cribl-provided Helm charts for most streamlined deployment of Cribl Edge on Kubernetes. If you want to customize your deployment and provide your own manifest, refer to Deploy Cribl Edge on Kubernetes Without a Helm Chart.

Cribl Edge supports both Docker and containerd runtimes.

Deploy Cribl Edge on Kubernetes via Helm Chart ​

The simplest way to deploy Cribl Edge on Kubernetes is to bootstrap it using the Cribl Edge Helm chart. The chart defines the Service Account, access control, tolerations, and other required configurations.

To deploy a Cribl Edge Node on Kubernetes using the Helm chart:

1. Install Helm 3. See Helm documentation for details.

2. Create or select a Fleet for the Nodes.

3. Set the Fleet API server Host to 0.0.0.0 under Fleet Settings > General Settings > API Server Settings > General. See Configure Fleet Settings for more information.

4. Create a provisioning token. See Create a New Provisioning Token for details.

5. Select Edge Nodes, then Add/Update Edge Node, then Kubernetes. Confirm or modify the settings, then copy the install script.

6. On your target host, run the copied script. The script will take the following form:

   helm install --repo "https://criblio.github.io/helm-charts/"
      --version "<version>"
      --create-namespace -n "<namespace>"
      --set "cribl.leader=tls://<authToken>@<leader-host>?group=<fleet>" "<deployment>" edge

   Replace <deployment> with a descriptive name for the Kubernetes DaemonSet that the script will create. The name must be a valid DNS subdomain.

7. If your cluster uses a certificate that Cribl Edge cannot validate, set the CRIBL_K8S_TLS_REJECT_UNAUTHORIZED environment variable to 0 on the Edge Pods. You can set this variable in the script with: --set extraEnv.CRIBL_K8S_TLS_REJECT_UNAUTHORIZED=0. See Environment Variables for more details.

Deploy Cribl Edge on Kubernetes Without a Helm Chart ​

To deploy Cribl Edge on Kubernetes with your own manifests for GitOps, operators, or platform-specific packaging, deploy the agent as a DaemonSet. You must then define the same objects the Cribl Edge Helm chart creates: Service Account, ClusterRole, and ClusterRoleBinding.

Prerequisites for Manual Deployment on Kubernetes ​

To deploy Cribl Edge on Kubernetes without a Helm chart, you need to:

• Configure a Service Account.
• Configure access control.
• Define a DaemonSet.
• Set tolerations.
• Configure health checks.

Configure a Service Account ​

Create a Kubernetes Service Account for the Edge Pods, or use an existing one. Define it in a ServiceAccount manifest in the namespace where you deploy Cribl Edge. Then reference its name in the ClusterRoleBinding and in the serviceAccountName field of the DaemonSet.

In the manifest, configure the following fields:

For example, create a Service Account with an Amazon EKS IAM role annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cribl-edge-sa
  namespace: <namespace>
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/cribl-edge-role
automountServiceAccountToken: true

If you want the Kubernetes Source to look for connection details in a different directory than the default /var/run/secrets/kubernetes.io/serviceaccount, use the CRIBL_SERVICEACCOUNT_PATH environment variable to set the new path to the ServiceAccount.

For information on how to configure the Service Account for the Leader Node, see Configure a Service Account in Cribl Stream documentation.

Configure Access Control ​

Cribl Edge Kubernetes Sources use the Kubernetes API to read logs, metrics, and events from the cluster. When the cluster uses role-based access control (RBAC), authorize the Cribl Edge Pod Service Account to read that data.

Add the rules for the Sources you enable under rules in a ClusterRole manifest. Bind that ClusterRole to the Edge Service Account with a ClusterRoleBinding, then apply the manifests with kubectl apply.

This is a read-only role, which does not require write permissions. Plan ahead with your Kubernetes admins and security team to authorize the installation of this ClusterRole.

RBAC for Kubernetes Events Source ​

To collect cluster-level events, the Kubernetes Events Source needs watch access to the Kubernetes API.

An example RBAC rule is:

    - apiGroups:
        - "events.k8s.io"
      resources:
        - events
      verbs: ['watch']

RBAC for Kubernetes Logs Source ​

The Kubernetes Logs Source requires the ability to get, list, and watch Pods in all namespaces.

An example RBAC rule is:

    - apiGroups:
        - ""
      resources:
        - pods
      verbs: ['get', 'list', 'watch']

RBAC for Kubernetes Metrics Source ​

The Kubernetes Metrics Source reads metric information from many resources in your cluster. The minimum permissions required are:

    - apiGroups:
        - ""
      resources:
        - configmaps
        - endpoints
        - limitranges
        - namespaces
        - nodes
        - persistentvolumeclaims
        - pods
        - replicationcontrollers
        - secrets
        - services
        - strategicMergePatches
        - nodes/log
        - nodes/metrics
        - nodes/proxy
        - nodes/spec
        - nodes/stats
      verbs: ['get', 'list', 'watch']

    - apiGroups:
        - "apps"
      resources:
        - daemonsets
        - deployments
        - replicasets
        - statefulsets
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "batch"
      resources:
        - cronjobs
        - jobs
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "autoscaling"
      resources:
        - horizontalpodautoscalers
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "policy"
      resources:
        - poddisruptionbudgets
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "networking.k8s.io"
      resources:
        - ingresses
        - networkpolicies
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "admissionregistration.k8s.io"
      resources:
        - mutatingwebhookconfigurations
        - validatingwebhookconfigurations
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "certificates.k8s.io"
      resources:
        - certificatesigningrequests
      verbs: ['get', 'list', 'watch']
    - apiGroups:
        - "storage.k8s.io"
      resources:
        - storageclasses
        - volumeattachments
      verbs: ['get', 'list', 'watch']

RBAC for Prometheus Edge Scraper ​

The Prometheus Edge Scraper Source requires specific Kubernetes RBAC rules depending on the Discovery Type you configure.

Kubernetes Node and Kubernetes Pods ​

To discover Kubernetes Nodes or Pods, the Source must be able to list Pods across all namespaces.

    - apiGroups:
        - ""
      resources:
        - pods
      verbs: ['list']

Kubernetes Service Monitor (v4.18+) ​

To discover ServiceMonitor custom resources and their associated Services, the Source needs the following permissions:

    - apiGroups:
        - "monitoring.coreos.com"
      resources:
        - servicemonitors
      verbs: ['list']
    - apiGroups:
        - ""
      resources:
        - services
        - namespaces
      verbs: ['list']

The namespaces permission is required only when any ServiceMonitor in the cluster defines namespaceSelector.any: true, causing the Source to enumerate all namespaces. If you know your ServiceMonitor resources use only the matchNames selector or default to their own namespace, you can omit namespaces from the rule.

Define a DaemonSet ​

Define a DaemonSet so one Cribl Edge Pod runs on each node. Set serviceAccountName to the Edge Service Account, and set the container image to cribl/cribl:<version>.

Add these environment variables under spec.template.spec.containers in the DaemonSet manifest. Set CRIBL_K8S_POD to the Pod name to allow Cribl Edge to detect that it is running inside a Pod. When TLS is enabled, use tls:// in CRIBL_DIST_LEADER_URL:

env:
  - name: CRIBL_DIST_LEADER_URL
    value: "tls://<authToken>@<leader>:4200?group=<fleet>"
  - name: CRIBL_DIST_MODE
    value: managed-edge
  - name: CRIBL_EDGE
    value: "1"
  - name: CRIBL_K8S_POD
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Set Tolerations ​

Kubernetes does not schedule a Pod on a node that has taints. Add a toleration so Cribl Edge can collect logs and metrics from those nodes.

Add the following field under spec.template.spec in the DaemonSet manifest. operator: Exists matches every taint, so the Pod can run on any node:

tolerations:
  - operator: Exists

Configure Kubernetes Health Checks ​

When you deploy Cribl Edge on Kubernetes, configure health checks so Kubernetes restarts an unresponsive Pod and detects when a Pod is ready.

An example health-check configuration for Kubernetes, including liveness and readiness probes, is:

readinessProbe:
    httpGet:
        path: /api/v1/health
        port: 9420
        scheme: HTTP
    initialDelaySeconds: 15
    timeoutSeconds: 1
livenessProbe:
    httpGet:
        path: /api/v1/health
        port: 9420
        scheme: HTTP
    initialDelaySeconds: 15
    timeoutSeconds: 1

If you have Fleets running Edge Nodes with version 4.8.2 and older, you can encounter failing health checks that require specific API server configuration.

By default, the Cribl Edge API listens on port 9420 instead of the Cribl Stream default port 9000. This requires special care when you configure your Edge Fleet in the Cribl UI.

On Fleets running Edge Nodes with version 4.8.2 and older, the API server is configured be default to listen on the loopback interface (127.0.0.1). For your Nodes to remain reachable, you must change the listening interface address to 0.0.0.0 in Fleet Settings > General Settings > API Server Settings > General. This will bind to all IPs in the container. If you do not make this change, your health checks will start failing after the Cribl Edge agent downloads its initial config bundle.

Another side effect of not changing the listening interface address is that the Edge Node will not appear on the Leader UI.

Kubernetes CPU Limit Configuration ​

The Cribl Edge Helm chart ensures that the number of Worker Processes is determined by the CPU resources allocated to the pod, rather than the physical CPU count on the host.

To achieve this, the chart sets the CRIBL_K8S_CPU_LIMIT environment variable to the value of the resources.limits.cpu setting on the Pod. Cribl Edge then uses the value as a CPU limit to determine how many Worker Processes to start.

We recommend maintaining the default configuration of the CRIBL_K8S_CPU_LIMIT environment variable to prevent performance problems, such as resource contention and Pod instability, caused by exceeding CPU limits.

Manually Deploy Cribl Edge on Kubernetes ​

Once you have configured all prerequisites, you can proceed with the manual deployment of Cribl Edge on Kubernetes.

Use the chart templates and values as a checklist when you translate the deployment to raw YAML.

1. Create or select a Fleet for the Nodes.
2. Set the Fleet API server Host to 0.0.0.0 under Fleet Settings > General Settings > API Server Settings > General. See Configure Fleet Settings for more information.
3. Create a provisioning token. See Create a New Provisioning Token for details.
4. Apply your manifests with kubectl apply. You can use any namespace.
5. If your cluster uses a certificate that Cribl Edge cannot validate, set the CRIBL_K8S_TLS_REJECT_UNAUTHORIZED environment variable to 0 on the Edge Pods. See Environment Variables for more details.

Run Kubernetes Sources Outside the Cluster ​

Optionally, you can run the Kubernetes Sources (Metrics, Logs, and Events) outside of a cluster, by specifying connection details in a kubeconfig file (like kubectl). Either place the file in $HOME/.kube/config, or use the $KUBECONFIG variable to point to it.

Deploy on OpenShift or Non-Root Environments ​

To deploy Cribl Edge on OpenShift or other non-root environments, add Linux capabilities to the Edge container. These capabilities let Cribl Edge bind to low ports, read host files, and inspect processes. The Cribl Edge Helm chart leaves securityContext empty, and the install script copied from the UI does not set it.

To add the capabilities with the Cribl Helm chart:

1. Create a local values.yaml file. Adjust the capabilities for your environment:

   securityContext:
     capabilities:
       add:
         - CAP_NET_BIND_SERVICE
         - CAP_SYS_PTRACE
         - CAP_DAC_READ_SEARCH
         - CAP_NET_ADMIN

   Copy

2. Install the chart with that file and the Leader URL from the install script:

   helm install --repo "https://criblio.github.io/helm-charts/"
      --version "<version>"
      --create-namespace -n "<namespace>"
      --set "cribl.leader=tls://<authToken>@<leader-host>?group=<fleet>"
      -f values.yaml "<deployment>" edge

   Replace <deployment> with a descriptive name for the Kubernetes DaemonSet that the script will create. The name must be a valid DNS subdomain.

For a description of the capabilities, see Set Capabilities for Cribl Edge.

To set the same capabilities without a Helm chart, add this securityContext block under the Edge container in the DaemonSet manifest. Then apply the manifest with kubectl apply.

Uninstall Cribl Edge from a Kubernetes Cluster ​

To remove the Cribl Edge agent from your Kubernetes cluster, run the following command:

helm uninstall -n <namespace> <deployment>

deployment is the deployment name you selected when configuring the Helm chart.

The helm uninstall command automatically deletes all deployed resources associated with the Cribl Edge Helm Chart.

Troubleshooting ​

Here are some common challenges you can encounter when deploying Cribl Edge via Kubernetes.

Duplicated Nodes in Node List ​

Edge Nodes on Kubernetes restart with new GUIDs, causing the Edge Nodes page to track and display each restart as a separate Node. To prevent this behavior in Fleets containing Kubernetes nodes, set General Settings > Fleet Configuration > Time to keep disconnected Nodes to 0. This ensures that Nodes in these Fleets will not be tracked between restarts.

Certificate Validation Fails ​

By default, the kubelet serves a certificate signed by an ephemeral self-signed certificate authority. If your cluster uses a certificate that Cribl Edge cannot validate, all connection attempts to the Kubernetes API will be rejected.

To avoid this, set the CRIBL_K8S_TLS_REJECT_UNAUTHORIZED environment variable to 0 to disable the expectation of valid certificates. When you disable this environment variable, all Kubernetes features (including Metadata, Metrics, Logs, and workload instrumentation metadata) will tolerate both self-signed and invalid TLS certificates when connecting to the Kubernetes APIs.

Log Loss Due to Polling Interval and Rotation ​

The Kubernetes Logs Source polls the local kubelet for Pod discovery. While this is a lightweight operation, the Polling interval setting is important for complete log capture, especially with high log volumes and frequent rotations. For details, see Polling Interval and Log Rotation.