.cribl.cloud`.
- **Static IPs**: For IP address allowlist or security group rules, you can find the static IPs of your Cribl.Cloud
Leader in **Workspace** > **Access** > **Leader NLB IPs**. You can also use the `nslookup` command to obtain static
IPs.
- **Private Link**: As an alternative to allowlisting static IPs, eligible Enterprise Organizations can reach the
Cribl.Cloud Leader over AWS PrivateLink, so the connection stays on your cloud provider's private network instead of
the public internet. Leader Private Link is AWS only, and the connecting environment must be able to create an AWS VPC
interface endpoint. An on-prem Leader that cannot create an AWS interface endpoint cannot use Private Link for this
connection. See [Private Link for Cribl.Cloud](/stream/privatelink-cloud#leader).
- **Workspaces**: Workspaces are separate infrastructure. For allowlists and security groups, ensure that you add each
IP address individually as each Workspace has a unique IP.
############################################################
# SOURCE: https://docs.cribl.io/edge/cloud-connected-env-how-to.md
############################################################
# How to Connect On-Prem Leaders to Cribl.Cloud
To connect existing on-prem Cribl Edge Distributed deployments to Cribl.Cloud,
select an on-prem Leader that is running Cribl Edge 4.8.2 or newer, then follow the steps.
>If you have multiple environments, such as staging and production environments
>when using GitOps, you must connect each environment individually.
{.box .info}
1. On the top bar, select **Products**, and then select **Cribl**.
1. In the sidebar, select **Workspace**, and then select **Connections**.
1. Select **Configure New Environment** and fill out the following fields:
|Field|Description|Example|
|-----|-----------|-------|
|Environment name
|A name for your environment. This name will display on the **Connected Environments** page. You can search for and filter environments by name.|Goat Farm East Production|
|Workspace hostname/ID|Hostname of your Cribl.Cloud Workspace.
This field is read-only.|`main-random-goat-a12bcdef.cribl.cloud`|
|Workspace port number|Port number that the Workspace Leader uses to communicate with the on-prem Leader.|`4200`|
|Workspace token| The auth token defined in the Cribl.Cloud Leader.|`a12bcDEFgHiJKLmnOpQRstuvw3x4z56`|
|Script type|The format of the script to connect to the on-prem Leader: |**Environment variable**:
CRIBL_DEPLOYMENT="Goat Farm West Production"
CRIBL_CLOUD_WORKSPACE_URL="tls://a12bcDEFgHiJKLmnOpQRstuvw3x4z56@main-random-goat-a12bcdef.cribl.cloud:4200"
**CLI**:
./cribl cloud-workspace
-d "Goat Farm West Production"
-H main-random-goat-a12bcdef.cribl.cloud
-p 4200
-u a12bcDEFgHiJKLmnOpQRstuvw3x4z56
|
## Use a Command-Line Script
If you're using the CLI option, follow these steps, in order:
1. Select **Copy Script**.
1. Open a command-line window in the Leader environment that you want to connect to Cribl.Cloud.
1. Navigate to the `$CRIBL_HOME/bin` subdirectory.
1. Paste the script results onto the command line, and run the script.
## Use an Environment Variable
The **Environment Variable** script populates with the details you enter when
you're configuring a new environment. For example:
```
CRIBL_DEPLOYMENT="On-prem Stream"
CRIBL_CLOUD_WORKSPACE_URL="tls://a12bcDEFGHIjKLMnopPqr123@main-random-goat-a12bcdef.cribl.cloud:4200"
```
If you're using the environment variable, copy its output,
and set it when - or before - you run `cribl start`.
How you set this variable depends on your deployment.
- For example, if your Cribl Edge deployment is running in a container,
set the environment variable in the container via run call or configuration.
- If you're running your own server, you can set environment variables for the `cribl` user,
or you can declare the variable along with the `start` command.
For details, see our [CLI Reference](cli-cloud-workspace).
## High Availability Leader Configuration
Before you configure a connected environment with a High Availability (HA) Leader,
ensure you are comfortable setting up and managing Leader HA. See
[Leader High Availability/Failover](deploy-add-second-leader) for more information.
When you set up the primary Leader to connect to Cribl.Cloud in an HA environment,
set the `CRIBL_DEPLOYMENT` and `CRIBL_CLOUD_WORKSPACE_URL` environment variables.
You can do this in the Cribl.Cloud UI, using the CLI, or directly in your environment.
These variables use the same format that the **Environment variable** script shows in the UI:
```
CRIBL_DEPLOYMENT="aws_leader_prod"
CRIBL_CLOUD_WORKSPACE_URL="tls://a12bcDEFgHiJKLmnOpQRstuvw3x4z56@main-random-goat-a12bcdef.cribl.cloud:4200"
```
If you manage the primary Leader as a `systemd` service, set the same variables in the
`[Service]` section of the unit file using `Environment=` entries:
```
Environment=CRIBL_DEPLOYMENT=aws_leader_prod
Environment=CRIBL_CLOUD_WORKSPACE_URL=tls://a12bcDEFgHiJKLmnOpQRstuvw3x4z56@main-random-goat-a12bcdef.cribl.cloud:4200
```
The failover Leader inherits the configuration from the filesystem changes
made by the primary Leader.
## Enable the On-Prem Leader Connection to Cribl.Cloud {#enable-connection}
You can connect a Leader Node to an existing Enterprise Cribl.Cloud Organization.
1. In the sidebar, select **Settings**.
1. On the **Global** tab, select **Distributed Settings**.
1. Select **Cloud Connection**.
1. Toggle **Join Cribl.Cloud Workspace** on.
1. Fill in the required fields:
|Field|Description|
|-----|-----------|
|**Connection URL**|The **Cribl.Cloud URL** to connect to this Leader. - In your Cribl.Cloud Organization, locate the URL field value on the sidebar **Workspace** > **Access** tab.
- Copy the URL and paste it into the **Connection URL** field, using the format: `main-random-goat-a12bcdef.cribl.cloud`.
|
|**Connection port**|The port for the Cribl.Cloud Leader. Default is `4200`.|
|**Connection auth token**|The authentication token for the Cribl.Cloud Leader. You'll need to get this from the Cribl.Cloud environment: - Navigate to **Workspace** > **Connections** and select **Configure New Environment**.
- Copy the value in the **Workspace token** field.
- In the on-prem environment, paste the Workspace token from Cribl.Cloud.
|
1. Select **Save**.
1. In the Cribl Subscription Services Agreement modal, select **Agree** to accept the agreement.
1. In the confirmation modal, select **Yes** to save changes and restart.
>Once you connect an on-prem Leader to a Cribl.Cloud Organization, the license
>type changes to a Cribl.Cloud billing account. This means the **Licensing** page will not
>be accessible. To return to using an on-prem license, toggle **Join Cribl.Cloud Workspace** off, select **Save**, and select **Yes** in the confirmation modal
> to save changes and restart.
{.box .info}
############################################################
# SOURCE: https://docs.cribl.io/edge/about-universal-subscription.md
############################################################
# Simplify Billing with Universal Subscription
>This feature requires an Enterprise Plan.
{.box .cloud}
If you have multiple instances of Cribl Edge on-prem, you're likely managing
multiple licenses. Instead of letting license management run your life, you can
take advantage of the [Universal Subscription](https://cribl.io/pricing/plan/?product=universal-subscription) feature.
Universal Subscription lets you manage an existing on-prem license with Cribl.Cloud credits.
To get started, connect your on-prem Leader to your Cribl.Cloud Leader to
send billing data payloads. Once your environments are connected, you can use
the [FinOps Center](/billing-licensing/finops-center) to get a unified, all-product view of your Cribl Edge usage
and spending.
This replaces an on-prem license, and gives you the flexibility to maintain
your on-prem deployments without managing multiple licenses and tracking
spend across environments.
> An on-prem Cribl Edge environment consists of at least one Leader Node (the control plane) and one or more Fleets.
{.box .info}
Connecting your on-prem Leader license to Cribl.Cloud billing allows you to:
- Monitor credit consumption for each of your on-prem Cribl Edge environments
from one user interface.
- Create a billing- and usage-based connection between the Cribl Edge on-prem
Leader and the Cribl.Cloud Leader, which receives metrics
from the on-prem Leader. For information about the data payloads, visit [Data Payloads for Connected Environments](cloud-connected-payloads).
## Configure Connected Environments
For the setup procedure, go to [How to Connect On-Prem Leaders to Cribl.Cloud](cloud-connected-env-how-to),
then come back here to explore how to view credit and usage data for your on-prem Connected Environments.
>Full access to Connected Environments requires you to upgrade Cribl Edge to 4.8.2 or newer.
{.box .warning}
## Explore Credit Consumption for Connected Environments
Once you set up a Connected Environment, you can view the incoming usage data
and credit consumption for your on-prem and Cloud deployments of Cribl Edge. You
must be an Organization Owner to view this data.
For more information on Cribl.Cloud billing, go to the [FinOps Center](/billing-licensing/finops-center) page.
### View Usage Data
1. In the sidebar, select **Organization**, then **FinOps Center**.
1. Select **Usage Overview**.
1. Go to **Per Product Consumption** and expand either Cribl Edge or Cribl Stream.
1. Find **Connected < Product > GBs In**.
### View Credit Consumption
1. In the sidebar, select **Organization**, then **FinOps Center**.
1. On the **FinOps Center** submenu, select **Invoices**.
1. Select a month to expand the cost items. Select **Stream** or **Edge**.
1. View **Connected < Product > GBs In** usage and cost.
Here, you will see each on-prem environment that is connected to this Cribl.Cloud
Organization, and the credits each environment is consuming.
## Limitations of the Universal Subscription Feature
Connected Environments has some usage limitations:
- The Connected Environments feature is available only on Enterprise Cribl.Cloud plans. For Enterprise plan details, see [Pricing](https://cribl.io/pricing/).
- Single-instance deployments of Cribl Edge aren't eligible for a Universal Subscription.
- The Distributed deployment on-prem Leader must be running Cribl Edge 4.8.2 or newer.
############################################################
# SOURCE: https://docs.cribl.io/edge/cloud-connected-payloads.md
############################################################
# Data Payloads for Connected Environments
In a [Connected Environment](cloud-connected-env) setup, on-premises Leaders connect to a Cribl.Cloud Leader in the same way that Edge Nodes connect to a Leader. The Cribl.Cloud Leader sends down a license, and the on-premises Leader sends up a heartbeat that includes telemetry, which encompasses metrics and other metadata.
## Why Is the Telemetry Important?
The telemetry sent from an on-premises Leader to a Cribl.Cloud Leader is encapsulated in an event. This event serves as a heartbeat, confirming that the connected environment is active and functioning properly. It includes key metadata such as the system's status, timestamp, configuration details, and metrics. Cribl.Cloud uses this data to:
- Monitor connectivity between the on-premises Leader and the cloud.
- Track system health by evaluating fields like status, edgeNodes, and metrics.
- Ensure compliance with licensing by verifying the config.featuresRev field.
- Provide insights into data flow, such as input and output metrics.
### How Often Is the Heartbeat Sent?
The heartbeat interval is determined by the **Task heartbeat period** field, which specifies the number of seconds between each event. In the provided example below, the heartbeat is sent every `10` seconds. This interval is configurable, meaning administrators can adjust it to optimize performance based on their needs. For details, see [Task Manifest and Buffering Limits](group-settings#task-manifest-and-buffering-limits) in Fleet Settings.
### What Happens if the Heartbeat Is Missed?
If a heartbeat event is not received within the expected interval, the following actions may occur:
- **Retries** - The on-premises Leader may attempt to resend the heartbeat after a short delay.
- **Alerting** - Cribl.Cloud can generate an alert if the heartbeat is missing for an extended period, indicating a potential issue.
- **Failure detection** - A prolonged absence of heartbeats may be interpreted as a disconnection or system failure, prompting further investigation or automated recovery steps.
## On-Premises Leader to Cribl.Cloud Leader Payload
This section describes the structure of the event payload sent from an on-premises Leader to a Cribl.Cloud Leader in a Connected Environment.
### Sample Event
Here is a sample event that Cribl.Cloud received from a connected environment.
```shell
{
"__criblEventType": "event",
"type": "info",
"_time": 1737577024.567,
"status": "healthy",
"localTime": 1737577024,
"config.hbPeriodSeconds": 10,
"config.featuresRev": "YbK7TIGe4jp2rdujnFiJqq0OBV67r2FKGvrHr4C6I5g=",
"edgeNodes": 50,
"__socketAddr": 192.168.1.100:443,
"__srcIpPort": 10.0.0.5:52314,
"metrics": [
[
0,
0,
"total.in_bytes",
233,
1737578280038,
1737578340038,
[
"ci",
"tcp:in_tcp",
"input",
"tcp:in_tcp",
"input_type",
"tcp",
"__worker_group",
"default",
"__dist_mode",
"worker",
"__worker_node",
"46731aec-d220-4cb0-9971-0dd0a7907d54",
"__product",
"stream"
],
null,
"float32"
]
],
"__raw": "{\"type\":\"info\",\"_time\":173757...}"
}
```
### Event Payload Fields
The following table describes the important fields in the event payload.
| Field | Usage | Example |
|-------------------------|--------------------------------------------------------------------|---------|
| __criblEventType | Type of event | event (always) |
| type | Type of event | info (always) |
| _time | Timestamp of the event in [Unix time](https://en.wikipedia.org/wiki/Unix_time) (seconds) | 1737577024.567 |
| status | Status of client | healthy |
| localTime | Local timestamp in Unix time (seconds) | 1737577024 |
| config.hbPeriodSeconds | Seconds between client heartbeats | 10 |
| config.featuresRev | Version of feature flags (internal use) | YbK7TIGe4jp2rdujnFiJqq0OBV67r2FKGvrHr4C6I5g= |
| edgeNodes | Number of connected Edge Nodes | 50 |
| __socketAddr | Connection detail | `192.168.1.100:443` |
| __srcIpPort | Connection detail | `10.0.0.5:52314`|
| metrics | Array of metrics (as nested arrays) included in this heartbeat. This sample only shows one nested metric array but an actual heartbeat will contain many metrics such as in bytes, out bytes, in events, out events for each input as well as an aggregated total. | ```[ [ 0, 0, "total.in_bytes", 233, 1737578280038, 1737578340038, [ "ci", "tcp:in_tcp", "input", "tcp:in_tcp", "input_type", "tcp", "__worker_group", "default", "__dist_mode", "worker", "__worker_node", "46731aec-d220-4cb0-9971-0dd0a7907d54", "__product", "stream" ], null, "float32" ] ]``` |
| __raw | Holds the full raw data of the event as a JSON string | ```{"type":"info","_time":173757...``` |
############################################################
# SOURCE: https://docs.cribl.io/edge/cloud-connected-data-transfer.md
############################################################
# Send Data from On-Prem to Cribl.Cloud
You can use Connected Environments to send data from an on-prem deployment to a Cribl.Cloud deployment. For example, if you previously had an on-prem deployment of Cribl and you are moving forward with a Cloud-first or Cloud only solution, you can use connected environments to onboard your data from a on-prem Edge Nodes to their counterpart Cribl.Cloud Edge Nodes.
You can transfer data from on-prem Edge Nodes to the Cribl.Cloud Edge Nodes without incurring any data transfer costs.
## Overview of the Data Migration Workflow
The workflow for migrating data from on-prem Edge Nodes to Cribl.Cloud Edge Nodes has these phases:
1. [Create a Connected Environment](#new-connected-env). This process generates the information you need to set up the data flow between the on-prem Edge Nodes and the Cribl.Cloud Edge Nodes.
1. [On-Prem Edge Nodes: Configure HTTP or TCP Destination](#on-prem-destination). Add an auth token on the on-prem Edge Nodes to configure the first half of the connection between the on-prem Destination and the Cribl.Cloud Source.
1. [Cribl.Cloud Edge Nodes: Configure HTTP or TCP Source](#cloud-source). Add a matching auth token on the Cribl.Cloud Edge Nodes to complete the connection.
1. [Validate the Data Flow](#validate-data-flow). Confirm that the data is properly flowing from the on-prem Edge Nodes to the Cribl.Cloud Edge Nodes.
### Create a Connected Environment {#new-connected-env}
To begin the process of migrating data from on-prem to Cribl.Cloud, you need to first set up a Connected Environment on the on-Prem Leader. This process generates the information you need to set up the data flow between on-prem Edge Nodes and the Cribl.Cloud Edge Nodes.
To set up a Connected Environment, follow the instructions on [How to Connect On-Prem Leaders to Cribl.Cloud](cloud-connected-env-how-to). These instructions also explain how to:
- Optionally set up environment variables at the time you create the Connected Environment.
- Use a command-line script to set up a Connected Environment if needed.
Ensure that you also follow the instructions to [Enable the On-Prem Leader Connection to Cribl.Cloud](cloud-connected-env-how-to#enable-connection).
### On-Prem Edge Nodes: Configure HTTP or TCP Destination {#on-prem-destination}
On the on-prem deployment, select a Edge Node and configure a new or existing [Cribl HTTP](destinations-cribl-http) or [Cribl TCP Destination](destinations-cribl-tcp). These Destinations enable internal data transfer between Cribl nodes to prevent data transfer billing costs.
To configure an HTTP or TCP Destination:
1. Follow the general configuration instructions on the [Cribl HTTP](destinations-cribl-http) or [Cribl TCP Destination](destinations-cribl-tcp) documentation.
1. In the **Auth Token** section of the configuration modal, select **Auth Token** to set up a shared secret between this Destination and the Cribl.Cloud Source:
- **Token secret (text secret):** Select the name of the secret that you have created previously or create a new secret using the **Create** button. See [Create and Manage Legacy Secrets](securing-secrets) for specific instructions about creating secrets.
- **Enable token:** Toggle the token on to enable it.
- **Description:** Optionally, enter a description summarizing the purpose of the secret.
> If you don't see the **Auth Token** section of the Destination, it means you have not set up a Connected Environment for your on-prem Leader.
>
{.box .warning}
1. Commit and deploy the changes.
You can add multiple tokens if you need to rotate auth tokens.
### Cribl.Cloud Edge Nodes: Configure HTTP or TCP Source {#cloud-source}
On the Cribl.Cloud deployment, select a Edge Node and configure a new or existing [Cribl HTTP](sources-cribl-http) or [Cribl TCP Source](sources-cribl-tcp). These Sources enable internal data transfer between Cribl nodes to prevent data transfer billing costs.
To configure an HTTP or TCP Source, follow the same steps listed in the previous section about [Configuring HTTP or TCP Destination for the on-prem Edge Nodes](#on-prem-destination).
When you select or create the secret from the **Token secret (text secret)** menu, this secret must match the value of the secret used on the on-prem Cribl HTTP or Cribl TCP Destination.
Commit and deploy after making these configuration changes.
### Validate the Data Flow {#validate-data-flow}
As a final step, confirm that the data is properly flowing from the on-prem Edge Nodes to the Cribl.Cloud Edge Nodes. A few methods for confirming data flow include:
- Capture sample data on your Cribl HTTP or TCP Source on your Cribl.Cloud Edge Nodes and verify it looks correct. See [Add Sample Data](data-preview#add) for specific instructions.
- In the configuration modal for your Cribl HTTP or TCP Source on your Cribl.Cloud Edge Nodes, select the **Charts** tab to view data events coming in to verify you see activity. You can also compare this to the outgoing data on the Cribl HTTP or TCP Destination on your on-prem Edge Nodes.
############################################################
# SOURCE: https://docs.cribl.io/edge/tutorials.md
############################################################
# Tutorials
Follow these tutorials to get a one-stop overview of the interface
and data flow functionalities of Cribl Edge.
- [Route Collected Data](route-collected-data) explains the options for routing data collected by Cribl Edge to other products or Destinations.
- [Forward Custom Data with Cribl Edge Tags](usecase-custom-tags) shows how to add unique identifiers or metadata to each event for better control of data at scale.
- [Monitoring your Infrastructure with Cribl Edge](usecase-infrastructure-monitoring) presents how to monitor infrastructure by collecting, processing, and forwarding data with low overhead cost and resources.
- [Monitor and Troubleshoot Cribl Edge](usecase-monitoring-troubleshooting) explores key metrics to monitor for optimal Cribl Edge operation, discusses how to gather this information, and provides effective monitoring strategies and general troubleshooting guidelines.
- [Migrate from Third-Party Agents to Cribl Edge](usecase-migrate-to-edge) shows how to migrate your data collection infrastructure from third-party agents to Cribl Edge.
############################################################
# SOURCE: https://docs.cribl.io/edge/route-collected-data.md
############################################################
# Route Collected Data
Process and route data collected by Cribl Edge.
---
You can route data collected in Cribl Edge in several ways:
- **To Cribl Stream:** For further processing and scalable distribution. See [Send Data from Cribl Edge to Cribl Stream
](usecase-edge-stream).
- **To Cribl Search:** For analysis or visualization. Use the [Cribl Search Destination](destinations-cribl-search) or see [Collect and Visualize Nvidia GPU Metrics](collect-nvidia-gpu-metrics) for an example use case.
- **Directly to Destinations:** Such as cheap storage, SIEMs, or other security and IT tools.
Cribl Edge is not intended for extensive data manipulation.
Limit Pipelines on Edge Nodes to basic Cribl Functions. Avoid complex operations, to prevent bottlenecks and ensure optimal performance.
A typical workflow offloads major processing from Cribl Edge Nodes,
for example by:
1. Collecting data locally.
1. Filtering out unwanted data, using the [Drop Function](drop-function).
1. Routing the remaining data to Cribl Stream for further processing and delivery to your desired Destinations.
> You can check the [Pipeline Profiler](data-preview#pipeline-profiling) to access statistics on each Pipeline's performance.
{.box .success}
## QuickConnect vs. Routes in Cribl Edge
**QuickConnect** and **Data Routes** are both interfaces for managing data flow in Cribl Edge, but they serve distinct purposes, based on the complexity of your routing needs.
QuickConnect is ideal for simple, straightforward scenarios
with a given Source sending the same set of data through an optional Pipeline to one or more Destinations.
It offers ease of use and speedy configuration.
Data Routes, which are the default option, are better suited for complex data routing involving conditional processing.
They provide flexibility, advanced logic, and scalability for handling intricate data flows.
Choose QuickConnect for rapid, minimal configuration setups,
and opt for Routes when your data requires more sophisticated routing and processing.
## Processing Cribl Edge Data with Pipelines
Once you have collected your data, you have options to transform it before sending it to your Destinations.
You can transform data within pre-processing, processing, or post-processing Pipelines.
Some examples of common operations performed in Pipelines in Cribl Edge include
dropping events and enriching them with metadata.
### Drop Events to Lower Ingest Cost
By using the [Drop Function](drop-function), you can effectively eliminate data from your ingestion pipeline without incurring additional costs.
There are few ways you can take advantage of this Function:
- Use pre-processing Pipelines to filter and drop specific events, based on criteria like event codes.
- Use processing Pipelines to drop data before cloning events to send them to multiple Destinations. This reduces data volume.
- Safely experiment with configurations, without incurring costs, by routing data to the DevNull Destination.
> The number of dropped events that lower the ingestion cost
> is calculated based on average event size.
> This means it may not represent the exact number of bytes you drop.
{.box .info}
### Enrich Events by Adding Metadata
To enrich and refine your data within Cribl Edge, effectively use the [Eval Function's](eval-function) **Add Field** option. This feature enables you to dynamically create new fields based on existing data elements - such as tags from mapping rules, or event metadata. By converting tags and metadata to fields, you can enhance data analysis and manipulation capabilities.
For instance, a File Monitor might be collecting IIS logs from a Windows server. To ensure that these events are stored correctly in your final analysis system, you could specify a value of `index = 'IIS'`. This creates a new field named `index` with the value `IIS`, providing essential context for your data.
############################################################
# SOURCE: https://docs.cribl.io/edge/usecase-custom-tags.md
############################################################
# Forward Custom Data with Cribl Edge Tags
Add unique identifiers to events forwarded from Cribl Edge by using tags.
---
Organizations often need to add unique identifiers or metadata to each event forwarded from Cribl Edge to supported Destinations. Whether you're managing a small deployment or thousands of nodes, incorporating Edge tags into your events provides the context needed to make sense of your data at scale. These identifiers can help:
* Track the source of events across a large enterprise
* Associate events with specific departments, teams, or business units
* Maintain custom organizational taxonomy in log data
* Enable better filtering and searching in the Destination systems
This guide will show you how to set tags during Cribl Edge Installations and use the Preprocessing Pipeline to forward custom data points to a Destination.
## Embed Source Context with Cribl Edge Tags
In this guide, we will follow Data Detective Dave, an operator with a knack for getting to the bottom of data origins. His current case involves enriching data streaming from all Development servers within the Engineering Organization before it's dispatched to a central analytics platform. Dave needs to ensure every piece of data clearly shouts its server role and department.
Here are the steps Data Detective Dave will follow:
1. **Set meaningful tags while installing Cribl Edge:** Dave's first move is to tag each Development server right at the Source, while installing Cribl Edge. He'll use the installation script to attach labels `development` and `engineering` -- ensuring each server wears its identity from the get-go.
2. **Gather the metadata evidence:** Cribl Edge, like a good informant, automatically collects metadata, including the tags Dave just applied. He'll double-check that the `cribl` metadata option is active in the Fleet Settings, confirming that this crucial data is being gathered.
3. **Set Up the "Interrogation Room" (Preprocessing Pipeline):** Dave will create a new Pipeline in Cribl Edge, which he'll cleverly name `host_tags`. This will be his interrogation room where the data will be questioned and its secrets revealed (in this case, the tags).
4. **Use the "Eval-uator" to surface the clues (Eval Function):** Inside the `host_tags` Pipeline, Dave will employ an **Eval** Function -- his "Eval-uator." This function will dig into the `__metadata.cribl.tags` array, where the installation tags are stashed. It will then create new, easy-to-read fields in the data: `role` and `department`, clearly displaying the tag values.
5. **Apply the interrogation to the Source:** Next, Dave will apply the `host_tags` Pipeline to the specific Source(s) feeding data from the Development servers. This ensures that all incoming data undergoes interrogation and reveals its tags.
6. **Verify the findings with Live Capture:** To ensure his "Eval-uator" is working correctly, Dave will use the **Live Data** capture feature on the configured Source. By examining the data before it's routed, he can confirm that the `role` and `department` fields are present and accurate - case closed (almost)!
7. **Ensure the Destination doesn't lose the evidence:** Finally, Dave will make sure the central analytics platform -- the Destination -- is set up to receive and hold onto these newly revealed role and department fields. He might even need to set up specific routes to handle this tagged data appropriately.
If you'd like to follow along Data Detective Dave's journey, we outlined the detailed steps for you below.
### Set Tags During Edge Installation
When installing Cribl Edge, you can set tags during the installation. Regardless of your deployment environment (Docker, Linux, Windows, macOS, or Kubernetes), the [Bootstrap script via the UI](edge-nodes-add-update#bootstrap), gives you an option to add tags. These tags help identify and categorize your Edge Nodes such as environment type, geographic location, or server role.
As you modify the fields within the script interface, it updates in real-time, reflecting all the configuration choices you make on that screen, including any tags you've added. The result is a new Edge Node configured with the specific options you've selected.
In this example, we're setting two tags:
* Server Role: `development`
* Department: `engineering`
{border="true"}
You can also add tags after you install the Edge Nodes by teleporting directly into the Node. For details, see [Add Tags to Edge Nodes in Teleport View](managing-edge-nodes#add-tags-to-edge-nodes-in-teleport-view).
### Configure Host Info/Metadata Collection
Cribl Edge allows you to configure Event metadata sources to enrich events with additional context. In Edge mode, all the Event metadata sources are enabled by default.
The Event metadata sources add a `__metadata` property to every event emitted from enabled sources. This metadata can be used to:
* Enrich events with additional context.
* Display metadata for instance exploration.
Ensure the `cribl` metadata option is enabled by checking in:
1. Navigate to **Fleet Settings** > **Limits** > Metadata.
2. Use the **Event metadata sources** drop-down or the **Add source** button to enable and select metadata sources. Ensure the `cribl` option is enabled.
{border="true"}
### Create a Pre-Processing Pipeline
To add tags or other custom identifiers to events:
1. Navigate to your **Fleet** > **More** > **Pipelines** in your Cribl Edge interface.
2. Add a new Pipeline (e.g., `host_tags`). Pipelines ID can contain only letters (`A-Z`, `a-z`), numbers (`0-9`), and the characters `_` and `-`.
3. Add an **Eval** Function to the Pipeline.
4. Configure the Eval function to surface the tags by adding the following to the **Evaluate fields** section:
1. In the first row enter:
**Name**: `role`
**Value Expression**: `__metadata.cribl.tags[0]`
2. Select **Add Field** and enter:
**Name**: `department`
**Value Expression**: `__metadata.cribl.tags[1]`
{border="true"}
1. Save the Pipeline.
### Apply the Pre-Processing Pipeline
To make sure this Pipeline runs for all your data:
1. Go to **Sources > [Your Source]**.
2. In the Source configuration modal, find the **Pre-Processing Pipeline** option.
3. Select your newly created `host_tags` Pipeline.
4. Save the configuration.
### Verify the Output
To verify that your custom fields are correctly added:
1. On your Source configuration modal, select **Live Data** > **Capture**.
2. In the **Capture** modal, under **Where to capture?**, select `2. Before the Routes`.
3. Verify that the `role` and `department` fields appear in the events.
{border="true"}
### Destination Configuration
Ensure that your Destination configuration isn't dropping these custom fields:
1. Navigate to **Data** > **Destinations**.
2. Configure the Destination.
3. If needed, create a Route that specifically handles these tagged events.
4. Verify the data captured includes the additional metadata fields you configured.
############################################################
# SOURCE: https://docs.cribl.io/edge/usecase-infrastructure-monitoring.md
############################################################
# Monitor your Infrastructure with Cribl Edge
Monitor your infrastructure and visualize performance using Grafana.
---
Cribl Edge helps you monitor your sprawling infrastructure by collecting,
processing, and forwarding data with low overhead cost and resources. Cribl Edge
simplifies agent administration and data collection for your endpoints, and
provides a number of charts, graphs, and alerts for the web when connected to
supported data sources.
To enhance the performance of your environments, combine Cribl Edge with your
data visualization tool of choice (this guide uses Grafana as an example). Now,
you can identify and investigate issues and monitor applications faster than
ever before.
## Useful Metrics Dashboards
Cribl has configured a [Prometheus dashboard](https://grafana.com/grafana/dashboards/15280)
to display system metrics, which is available in Grafana's public library. This
is a relatively simple dashboard, suitable for showing basic aggregate metrics.
Another useful dashboard for displaying Linux metrics is the [Node Exporter Full](https://grafana.com/grafana/dashboards/1860)
dashboard, commonly used with Prometheus and its Node Exporter agent.
This dashboard can handle highly detailed metrics.
To display Windows metrics, consider one of Grafana's [Windows dashboards](https://grafana.com/grafana/dashboards/?search=windows).
## Guide Overview
This setup guide will walk you through these configuration steps:
1. Configure your Cribl Edge Sources. Depending on your endpoints, you can use
the [System Metrics Source](sources-system-metrics) for Linux,
[Windows Metrics](sources-windows-metrics), or
[Kubernetes Metrics](sources-kubernetes-metrics).
1. Configure the [Grafana Cloud Destination](destinations-grafana_cloud) to send
data to Grafana Cloud.
1. Access Grafana Cloud to see the metrics and visualizations.
## Configure the System Metrics Source on Cribl Edge {#source}
1. In Cribl Edge, start by configuring and enabling the applicable Source for system metrics:
- Linux [System Metrics Source](sources-system-metrics)
- [Windows Metrics](sources-windows-metrics)
- [Kubernetes Metrics](sources-kubernetes-metrics)
1. Decide what level of metrics detail you want to collect and
display.
In the [System Metrics Source](sources-system-metrics) and
[Windows Metrics](sources-windows-metrics) Sources, go to the **Host Metrics**
and **Container Metrics** tabs:
- For the relatively simple dashboard showing aggregate metrics, choose the
**Basic** mode for most of your metrics.
- For the highly detailed metrics visualizations dashboard, choose the
**All** mode for all of your metrics.
For the [Kubernetes Metrics](sources-kubernetes-metrics) Source, use the
**Filter Rules** to specify the Kubernetes objects that the Source should parse
to generate metrics. If you specify no restrictive rules here, the Source will
emit all events.
1. For all of the Sources, select **Processing Settings > Pre-Processing**, then
select the `prometheus_metrics` **Pipeline**.
1. When you're finished, **Commit** and **Deploy** the changes. Before you move
on to the next step, check the **Live Data** tab to make sure the metrics are
generated.
{border="true"}
{border="true"}
## (Optional) Adjust the `prometheus_metrics` Pipeline
Optionally, you can adjust the `prometheus_metrics` Pipeline to include the
`host` dimension, or to add/remove more dimensions.
Access the `prometheus_metrics` Pipeline by selecting **More** > **Pipelines** in the Fleet submenu.
{border="true"}
## Configure the Grafana Cloud Destination
Next, head over to [Grafana Cloud Destination](destinations-grafana_cloud) to
configure the Destination and send data to Grafana Cloud.
Preview your data on the Grafana Cloud config modal's **Live Data** tab.
{border="true"}
## Visualize the Data in Grafana
You can now visualize metrics in Grafana Cloud. To get started:
1. Navigate to your Grafana Cloud instance, and select **Explore** on the left
panel to review the Node metrics.
You can select metrics and run a query to produce a chart and table of metric
events, which you can add to new or existing dashboards for analysis. If
expected metrics are missing, try adjusting the variables in your Grafana
Cloud instance. See Grafana's
[Variables](https://grafana.com/docs/grafana/latest/dashboards/variables/)
topic for details.
1. Use Grafana's **Filter** option to browse the dashboards for your Prometheus
data. Below are sample dashboards with the basic and detailed metrics shown.
- To show basic aggregate metrics, use the [Cribl Node Observability](https://grafana.com/grafana/dashboards/15280) dashboard.
{border="true"}
- For the `OS` metrics with `CPU` / `Disk` details, the basic dashboard will
not suffice. Instead, we recommend configuring your System Metrics Source
(as discussed [above](#source)) to receive **All** metrics, and then using
the [Node Exporter Full](https://grafana.com/grafana/dashboards/1860)
dashboard below for visualization.
{border="true"}
############################################################
# SOURCE: https://docs.cribl.io/edge/usecase-monitoring-troubleshooting.md
############################################################
# Monitor and Troubleshoot Cribl Edge
Monitor and troubleshoot your Cribl Edge deployment.
---
Cribl Edge is a data collection agent for Windows, Linux, macOS, and Kubernetes. Even robust agents like Cribl Edge can encounter issues that disrupt data flow. To maintain optimal performance and reliability, it's essential to implement effective monitoring and troubleshooting strategies.
In this topic, we explore key metrics to monitor for optimal Cribl Edge operation, discuss how to gather this information, and provide effective monitoring strategies and general troubleshooting guidelines.
Additionally, we highlight the importance of integrating Cribl Edge with your existing IT monitoring and alerting tools to gain comprehensive visibility into its health and performance.
## Cribl Edge and Resource Constraints
Cribl Edge is designed to be a lightweight agent that often runs on machines that have limited resources. Unlike Cribl Stream, Edge uses a single Worker Process to limit resource usage and avoid affecting other functions of that machine.
While this is beneficial for resource-constrained environments, it's important to remember two things when implementing Edge:
* **Data consumption:** A single Worker Process can handle a limited amount data. If you try to ingest a massive amount of data, the process might become overloaded. For guidelines, see [Performance Considerations](deploy-planning#performance-considerations) and [Sizing and Scaling](/stream/scaling/).
* **Data manipulation:** Extensive manipulations within Pipelines can also strain the single Worker Process. Be strategic about the transformations you apply to the data.
## Key Metrics to Monitor
Cribl Edge offers you multiple ways to monitor your deployments in order to get an operational view of the health of your Fleets, Edge Nodes, Sources and Destinations, and more.
### Edge Node Status
The [Edge Nodes](managing-edge-nodes#view-edge-node-status-information) page provides status information for each Edge Node in the selected Fleet. You can use this page to quickly answer the following questions:
* Is the agent running?
* Is it connected to Cribl Leader?
#### Edge Node Connectivity
To verify the connectivity of your Edge Nodes, check the **Health** column in the **Edge Nodes** page. A status of "alive" indicates that the Edge Node is running and successfully communicating with the Cribl Leader.
For details, see [View Edge Node Status Information](managing-edge-nodes#view-edge-node-status-information) and [Tracking Data Activity for an Edge Node](node-data-activity).
#### Set Up Continuous Monitoring for Your Edge Nodes
While the Edge Nodes page provides essential information, it's important to note that disconnected Edge Nodes may be automatically removed after a certain period. To ensure continuous monitoring of your Edge Node health and performance, consider implementing centralized monitoring with a third-party solution like Grafana. By configuring your Edge Nodes to send [internal metrics](sources-cribl-internal) to a centralized monitoring solution, you can gain real-time insights into their statuses and troubleshoot potential issues proactively. For details and resources, see the [Centralized Monitoring](#centralized-monitoring) section below.
### Sources and Destinations Health
To maintain optimal performance and data reliability, monitor the health of your Sources and Destinations at both the Edge Node and Fleet levels. Monitoring at the Edge Node level provides granular insights into the performance of individual Nodes, while Fleet-level monitoring offers a broader view of overall system health and potential bottlenecks.
You can monitor and track the health of Sources and Destinations associated with an Edge Node on the **Edge Nodes** page. By filtering the list to display only Edge Nodes with "Warning" or "Error" statuses for their Sources or Destinations, you can quickly identify and troubleshoot potential issues. For details, see [Monitor Source and Destination Health on Edge Nodes](managing-edge-nodes#monitor-source-and-destination-health-on-edge-nodes) and [Filter Edge Nodes](edge-nodes-filter).
Alternatively, you can monitor the health of Sources and Destinations within a specific Fleet. To do this, navigate to the desired Fleet, select **Health**, and then choose either **Sources** or **Destinations**. This will display a color-coded health status for each Source or Destination, providing a quick visual overview of their operational status. For details, see [Overall Source and Destination Health.](monitoring#overall-source-and-destination-health)
### Data Throughput
There are several pages you can use to monitor the rate at which data is collected and transmitted (data throughput) across your Edge Nodes. This information can help you determine if an Edge Node is capable of handling the current data generation rate and if data is being successfully transmitted to Destinations.
* **Cribl Edge home page:** The Cribl Edge home page displays the aggregate data for all Fleets, Subfleets, and Edge Nodes. The charts display information about traffic in and out of the system. For details, see [Monitor Health on the Edge Home Page](monitoring#monitor-health-on-the-edge-home-page).
* **Fleets page:** The **Fleets** page gives you access to more information about each of your Fleets and the health of Sources and Destinations associated with the Fleets. You can view the **Monitor** tab of a selected Fleet to analyze the traffic flowing through your Fleet. For details, see [Monitor Fleet Information](monitoring#monitor-fleet-information).
* **Edge Nodes page:** To monitor the activity of your Edge Node and identify potential issues with Sources, Destinations, Routes, Pipelines, or Packs: Navigate to the **Edge Nodes** page, pick a specific Node and click anywhere on the row to launch the **Node Info** modal. Then, select **Data Activity**. This will provide detailed insights into the Node's data flow without teleporting into the Node itself. For details, see [Viewing the Data Activity for a Node](node-data-activity#viewing-the-data-activity-for-a-node).
### Resource Utilization
To effectively monitor the health and performance of your Edge Nodes, use node and process information as well as details from logs.
Sources of node and process information include:
- **System Activity** tab: Gain a detailed view of your Edge Node's health, including CPU usage, memory consumption, network traffic, and disk operations. Please note that these metrics represent the total system usage and may not solely reflect the resource utilization of the Edge Node itself. To view **System Activity**, click the row for an Edge Node. For details, see [Viewing the System Activity for a Node](node-data-activity#viewing-the-system-activity-for-a-node).
- **Process-Level Monitoring**: Delve deeper into specific processes, such as `cribl` processes, to pinpoint resource bottlenecks and optimize performance. For details, see [Processes](explore-edge-linux#processes).
Logs to analyze include:
- **API Server Logs**: Examine the `cribl.log` file at `$CRIBL_HOME/log/cribl.log` for high-level system telemetry and license validation information.
- **Worker/Edge Node Process Logs**: Analyze the `cribl.log` files at `$CRIBL_HOME/log/worker/N/cribl.log` to scrutinize detailed resource utilization metrics for each Worker/Edge Node Process at a granular one-minute interval.
Combine these approaches to proactively identify and address potential performance issues and ensure optimal data processing and system stability.
## Effective Monitoring Strategies
This section explores various strategies for effectively monitoring your Cribl Edge deployments. We'll delve into Edge Node logging, centralized monitoring, and network monitoring techniques to ensure optimal performance and data flow.
### Edge Node Logging
Cribl Edge logs at the default `Info` level, providing a good balance between detail and performance for most environments. Analyze logs regularly to identify trends and potential issues.
If you encounter a specific issue or need more detailed information, Cribl Edge settings allows you to adjust the log level for individual channels. However, frequently changing log levels from the default `Info` might indicate an underlying issue requiring further investigation.
By carefully selecting the appropriate log level, you can obtain valuable troubleshooting information without compromising performance. For details, see [Logging Settings](monitoring#log_settings).
### Centralized Monitoring
Centralized monitoring allows you to efficiently collect and analyze data from multiple Edge Nodes. This approach helps you proactively detect issues by enabling you to:
* **Implement alerts:** Set alerts for critical metrics, such as agent failures or performance degradation.
* **Create dashboards**: Visualize key metrics and gain valuable insights into the overall system health.
Combine Cribl Edge with your existing IT monitoring tools to gain a unified view of your deployment's health. This allows you to identify and address issues before they impact performance. By sending Cribl Edge logs to platforms like Elastic, Grafana, or Loki, you can:
* **Centralize log management:** Consolidate logs from various Sources for efficient analysis and troubleshooting.
* **Create advanced dashboards:** Visualize complex metrics and correlations to gain deeper insights.
* **Implement sophisticated alerts:** Set up alerts based on specific conditions and triggers to proactively address issues.
For details, see [Monitoring your Infrastructure with Cribl Edge](usecase-infrastructure-monitoring). Additionally, you can use [CriblVision](https://github.com/criblio/criblvision/tree/main) to troubleshoot specific issues in CriblSearch, Grafana, and Splunk.
### Network Monitoring
Monitor network traffic between the Edge Node and the Cribl Leader to identify and resolve network-related issues. Use third-party network monitoring tools to track network performance and troubleshoot connectivity problems efficiently.
### Troubleshooting Edge Nodes
To troubleshoot potential issues with your Edge Nodes, consider the following:
* Start by verifying the Edge Node's configuration, including connection settings, data collection intervals, and permissions.
* Analyze the logs in Cribl Edge. For details, see [Search Internal Logs](monitoring#log_search). To isolate issues related to a specific Edge Node, filter the logs by the Edge Node's IP address. This will help you identify any error messages or exceptions that may require attention.
* Check network connectivity between the Edge Node and the Cribl Leader by reviewing the Health column on the Edge Nodes page.
* To monitor Edge resource utilization, examine the API server logs and Worker Process logs. These logs provide detailed information on CPU usage, memory consumption, network traffic, and other relevant metrics. You can also monitor the System activity for the entire host by [Viewing the System Activity for a Node](node-data-activity#viewing-the-system-activity-for-a-node).
Refer to [Common Challenges on the Edge](edge-common-challenges) topic for other troubleshooting tips.
############################################################
# SOURCE: https://docs.cribl.io/edge/usecase-migrate-to-edge.md
############################################################
# Migrate from Third-Party Agents to Cribl Edge
Migrate your data collection infrastructure from third-party agents to Cribl Edge
---
This guide provides strategic considerations for migrating your data collection infrastructure from traditional third-party agents to Cribl Edge. Whether you're migrating from Splunk Universal Forwarders (UFs), Fluentd, Filebeat, or other agent technologies, the fundamental migration principles remain consistent.
Any migration to Cribl Edge generally follows the same path:
1. Decide on your migration strategy
2. Complete pre-migration planning
3. Implement your Edge configuration
4. Rollout your changes in a controlled environment
We'll examine each phase and what you need to consider along the way.
> This is a considerations guide, not step-by-step instructions. Complex migrations require careful planning and benefit from professional services engagement.
>
{.box .info}
For specific details on UF migrations, see: [Migrating from Splunk Universal Forwarders to Cribl Edge](https://knowledge.cribl.io/edge-55/migrating-from-splunk-universal-forwarders-to-cribl-edge-1722).
## Migration Strategy Overview
Your migration success depends heavily on choosing the right approach from the start. The complexity of your current deployment, available resources, and timeline constraints will all influence which path you should take. Understanding these options upfront will save time and reduce risk throughout the migration process.
### Choosing Your Migration Approach
Before beginning any migration, you must decide between two primary approaches:
- Customer-managed migration
- Professional services engagement
#### Customer-managed Migration
* **Best for:** Simple deployments with limited configuration complexity.
* **Challenges:** Trying to migrate complex agent deployments, beyond simple deployments, can be challenging and time-consuming. Requires deep understanding of both current infrastructure and Cribl Edge architecture.
* **Suitable for:** Organizations with dedicated resources and Cribl Edge expertise, or proof-of-concept implementations.
#### Professional Services Engagement
* **Best for:** Large-scale or complex agent deployments.
* **Provides:** Automated migration tools, in depth analysis, and enhanced support.
* **Recommended for:** Organizations requiring minimal downtime, deployments with extensive customization, time-sensitive migrations.
## Pre-Migration Planning
This phase involves understanding your current environment, identifying potential challenges, and ensuring you have the necessary resources in place.
### Environment Assessment
The first step in migration planning is gaining a complete understanding of your current deployment. This comprehensive assessment will inform every subsequent decision and help identify potential migration challenges before they become problems.
Begin by conducting a thorough inventory of your current deployment, document both essential infrastructure components and organizational factors that could impact your migration strategy.
**Required inventory items:**
* OS types and versions across your infrastructure.
* Current data volumes and peak load patterns.
* Existing input configurations and custom settings.
**Additional considerations that may impact your migration:**
* Company span, subsidiaries, separate entities.
* Network topology, connectivity, and bandwidth constraints.
* Change control processes for your infrastructure.
* Rollback plan documentation before beginning the migration.
### Configuration Analysis
The goal of configuration analysis is to determine the data flow and topology of your current system, so that you can use this information to inform your Fleet design. For example, in environments using centralized configuration management (like Splunk's Deployment Server), the way your current server classes are organized will significantly impact your Edge Fleet design decisions.
Key areas to document include:
* Centralized configuration management systems and their hierarchies
* Application or configuration packages deployed to agents
* Client-to-configuration mappings
* Custom configurations and transformations
For example, in Splunk UF environments specifically, you would examine:
* Deployment Server configuration using serverclass.conf
* Splunk Apps stored in $SPLUNK\_HOME/deployment-apps
* Client-to-serverclass mappings
### Resource Requirements Planning
Inadequate resource planning is one of the most common causes of migration failures. Cribl Edge introduces new components and data flow patterns that require careful capacity planning. Understanding these requirements upfront prevents performance issues and data loss during the migration.
#### Leader Scaling
For Cribl.Cloud deployments, first assess your current Leader's performance before adding Edge Nodes. Evaluate whether your Leader is already overtaxed or operating smoothly, examining CPU utilization, memory usage, disk space availability, and disk I/O performance. For details, see [Monitoring Health and Metrics](monitoring).
Once you've confirmed your Leader can handle additional load, ensure Leader resources are scaled to support expected Fleet/Worker Group counts. To spin up the right number of connection processes, add the number of Edge Nodes you will deploy into the [Edge Nodes Count field](deploy-planning#configure-edge-node-count). This field crunches numbers in the background and automatically spins up the correct number of Edge connection processes for you. For on-prem deployments as well as other considerations, see [Deployment Planning](deploy-planning).
#### Cribl Stream Worker Nodes
Ensure Stream Worker Nodes receiving data from Edge Fleets are properly sized for peak load. Undersized infrastructure can cause performance bottlenecks and data loss during high-volume periods.
#### Data Routing and Flow
Cribl Edge offers more sophisticated data routing capabilities than traditional agents, but this flexibility requires careful planning to avoid performance bottlenecks and ensure reliable data delivery.
Key considerations:
* Configure appropriate outputs based on your data routing needs
* Consider routing complexity when routing from Cribl Edge to Cribl Stream then to Destinations
* Plan for backpressure management, especially when routing to multiple Destinations
## Understanding Architectural Differences
Most third-party agent migrations involve fundamental architectural changes that affect every aspect of your migration planning. Understanding these differences is critical for designing an effective migration strategy.
### Configuration Management Models
Traditional agents often use different configuration management approaches than Cribl Edge.
#### Composition vs. Inheritance
* **Traditional agents:** Many use composition models where a single agent can belong to multiple configuration groups or classes
* **Cribl Edge:** Uses inheritance where one Edge Node belongs to a single Fleet/Subfleet
For example, in Splunk UF environments, one UF can belong to multiple serverclasses, but in Cribl Edge, one Edge Node belongs to a single Fleet/Subfleet.
#### Fleet Design Strategy
Fleet design is perhaps the most critical decision in your migration. Poor Fleet architecture can lead to management complexity, performance issues, and operational challenges. The goal is to balance functionality with maintainability while leveraging Cribl Edge's strengths.
##### Fleet Rationalization Approach
* Review your current configuration management to understand what needs to be collected
* Use tags, Pipelines, and Cribl Stream to reduce configuration variability
* Consolidate similar configurations to minimize Fleet count
* Group by function rather than direct configuration translation
##### Fleet Count Optimization
* Keep Fleet counts to a minimum for optimal performance
* Avoid direct translation of existing configuration groups to Edge Fleets (leads to Fleet proliferation)
> Direct translation of existing configuration management structures to Edge Fleets often leads to management complexity and potential performance issues.
>
{.box .warning}
* Consider creating Fleets for unique combinations of configuration requirements only when necessary
### Configuration Translation Challenges
Understanding these caveats upfront allows you to plan alternative approaches and avoid migration roadblocks.
#### Automatic Metadata Population
One advantage of Cribl Edge is its automatic handling of common metadata fields, reducing configuration complexity compared to traditional agents. Cribl Edge automatically populates these metadata fields without additional configuration:
* `sourcetype`
* `index`
* `source`
* `host`
* `connection_host`
### Configurations Requiring Alternative Approaches
Some agent configurations have no direct equivalent in Cribl Edge and require additional steps. Common examples include:
* **Path-based host extraction:** Some agents (like Splunk UF) can automatically extract the hostname from the file path itself. For example, the `host_regex` and `host_segment` settings from UF don't translate directly. However, you can achieve the same functionality using Cribl Edge's Pipeline processing to extract and transform the hostname data as needed.
* **File handling behaviors:** Some agents often have granular control over how they interact with files. Cribl Edge may not have equivalent settings for every file handling behavior. For example, symbolic linking or file opening behavior.
* **Custom transformations:** Some agents can perform data transformations at the collection point. Cribl Edge operates on a different model where most data transformation happens in Cribl Stream rather than at the edge. While Cribl Edge can do some processing, it's not designed to replicate all the transformation capabilities that agents might have built-in.
#### Alternative Implementation Methods
* Create corresponding Pipelines in Cribl Edge
* Use Edge Node metadata mapping for system information
* Implement data transformation through Routes (all inputs forward to Routes)
## Migration Implementation Considerations
Once you've completed your planning phase, the implementation phase requires attention to configuration details and platform-specific considerations.
### Configuration Review Areas
Each configuration area requires careful review to ensure optimal performance and functionality.
#### File Monitor Settings
* Determine whether to start from the beginning or end of files
* Consider the impact of large file sizes or many old events
* Assess network bandwidth and Destination capacity implications
#### System State Configuration
The System State Source is enabled by default on Cribl Edge Fleets, but you can disable the Source or specific collectors if System State data is not needed.
#### Pipeline Configuration
* Review existing agent configurations and enable similar functionality in Cribl
* Optimize regexes for performance
* Consider data transformation requirements previously handled in agents
#### Persistent Queue Management
* Review Persistent Queue settings in your agents, Cribl Edge, and Stream
* Ensure proper queue sizing for expected data volumes
### Platform-Specific Considerations - Windows Environments
Windows environments present unique challenges during migration, particularly around policy management.
#### Group Policy Objects
Check for Group Policy Objects (GPOs) that might automatically reinstall or enable removed third-party agents, and update policies before migration to prevent conflicts.
#### PowerShell Usage
For optimal performance on Windows Edge Nodes, ensure the "Use Windows Tools" setting is disabled for all data Sources. When disabled, the system uses native Windows Management Instrumentation (WMI) API methods for faster and more reliable data collection. When enabled, the system falls back to legacy PowerShell cmdlets.
The PowerShell-based collection methods will be removed in future releases as native methods provide superior speed and reliability.
#### Policy Management
Some sites might have policies to re-enable/re-install agents if missing/removed. Update or exclude policies before migration.
#### Windows Event Log Read Mode
When configuring the **Windows Event Log** Source, always review the **Read mode** setting. The default is **From last entry**, which only collects new events. While this is the recommended setting, you have the option to change it to read the **Entire log**. For machines that have been in use for a long time, this can cause a massive, one-time spike in data ingestion as it sends years of historical events. This sudden influx can easily overwhelm your infrastructure, causing performance issues and data loss. Sticking with the default setting helps you avoid this risk.
### Addressing Inheritance Limitations
When working with Edge's inheritance model, be aware of these challenges:
* **Configuration integrity**: Changes to a Source/Destination in a Subfleet can break inheritance from the parent
* **Limited flexibility**: Sources that allow only one configuration at a time create inheritance challenges when customization is needed
## Migration Rollout Strategy
The rollout phase is when your plans become reality. A good rollout ensures things go smoothly, problems are fixed fast, and your data collection isn't interrupted.
### Controlled Rollout Process
A controlled rollout allows you to validate your migration approach on a small scale before committing to full deployment. This approach significantly reduces the risk of widespread issues and provides opportunities to refine your process.
#### Progressive Batch Sizing
* Start with test servers
* Then proceed to small batches (e.g., 200 machines)
* Progress to larger batches (400, 800, 2000 machines, etc.)
* Validate system performance and data flow at each stage
* Only proceed when previous stage is validated
#### Validation Requirements
* Execute controlled rollout with proper validation at each stage
* Address any issues early in rollout before expanding
* Maintain visibility into migration rollout progress
### Continuous Monitoring
To ensure a smooth migration, you need to monitor everything. This means watching both how things are performing technically and making sure your important data is flowing correctly. Additionally, monitor the Edge Nodes themselves as they get onboarded to track their connectivity status. Cribl Edge v.4.13 introduced Disconnected Node Tracking to help you identify and troubleshoot nodes that lose connection during the migration process.
#### Resource Utilization
Monitor Cribl Edge, Leader, and Stream components continuously to watch for performance bottlenecks or data loss indicators. Validate data flow and collection accuracy throughout the process.
#### Rollback Capabilities
Maintain (previously-documented) rollback capabilities in case issues are discovered.
## TL;DR
Migrating from third-party agents to Cribl Edge requires careful planning and a systematic approach. The key to success lies in understanding the architectural differences between your current system and Cribl Edge, properly planning your Fleet design, and executing a controlled rollout process.
Remember that this is a considerations guide, and complex migrations often benefit from professional services engagement to ensure success and minimize risk to your data collection infrastructure.
############################################################
# SOURCE: https://docs.cribl.io/edge/deploy-planning.md
############################################################
# Deployment Planning
Plan your Cribl Edge deployment, taking into account requirements, deployment types,
scaling, and billing.
---
When you're planning an Edge deployment, consider these key factors:
* The amount of data (metrics, logs, custom command outputs, and so on) that you
plan for Edge to collect from the endpoint and ingest per unit of time. For
example, MB/s or GB/day.
* The amount of processing that will happen on incoming data. For example, are
there a lot of transformations, regex extractions, parsing functions, field
obfuscations, and so on?
* Routing and/or cloning: Is most data going to a single Destination, or is it
being cloned and routed to multiple places? This is important, because
Destination-specific serialization tends to be relatively expensive.
* Deploying onto servers with no internet access: If you plan to deploy
Edge Nodes on air-gapped on-prem servers (servers with no internet access),
you must ensure that every Edge Node can communicate with a Leader that is
also on-prem (that is, not in Cribl.Cloud).
> See [Requirements](requirements) for detailed information about system and OS requirements for a Cribl Edge deployment.
{.box .info}
## Types of Deployment
- **[Cribl.Cloud](deploy-cloud)**: Quickly launch a Cribl-managed
deployment of the combined Cribl applications suite (Edge, [Stream](/stream/),
[Search](/search/), and [Lake](/lake/)). In Cribl.Cloud, Cribl provisions and manages all
infrastructure on your behalf.
- **Managed Edge**: A central Leader Node controls, manages, and pushes
configuration updates to all managed Edge Nodes. This deployment type provides
a centralized view for multiple Edge instances, which is beneficial for
large-scale deployments of Cribl Edge.
- **Single-instance**: Single-instance Edge Nodes operate without a central
Leader. You will configure and manage Edge Nodes locally.
Single-instance Edge is suitable for smaller deployments or environments with
limited network connectivity.
When you have a large number of Edge Nodes to manage, you can use Fleets to
group Nodes together and apply configurations centrally. Use [Fleet Management](fleet-management)
to accommodate increased load. See [Add and Update Edge Nodes](edge-nodes-add-update#bootstrap)
to streamline deploying Edge Nodes via scripting.
## Leader Scalability {#scaling}
Our [System Requirements](#requirements) suffice for on-prem Leaders handling up to 10,000 Edge Nodes.
To support more than 10,000 Nodes, see our [Planning Large-Scale Deployments of Cribl Edge](how-to-scale-edge) guide.
For large on-prem deployments handling high traffic, review the recommended OS-level tuning detailed in [OS Tuning for Large Deployments](os-tuning).
## About Billing and Usage
Cribl.Cloud offers a one-page view of your total credits consumed and remaining
credits for the current active contract in place. This is called Cribl FinOps
Center. Here, you can view per-product consumption for the Cribl products you
use. The **Invoices** tab lets you download and export your usage by month. For more
details, go to the [FinOps Center](/billing-licensing/finops-center) docs.
### On-Prem Licenses
If you're deploying on-prem, there are three license options. The [Cribl Plans](https://cribl.io/pricing/edge/?product=software-license) page provides details feature comparisons for each license.
You can configure [Connected Environments](cloud-connections) to use the
Cribl.Cloud FinOps Center billing model with your on-prem deployments instead of
managing multiple licenses.
## Permissions and Rights
You do **not** need elevated privileges to install Edge. However, you will need:
* Administrator (or equivalent) rights in order to enable Edge to start on boot.
* Sufficient rights to access the resources that need monitoring - files, metrics, and so on.
> ##### Network Requirement (DPI/IPS Exclusion)
> Deep Packet Inspection (DPI) and Intrusion Prevention Systems (IPS) on network devices situated between your Edge Fleets can cause high resource utilization and connection instability in all environments (both Cribl.Cloud and on-prem).
> Explicitly exclude all Cribl data streams from DPI/IPS inspection for stable operations and optimal performance.
{.box .warning}
## Performance Considerations
As with most data-collection and -processing applications, Cribl Edge's expected resource utilization will be proportional to the data volume and type of processing. For instance, a Function that adds a static field on an event will perform faster than one that applies a regex to find and replace a string.
* Processing performance is proportional to CPU clock speed.
* All processing happens in-memory.
* Processing does not require significant disk allocation.
Thinking about your planned Edge deployment as a whole, it's critical to consider **cardinality**: how many Fleets, of what size, will send metrics to a given Leader. In a very low-cardinality deployment, a Leader Node might aggregate metrics from one small Fleet; in a very high-cardinality deployment, a Leader Node might aggregate metrics from many large Fleets.
Too many Edge Nodes sending too many metrics can degrade Leader performance and/or integrity. By carefully choosing what metrics each Fleet sends to its Leader Node, you can ensure that you collect the metrics most important to you, within the limits of the Leader's capacity to process them.
As part of your deployment planning, decide which of the following four sets of metrics each Fleet in your deployment will send to its Leader Node:
* **Minimal**: Limited to metrics for events and bytes in and out. Recommended for high-cardinality deployments.
* **Basic** (the default set): All the metrics required to display all monitoring data available in the Edge UI. Contains all the metrics in the **Minimal** set, and more.
* **All**: All metrics, sent without filtering.
* **Custom**: Only metrics that match a JavaScript expression that you define.
For a listing of the metrics each set provides, see [Control Metrics in Cribl Edge](internal-metrics#edge-metrics-controls).
## Configure Edge Node Count
Enter the estimated number of Edge Nodes in this deployment, which is used to
calculate how many Edge-specific connections service processes to spawn.
Changing this number could restart any currently running service processes. If
left blank, defaults to 0.
To configure the Edge Node Count:
1. From **Settings**, select the **Global** tab, then select **General Settings**.
1. In an on-prem deployment, select **Limits**, and then **Others**.
1. Enter the number of Edge Nodes in the **Edge Nodes count** field.
## Configure Connection Processes {#connection}
To configure the connection processes for your Edge Nodes in an on-prem deployment:
1. From **Settings**, select the **Global** tab, select **Service Processes**, and then select **Services**.
1. Confirm that **Connections listener number of processes** is set to its default `1`.
> The **Connections listener number of processes** is automatically determined by the **Edge Node Count** setting you enter above. Any value you enter manually will be overridden by this calculation.
{.box .warning}
############################################################
# SOURCE: https://docs.cribl.io/edge/requirements.md
############################################################
# OS and System Requirements
For a successful Cribl Edge deployment, verify that your system meets the minimum hardware and software specifications outlined below.
> All the quantities listed below are **minimum** requirements.
> Adjust them as needed based on your specific use case.
>
{.box .info}
## OS Requirements
You can install Cribl Edge Nodes on Linux, Windows, and macOS systems.
### Linux Requirements
Cribl Edge Nodes can run on any common Linux distribution that fulfils the following criteria:
- 64-bit kernel >= 3.10
- glibc >= 2.17
- OpenSSL >= 3.0 (if FIPS mode is enabled)
Highly customized or rare Linux distributions might not be supported even when based on a supported kernel version.
Enforcing mode in SELinux is supported, but not required.
#### Tested Linux Platforms
Cribl tests the following Linux versions to confirm compatibility with the supported kernel.
This list does not constitute a comprehensive record of all supported distributions and versions.
- Ubuntu 24.04
- RHEL 9, 9.3
- CentOS Stream 9
- SUSE Linux Enterprise Server 15.6
- Amazon Linux 2023
> As of October 16, 2024, Cribl has deprecated support for the CentOS Linux 7 distribution, which reached its end of life on June 30, 2024.
>
{.box .info}
### Windows Requirements
Cribl Edge supports these versions of Windows:
|Version
|Notes|
|-------------|----------------|
|Windows 10 and 11| You can deploy Cribl Edge on laptops, desktops, and workstations running Windows 10 and 11.
Keep these things in mind when deploying Cribl Edge on Windows 10 and 11 workstations:- Cribl Edge might lose some data sent to TCP Destinations after a workstation resumes from a suspended or sleep state. This happens when the data is in local network buffers but hasn't been sent out to the Destination. Instead, you can send data to an HTTP Destination.
- You might see partially broken events after a laptop resumes from a suspended state.
- HTTP Destinations might contain duplicate events.
- Some users have observed excessive logging when a Leader is unavailable to receive heartbeat messages from an Edge Node.
|
| Windows Server 2016
Windows Server 2019
Windows Server 2022
Windows Server 2025 | Functionality is the same as generally supported Windows Server platforms. |
### MacOS Requirements {#macosreq}
Cribl Edge supports the following macOS versions:
26 (Tahoe), 15 (Sequoia), 14 (Sonoma).
You can install Cribl Edge on both Intel and Apple silicon processors.
## System Requirements
Edge Nodes should have sufficient CPU, RAM, network, and storage capacity to handle your specific workload.
While minimum specifications are provided, optimal configurations vary widely.
Consider data volume, types, processing needs, and organizational standards when selecting hardware.
Make sure to test the performance of your Edge Nodes before deploying to production.
Minimum application requirements for Edge Nodes are:
- ~1 Ghz processor
- 1024 MB RAM
- 5 GB free disk space per volume for operational purposes
If your installation uses multiple volumes
(for example, one volume to store [persistent queues](persistent-queues)
and one for Destination [staging directories](destinations#staging-directory)),
5GB is the minimum disk space for each of those volumes.
You can configure the disk space limit per Fleet with the [**Min free disk space**](fleet-settings#storage) setting.
> See also [Cribl Stream Requirements](/stream/requirements) to learn about the requirements for the Leader Node.
{.box .info}
## Browser Support {#browser}
The Cribl Edge UI can run on Chrome, Firefox, Safari, and Microsoft Edge, on their five most-recent versions.
############################################################
# SOURCE: https://docs.cribl.io/edge/how-to-scale-edge.md
############################################################
# Plan Large-Scale Deployments of Cribl Edge
Learn how to scale on-prem Cribl Edge deployments to support one Leader managing up to 250,000 Nodes.
---
Our standard [Deployment Planning](deploy-planning) guide is a great place to
start if you intend to deploy 10,000 or fewer Nodes in your Cribl Edge environment.
For larger deployments, an on-prem Leader can support a maximum of 250,000 Edge Nodes
when metrics are set to **Minimal** mode and 100,000 Edge Nodes with metrics set
to **Basic**.
This guide covers:
- The specific Leader configuration we recommend for best performance.
- Overview of upgrading a large Cribl Edge deployment.
- Limitations to the UI resulting from a large number of Edge Nodes.
## Recommended On-Prem Leader Configuration
The base configuration detailed on [Cribl Stream Sizing and Scaling](/stream/scaling#vms)
is a good starting point.
From there, you can configure these settings on the on-prem Leader so it can manage up
to 250,000 Edge Nodes:
- Add 2GB of RAM for every 10,000 Edge Nodes.
- Deploy one vCPU core for every 10,000 Edge Nodes.
- For Cribl Edge to support up to 100,000 Nodes, you can [configure internal metrics](internal-metrics#edge-metrics-controls) to **Basic** in **Fleet Settings**.
- For 250,000 Nodes, you must configure internal metrics to **Minimal** in **Fleet Settings**.
>You should configure these metrics for every Fleet if your Edge Node count
>is more than 10,000. See [Performance Considerations](deploy-planning#performance-considerations)
>for more information.
{.box .info}
- To spin up the right number of connection processes, add the number of Edge
Nodes you will deploy into the [**Edge Nodes Count**
field](deploy-planning#configure-edge-node-count).
This field crunches numbers in the background and automatically spins up the
correct number of Edge connection processes for you.
- There is no change to the default values when pushing config bundles from the Leader.
## Upgrading a Large Number of Edge Nodes
The [upgrading framework](upgrading) allows Edge Nodes to pull down upgrade
packages from the Leader asynchronously, rather than the Leader pushing packages
to Nodes all at one time. Nodes communicate to the Leader every 60 seconds via a
heartbeat.
Additionally, Nodes request upgrade packages from the Leader in batches of 500.
This allows you to upgrade Nodes without overloading the Leader, even with a
large number of Nodes.
>In our controlled test environment, we found that
>upgrading can take 25-40 minutes. This is an expected time frame
>to upgrade that number of Nodes.
{.box .info}
## User Interface Limitations
For Fleets with a large number of Edge Nodes, the UI is limited in certain places
where a complete list of Nodes can't be displayed.
### Searching for Nodes in the Explore Tab
The [**Node**](explore-nodes) field in the **Explore** tab returns only the first
100 matching Nodes. When you start typing, the results match the search query
and are sorted alphabetically by hostname.
### Previewing Node Mappings
Preview Node mappings by selecting **Mappings** in the Leader Node's sidebar. (Only the first 10,000 mappings are displayed.)
### Other Limitations
- **Logs**: The drop-down where you can select an Edge Node will display a
maximum of 100 Nodes.
- **Import Edge Data**: The **Import Edge Data** window in Cribl Stream returns a
maximum of 100 Edge Nodes.
- **Test tab**: The **Test** tab on Destinations displays a maximum of 100 results.
- **KMS Status report**: In **Fleet Settings**, the **KMS Status report** (under
**Security**) does not display all Nodes. Instead, search for your desired
Node.
############################################################
# SOURCE: https://docs.cribl.io/edge/setting-up-leader-and-edge-nodes.md
############################################################
# Set Up Leader and Edge Nodes
Configure a Leader Node and a Cribl Edge Node.
---
This page covers how to:
- [Configure a Leader Node](#config-leader)
- [Configure an Edge Node](#config-edge)
## Configure a Leader Node {#config-leader}
You can configure a Leader Node either through the UI or through the `instance.yml` config file.
### Use the UI
1. Navigate to **Settings** > **Global** > **Distributed Settings** > **General Settings**.
1. Select **Mode**: **Leader**.
1. Select the **Leader Settings** menu.
1. Confirm or enter the required Leader settings (**Address** and **Port**).
2. Confirm Leader API Access: Toggle **Forward to Leader API** on (default) to enable API requests on the distributed port and provide backward compatibility.
3. Customize the optional settings if desired.
4. Select **Save** to restart the Cribl Edge server.
### Add Authentication Tokens
To enable the Leader to communicate with Edge Nodes, you need to create an authentication token.
A provisioning token lets you deploy multiple Edge Nodes.
See [Create a New Provisioning Token](securing-auth-token#create) for information on how to create and manage authentication tokens.
### Edge Node UI Access {#edge-access}
This useful option enables you to click through from the Leader's **Edge Nodes** page to an authenticated view of each Node's UI. We call this [Teleporting into an Edge Node](managing-edge-nodes#teleport-into-an-edge-node).
The instructions below correspond to enabling the `groups.yml` file's `workerRemoteAccess` configuration key.
To enable teleporting (remote UI access) to Nodes from the Leader's UI:
1. Select **Fleets** from the sidebar.
1. For each desired Fleet, toggle on **Teleport**.
{border="true" scale="70%"}
3. From the sidebar, select **Edge Nodes**.
4. On the **Edge Nodes** page, click the link for any Node you want to teleport
into.
To confirm that you are remotely viewing a Edge Node's UI, Cribl Edge
displays a purple border, with a badge labeled **Viewing host:**
``.
{border="true"}
> The Leader will override any changes that you make directly to this Edge Node.
{.box .warning}
#### Use the YAML Config File
In `$CRIBL_HOME/local/_system/instance.yml` (`C:\ProgramData\Cribl\local\_system\instance.yml` on Windows), under the `distributed` section, set `mode` to `master`:
```yaml {title="$CRIBL_HOME/local/_system/instance.yml or C:\ProgramData\Cribl\local\_system\instance.yml"}
distributed:
mode: master
master:
host:
port: 4200
forwardToLeaderApi: true
tls:
disabled: true
ipWhitelistRegex: /.*/
authToken:
enabledWorkerRemoteAccess: false
compression: none
connectionTimeout: 5000
writeTimeout: 10000
```
### Persisting Socket Connections {#socket}
A distributed deployment creates socket files for inter-process communication (IPC) between the Leader and distributed processes and services. These sockets are essential for ensuring that Edge Nodes successfully connect to the Leader, and for certain metrics services. On the Leader's host, the default location for these files is the operating system's temp directory (for example, `/tmp`).
Many Linux distros maintain a system cleaner service (for example, [systemd-tmpfiles](https://man.archlinux.org/man/systemd-tmpfiles-clean.service.8)) that removes files from this directory periodically, such as every 10 days. If Cribl's sockets are removed, this breaks certain UI pages, such as those for [Fleets](managing-edge-nodes#fleets-tab) and [Monitoring](monitoring). You can protect the sockets in either of two ways.
#### Block the Cleaner {#block}
Stop the host OS from cleaning socket files out of `/tmp/cribl-*` subdirectories. For example, on Amazon Linux 2 instances, add a new `tmp.conf` file to `/etc/tmpfiles.d` with the line: `X /tmp/cribl-*`.
To restart the system cleaner here and reload its configuration, use this command:
```shell
sudo systemctl restart systemd-tmpfiles-clean.service
```
#### Move the Sockets {#move}
Alternatively, you can move Cribl's socket files to a different directory. This directory must be outside your operating system's temp directory, and the user that owns the `cribl` process must have the necessary permissions to write to it.
In Unix-like operating systems, the maximum length for Unix domain socket paths is 108 bytes. To minimize the path length, use a directory that is close to the root, such as `/var/tmp`.
In the UI, you specify the directory at **Settings** > **Global Settings** > **System** > **Distributed Settings** > **Leader Settings** > **Helper processes socket dir**.
## Configure an Edge Node {#config-edge}
On each endpoint, you can configure Cribl Edge variously through the UI, the `instance.yml` config file, environment variables, or the command line.
### Use the UI
1. Navigate to **Settings** > **Global** > **Distributed Settings** > **General Settings**.
1. Select **Mode**: **Managed Edge**.
1. Select the **Leader Settings** menu.
1. Confirm or enter the required **Address** (for example,
`criblleader.mycompany.com`). Customize the optional settings if desired.
1. Select **Save** to restart the Cribl server.
### Use the YAML Config File
In `$CRIBL_HOME/local/_system/instance.yml` (`C:\ProgramData\Cribl\local\_system\instance.yml` on Windows), under the `distributed` section, set `mode` to `managed-edge`:
```yaml {title="$CRIBL_HOME/local/_system/instance.yml or C:\ProgramData\Cribl\local\_system\instance.yml"}
distributed:
mode: managed-edge
envRegex: /^CRIBL_/
master:
host:
port: 4200
authToken:
compression: none
tls:
disabled: true
connectionTimeout: 5000
writeTimeout: 10000
tags:
- tag1
- tag2
- tag42
group: teamsters
```
### Use Environment Variables
You can configure Edge Nodes via environment variables, as in this example:
`CRIBL_DIST_LEADER_URL=tcp://@:4203 ./cribl start`
For additional details, see [Configure Environment Variables](configure-env-vars).
### Use the Command Line
You can configure an Edge Node using CLI commands of this form:
```shell
./cribl mode-managed-edge -H -p -u [options] [args]
```
The command requires the `-H`, `-p`, and `-u` parameters. For other options, see the [CLI Reference for mode-managed-edge](cli-mode-managed-edge). Here is an example command:
```shell
./cribl mode-managed-edge -H -p -u && ./cribl restart
```
Edge will need to restart after you issue this command.
############################################################
# SOURCE: https://docs.cribl.io/edge/outpost.md
############################################################
# Connect Nodes to Leader Through Cribl Outpost
Simplify Node-to-Leader communication with Cribl Outpost.
---
Cribl Outpost simplifies management of deployments
by acting as a relay between individual Edge Nodes or Stream Workers and the Leader Node.
Once you establish an Outpost and configure it to accept traffic from Edge Nodes or Stream Workers,
control plane communication between those Nodes and the Leader will pass through the Outpost.
Cribl Outpost is ideal in situations where you can't establish direct communication
between Edge Nodes or Workers and the Leader, for example due to restrictions in opening up firewall ports.
Using an Outpost limits the need to set up third-party proxies
and simplifies maintenance by reducing the complexity of firewall setups.
> Cribl Outpost requires an Enterprise plan or Enterprise license. For details, see [Pricing](https://cribl.io/pricing/).
>
> If you connect an Outpost to a Leader that does not have a proper plan or license,
> the Outpost node will close the listener for communication incoming from Edge Nodes or Workers,
> but will be shown as healthy on the Leader.
{.box .info}
## How Outpost Works
An Outpost is a special type of Node that rests between Edge Nodes or Stream Workers and the Leader.
{border="false" scale="90%"}
For an individual Edge Node or Worker, an Outpost is indistinguishable from the Leader:
Nodes communicate with the Outpost the same way they would with a Leader,
using the same configuration (for example, using the URL defined in the `CRIBL_DIST_LEADER_URL` environment variable or in `instance.yml`).
Instead of connecting directly to multiple Nodes (possibly thousands),
the Leader maintains those connections with one Outpost only.
This means you only need to configure communication lines such as firewall exceptions once per Outpost.
## Outpost Capacity
You can connect a maximum of 28,000 Edge Nodes to a single Outpost.
The specific number of possible connections up to this limit
depends on the Outpost Node specifications, in particular on CPU and RAM.
The recommended specifications for serving a given number of Edge Nodes are as follows:
| Number of Edge Nodes | vCPU | RAM | Connection listener processes |
| --- | --- | --- | --- |
| up to 10,000 | 2 | 4 GiB | 1 |
| 10,000 - 25,000 | 4 | 8 GiB | 2 |
While the maximum limit is 28,000 Edge Nodes, if you need to connect more than 25,000 Nodes,
Cribl recommends using more than one Outpost to share the load.
## High Availability for Outposts
Outposts only participate in control plane communication between the Leader and Edge Nodes.
This means that data plane communication (actual data flow) continues successfully
even if an Outpost Node becomes unavailable.
However, and Outpost Node going down will interrupt control plane communications,
including downloading latest configuration changes or product updates.
You can ensure continued flow of control plane communications
by deploying multiple Outpost Nodes behind a network load balancer or a round-robin DNS.
In such a case, point all Node that should connect to the Outpost
at the load balancer instead.
############################################################
# SOURCE: https://docs.cribl.io/edge/set-up-outpost.md
############################################################
# Set Up Cribl Outpost
Create a Cribl Outpost to relay communication between Edge Nodes and the Leader.
---
## Prerequisites
Cribl Outpost requires an Enterprise plan or Enterprise license. For details, see [Pricing](https://cribl.io/pricing/).
You can install Cribl Outpost on Linux only.
Do not run an Outpost Node on the same host as a Stream Worker to avoid both Nodes competing for resources.
Before deploying an Outpost Node, ensure you have an authentication token created on the Leader.
See [Create and Manage Authentication Tokens](securing-auth-token) for information on how to create and manage authentication tokens.
## Prepare Outpost Group
The Outpost Group defines the listener URL on which Edge Nodes connect to the Outpost,
TLS settings that secure that connection, as well as other options such as the retention time for disconnected Outpost Nodes.
Before you add an Outpost Node, revise the settings in the `default_outpost` Group
and, if needed, change them, or create a new Outpost Group.
To create a new Outpost Group:
1. In the sidebar, select **Outposts**, then select **Add Outpost Group**.
1. Enter a **Name** for the group, an optional **Description**, and **Tags**.
1. In **Time to keep disconnected Outposts**, define how long the Group should retain information about an Outpost that disconnects from the Leader. Leave empty to use the default of one day.
1. Select **Save**.
1. Select the new Group you created, and then **Settings** to configure its more advanced parameters.
1. In **Distributed Settings > Listener Settings**, configure **Address** and **Port**.
These will form the URL that Edge Nodes will use to connect to Outposts in this Group.
1. In **Distributed Settings > TLS Settings**, enable **Use TLS** to secure Outpost-Edge Node communication with TLS.
If you enable TLS, you also need to provide certificate and private key information, as needed.
1. Optionally, configure any other Outpost Group settings.
Refer to [Outpost Group Settings](outpost-group-settings) for more information.
> In versions older than v4.16, TLS information for Outposts was included in the individual Outpost Node settings
> instead of the Outpost Group.
> On upgrade to v4.16, Outposts Nodes running older versions are placed in the `default_outpost` Group.
{.box .info}
## Create an Outpost
There are two ways of adding a new Outpost Node:
- using the [bootstrap script](#bootstrap) provided in the UI
- from a [binary file](#binary)
### Add an Outpost with a Bootstrap Script {#bootstrap}
To add a new Outpost Node using a CLI bootstrap script:
1. From the sidebar, select **Outpost**, then select an Outpost Group.
1. Select **Add/Update Outpost Node**, then select **Add**.
1. Fill and modify the deployment option fields:
- The **Install package location** defaults to `Cribl CDN`. If desired,
change this to `Download URL`.
- Enter the authentication token in **Auth token**.
(For more information about creating tokens, see [Create a New Provisioning Token](securing-auth-token#create).)
- As needed, correct the target **Outpost Group**, as well as the **Leader hostname/IP**
(URI).
- As needed, correct the **User** and **User Group** to run Cribl as.
- As needed, correct the **Installation directory**.
1. Optionally, add **Tags** that you can use for filtering and grouping Outposts.
Use a tab or hard return between (arbitrary) tag names.
1. Copy the resulting script to your clipboard.
Finally, to add the Outpost, paste the script onto the command line where you want to set up the node, and execute it.
If your script successfully added the Outpost, you will see it in the **Outposts** list for your selected group.
### Add an Outpost from a Binary File {#binary}
To add a new Outpost Node using a downloaded binary file:
#### Download and Untar the Binary
1. Go to the Cribl [Download page](https://www.cribl.io/download)
and set the **Software** drop-down to `Cribl Suite (Edge and Stream)` for your processor architecture.
1. Select **Download now** to get the `.tgz` archive.
1. Un-tar the install package in your directory of choice:
```shell
cd /opt/
tar xvzf cribl---.tgz
```
#### Set Installation Mode
Set the installation mode to `mode-outpost` to connect the Outpost to the designated Leader.
Use one of the authentication tokens as `` to allow Nodes to connect to the Leader:
```shell
/opt/cribl/bin/cribl mode-outpost /
-H /
-p /
-u /
-S 1
```
> You can use SOCKS proxies for the communication between Outpost Nodes and the Leader.
> See [Configure SOCKS Proxy](proxy-socks-config) for more information.
{.box .success}
##### Token
`` must match an existing authentication token.
For more information about creating tokens, see [Create a New Provisioning Token](securing-auth-token#create).
##### TLS
The `-S 1` parameter enables secure communication between the Outpost Node and the Leader via TLS.
If you omit this parameter, the connection will not use TLS.
However, TLS must be enabled for the Outpost Node to connect to a Leader in Cribl.Cloud.
##### Group
By default, a new Outpost Node will be placed in the `default_outpost` group.
You can specify a different Group by providing the `-g` parameter to the `mode-outpost` command, for example:
```shell
/opt/cribl/bin/cribl mode-outpost /
-H /
-p /
-u /
-g /
-S 1
```
#### Configure Installation as a Service
Finally, configure the installation as a service and start it:
```shell
sudo /opt/cribl/bin/cribl boot-start enable -m systemd -u cribl
sudo systemctl restart cribl-outpost.service
```
At this point your Cribl Outpost is ready for use
and you can start configuring Edge Nodes to connect to it.
## Connect Nodes to an Outpost {#connect}
Edge Nodes see Outpost in the same way as a Leader Node.
This means that, to connect a Node to an Outpost, you only need to provide
the Outpost listener URL and the Leader auth token.
The Outpost doesn't have a separate auth token.
When configuring Node connection, use the auth token of the Leader.
Configure the connection either from Settings (**Global > Distributed Settings > Leader Settings**),
or by setting the `CRIBL_DIST_LEADER_URL` environment variable:
```
CRIBL_DIST_LEADER_URL=tcp://@:
```
############################################################
# SOURCE: https://docs.cribl.io/edge/manage-outpost.md
############################################################
############################################################
# SOURCE: https://docs.cribl.io/edge/outpost-group-settings.md
############################################################
# Configure Outpost Group Settings
Configure settings for an Outpost Groups, including connection parameters, retention of disconnected nodes,
proxy, storage, and TLS settings.
---
> Required Permission: **Admin**
>
{.box .info}
To edit Outpost Group settings:
1. In the sidebar, select **Outposts**.
1. Select an Outpost Group, then go to the **Settings** tab.
## General Settings {#general}
### Outpost Group Configuration
You can configure the following general Outpost Group options on the **Overview** tab:
|Outpost Group Configuration Setting
|Description|
|---------------------------|-----------|
|**Description**|Optionally, add or edit the Outpost Group description.|
|**Tags**|Use tags to organize Outpost Groups into logical categories.|
|**Time to keep disconnected Outposts**|Configure how long the Outpost Group should retain information about an Outpost that disconnects from the Leader. Format examples: `8h`, `5d`, `1w` (format defaults to days if you only specify a number). Default: 1 day.|
### API Server Settings
#### General
You can set the following options for the API server.
|API Server Setting
|Description|
|------------------|-----------|
|**Host**|The hostname or IP address you want to bind the API server to.- Defaults to `0.0.0.0`.
- You can override the value with the `CRIBL_API_HOST` [environment variable](environment-variables).
|
|**Port**|API port to listen to.- Defaults to `9000`.
- You can override the value with the `CRIBL_API_PORT` [environment variable](environment-variables).
|
#### TLS
For information on TLS options, see [Configure TLS](securing-configure-tls).
### Proxy Settings
|Proxy Setting
|Description|
|---------------------------|-----------|
|**Use proxy env vars**|Honors the `HTTP_PROXY/HTTPS_PROXY` environment variables. Defaults to toggled on.
Cribl prioritizes environment variables for proxy settings in this order: Process, User, and System.
If your Cribl service is managed by a service manager other than systemd (such as upstart or init), the `Use proxy env vars` toggle might not behave as expected because Cribl might prioritize environment variables set by the service manager instead of using the proxy settings you intended.|
### Limits
The **Limits** tab provides access to controls for metrics, storage, metadata,
jobs, the Redis cache and connections to it, and CPU settings.
#### Storage {#storage}
You can configure the following options for storage:
|Storage Setting
|Description|
|---------------------------|-----------|
|**Min free disk space**| The minimum amount of disk space on the host before various features take measures to prevent disk usage (KB, MB, and so on). Default: `5 GB`. To avoid the ["Unable to write to the filesystem, disk space low" error](/stream/common-errors/#unable-to-write-to-the-filesystem-disk-space-low), ensure that the available disk space remains above this threshold by monitoring disk usage and configuring alerts.|
#### Metadata
**Event metadata sources**: List of event metadata sources to enable.
Available sources: `os`, `cribl`, `aws`, `env`, and `kube`.
No sources are enabled by default.
### SNI
|SNI Setting
|Description|
|---------------------------|-----------|
|**Disable SNI-based connection routing**|This setting affects how the Cribl Edge control plane routes connections.|
> Do not change this setting without consulting Support first.
>
{.box .warning}
### Support
|Support Setting
|Description|
|---------------------------|-----------|
|**Feature Flag Overrides**|Cribl thoroughly tests all releases in Cribl.Cloud environments to ensure stability and reliability. However, new features can sometimes behave unpredictably in customer-managed data environments. The scale of some environments and the complexity of many possible configurations make it difficult to test for every possible scenario, occasionally leading to unforeseen issues.
To ensure the stability of your production environment, Cribl Stream includes a safety mechanism that makes it possible to disable specific new features if they are causing unexpected issues. This is a customer-first tool designed to minimize impact and avoid the need to roll back upgrades or wait for a full patch release.
If you suspect that a new feature is impacting your environment, please contact the Support team for assistance. This process ensures that Cribl can quickly stabilize your environment while also collecting the necessary information to improve Cribl Stream for everyone.
**Do not change this setting without consulting Support first.**|
|**Log file max size**|Maximum size of each log file before rotation. Enter a numeral with units of KB or MB (for example, "5 MB", "1024 KB"). Default: 5 MB. Minimum: 1 MB.|
|**Number of log files to keep**| Number of rotated log files to keep before purging. Default: 5. Range: 1-100. Only new rotations use this limit. Existing files above the previous limit are not removed.|
### Distributed Settings
#### Listener Settings
|Listener Setting
|Description|
|------------------|-----------|
|**Address**|The hostname or IP address you want to bind the API server to.|
|**Port**|API port to listen to. |
|**IP allowlist regex**|Regex matching IP addresses that are allowed to establish a connection.|
#### TLS Settings
For information on TLS options, see the documentation for any Source or
Destination that supports TLS.
### Service Process
#### Services
|Services Setting
|Description|
|---------------------------|-----------|
|**Number of connection listener processes**|The number of Outpost connection listeners to spawn. Positive numbers specify an absolute number of listeners. Negative numbers specify a number of listeners relative to the number of CPUs in the system. The default is `-2`.|
## Other Settings
This page's remaining options work essentially the same way as their counterparts in **Settings** > **Global**. Use the following links for details about [logging](monitoring#log_settings) levels/redactions and [security](securing-and-monitoring).
############################################################
# SOURCE: https://docs.cribl.io/edge/version-control.md
############################################################
# Set Up Version Control
As part of the Cribl Edge installation process for on-prem deployments, verify that Git is installed on the hosts running your deployment. Git is a required component for various reasons:
- It is a dependency for many Cribl Edge features.
- The [Monitoring > Licensing](monitoring#licensing) dashboard will display configuration change markers only if you have `git` installed.
- All configuration changes must be committed before they are deployed. The Leader notifies Edge Nodes that a new configuration is available, and Edge Nodes pull the new configuration from the Leader Node.
> See [Commit and Deploy Changes](commit-deploy) for more information about how to deploy changes with the UI. You can also use the Cribl API to automate commit and deploy commands. Read [Commit and Deploy Changes](/cribl-as-code/commit-deploy/) in the API documentation for more information.
>
{.box .info}
## Requirements
Cribl Edge requires Git **1.8.3.1.** or later.
For [distributed deployments](requirements#git), `git` **must** be installed and available locally on the host running the Leader Node.
## Verify Git
To verify that `git` is available and running the required version, run:
```bash
git --version
```
## Install Git
To install Git, see the [Git Installation](https://git-scm.com/install/) steps in the official Git documentation.
############################################################
# SOURCE: https://docs.cribl.io/edge/edge-nodes-config-bundle.md
############################################################
# Manage Edge Node Config Bundles
Learn about Edge Node config bundles and how to manage them
---
## Config Bundles
Config bundles are compressed archives of all config files and associated data
that an Edge Node needs to operate. Both Leader and Worker Nodes automatically manage config bundles in the following ways.
## Bundle Storage and Delivery
The delivery mechanism for config bundles depends on where your Leader resides:
- **Cribl.Cloud Leader**: Bundle storage is managed for you in a Cribl-owned Amazon S3 bucket. You do not need to configure S3 settings. Edge Nodes will automatically attempt to pull from the Cribl S3 bucket first, then fall back to the Leader over port `4200`. For details on preventing deployment delays by white-listing the required S3 and Leader endpoints, see [Network Access for Config Bundles](/stream/cloud-workers#network).
- **Customer-managed Leader**: By default, the Leader serves bundles directly. To reduce Leader load, you can store bundles in S3 using your own bucket. For details, see [Store Bundles Remotely](#remote-bundle).
For details on ports, see [Ports](ports) and [Required Ports in Cribl.Cloud](reference-architectures/arch-cloud-ports).
## Bundle Lifecycle and Retention
Regardless of where bundles are stored, Cribl Edge maintains a cleanup policy to optimize disk space.
### Leader Node Bundles
The Leader creates bundles upon Deploy, and
manages them as follows:
- Bundles are wiped clean on startup.
- While running, at most 5 bundles per group are kept.
- Bundle cleanup is invoked when a new bundle is created.
### Edge Node Bundles
The Edge Node pulls bundles from the Leader and manages them as follows:
- Last 5 bundles and backup files are kept.
- At any point in time, all files created in the last 10 minutes are kept.
- Bundle cleanup is invoked after a reconfigure.
## Store Bundles Remotely {#remote-bundle}
For customer-managed Leaders, you can optionally reduce Leader load by storing bundles in an Amazon S3 bucket. This offloads bundle distribution from the Leader to your designated S3 bucket. Edge Nodes will pull bundles directly from S3, minimizing Leader strain.
You can configure bundling in S3 through the following ways:
- **UI**: [Configure](setting-up-leader-and-worker-nodes#config-leader-ui) the Leader settings on the Leader Node. In the sidebar, select **Settings**. In the **Global** tab, select **System > Distributed Settings > Leader Settings**. In the **S3 Bundle Bucket URL** field, specify an S3 bucket for remote bundle storage. Format: `s3://${bucket}`.
- **YAML**: In the [`instance.yml` config](instanceyml) file, specify the S3 bucket in `master.configBundles.remoteUrl`.
- **Environment Variable**: Use the `CRIBL_DIST_LEADER_BUNDLE_URL` [environment variable](environment-variables#distributed).
### Optimized Bundle Delivery from S3 {#optimized-S3bundle}
Cribl prioritizes efficient updates for Edge Nodes, regardless of their network environment. Here's how:
- **Direct S3 Downloads (Preferred)**: Cribl prioritizes downloading upgrade packages directly from Amazon S3 (S3) whenever possible, ensuring efficient updates for Edge Nodes connected to Cribl.Cloud or an on-premise Leader. The source is specified in the Leader's **Global Settings** > **Upgrades** > **Package source** setting.
- **Automatic Failover**: If the S3 download fails, a built-in failover automatically retrieves the bundle from the Leader itself (port `4200`). This redundancy guarantees updates even in challenging network environments, including private networks with restricted internet access. For these networks, Cribl prioritizes a CDN download before reverting to the Leader Node as a final resort.
- **Allowlist S3 Access (Optional)**: To further optimize downloads for Edge Nodes with internet access, consider allowlisting the regional Amazon S3 endpoint used by your bundle bucket in your firewall or proxy configuration. For example, endpoints look like `s3.eu-central-1.amazonaws.com` or `s3.us-west-2.amazonaws.com`, and dual-stack variants look like `s3.dualstack.us-west-2.amazonaws.com` or `bucket-name.s3.dualstack.us-west-2.amazonaws.com`.
S3 endpoints are regional and do not share a single wildcard hostname, so patterns such as `*.s3.amazonaws.com` are not sufficient for firewall or proxy allowlists. Refer to the [AWS documentation: Using Amazon S3 dual-stack endpoints](https://docs.aws.amazon.com/AmazonS3/latest/API/dual-stack-endpoints.html) for the complete and current list of S3 regional and dual-stack endpoints.
### Authenticating to S3
When storing configuration bundles in an Amazon S3 bucket, Cribl Edge needs appropriate credentials to access the bucket. There are several ways to provide these credentials, with IAM roles being the most secure and recommended approach.
#### IAM Roles (Recommended)
The best practice is to configure an IAM role with read access to your S3 bucket and attach this role to the EC2 instance (or other AWS service) where your Leader and Edge Nodes are running. Cribl Edge will automatically use the instance's IAM role to authenticate to S3. This method avoids the need to manage AWS credentials directly within Cribl and is the most secure option.
To authenticate via IAM roles:
1. Create an IAM role with the `AmazonS3ReadOnlyAccess` policy (or a custom policy with more restrictive permissions if needed) for the S3 bucket containing your bundles.
2. Attach this IAM role to the EC2 instance(s) running your Leader and Worker Nodes.
3. Provide the path to your S3 bucket using the [CRIBL_DIST_LEADER_BUNDLE_URL environment variable](environment-variables#distributed) or the [S3 Bundle Bucket URL field in the **Distributed**/ **Leader** settings](setting-up-leader-and-edge-nodes#use-the-ui) (For example, `s3://your-bucket-name/path/to/bundles`). `path/to/bundles` is optional.
#### AWS Access Keys
Alternatively, you can provide AWS access keys directly. Ensure the environment in which Cribl Edge runs in contains the following variables:
```bash
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export CRIBL_DIST_LEADER_BUNDLE_URL="s3://your-bucket-name/path/to/bundles"
```
### Encryption and Backward Compatibility {#compatibility}
If your Edge Nodes are running versions older than 4.9.0 and your Leader is v.4.9.0 or newer, deployments will always use Leader-hosted bundles. This happens because of how encryption works:
- Newer Leaders encrypt bundles when uploaded to S3 to enhance security and prevent the exposure of sensitive information like secret keys.
- Older Edge Nodes check the code for errors by comparing its checksum (a unique identifier).
- Since the Edge Nodes check the encrypted code, they will always find errors because the checksum changes after encryption.
You might see an error similar to the one below in your logs:
`Checksum mismatch, expected=f781b2bace2e1869e840b0ee200b57378ecae985, found=a59af697c140521ea7f42af7630ac1fce88e4ff0"**`
############################################################
# SOURCE: https://docs.cribl.io/edge/configs-and-restarts.md
############################################################
# Configurations and Restart
In a Distributed deployment, Edge Nodes poll the Leader for configuration changes. Many of these changes require a quick reload to read the new configuration, while others require a restart of the Cribl processes on the Edge Node.
You can **Restart** and **Reload** via the UI. In the sidebar, select **Settings**, then **Global**. Under **System** > **Controls** select **Reload** or **Restart**.
Upon restarts, be aware of the following:
- Syslog data still being received over UDP might be dropped.
- Edge Nodes will temporarily disappear from the Leader's **Workers** or **Edge Nodes** page.
- Aggregation and suppression operations will start over.
- Edge Nodes' local copies of Monitoring metrics will be dropped.
- Cribl Edge will drop any events still in RAM that were bound for persistent queues. (However, PQ data already written to disk will persist through the restart.)
Changes that require reloads include configuration changes to:
- Functions
- Pipelines
- Packs
- Routes
- Lookups
- Parquet schemas
- Global variables
- Fleet Settings > Limits
- Fleet Settings > Logging > Levels
Changes that require restarts include configuration changes to:
- Distributed mode (Leader versus Managed Edge Node or Single instance)
- Fleet assignment
- Event Breakers
- QuickConnect configs
- Sources
- Destinations
- Fleet Settings > General Settings > TLS
- Fleet Settings > General Settings > Advanced
- Fleet Settings > Worker Processes > Process count and Memory
## How Configuration Changes Apply {#rolling}
In a Distributed deployment, how a change applies across each Edge Node in a Fleet depends on the type of change:
- **Sources and Destinations**: Cribl Edge restarts Worker Processes on each Edge Node to apply the change.
- In Cribl Stream, a Worker Node can run multiple Worker Processes, so they restart in a rolling fashion and the Node keeps processing data during the change. For the rolling rollout mechanics and configurable settings, see [Worker Process Configuration Rollout](/stream/deploying-configurations#worker-process-rolling-restart) in the Cribl Stream docs.
- In Cribl Edge, each Edge Node runs a single Worker Process, so it simply restarts that process.
- **Routes, Pipelines, and Lookups or other knowledge objects**: Cribl Edge reloads the configuration without restarting Worker Processes, so data flow continues uninterrupted.
- In Cribl Stream, reloads also roll out in batches across Worker Processes. If a Worker Process does not finish reloading within the configured timeout, Cribl Stream restarts it. For details, see [Worker Process Configuration Rollout](/stream/deploying-configurations#worker-process-rolling-restart).
- **Fleet settings that involve the API Process** (such as API, TLS, Proxy, and Security settings): Cribl Edge restarts all Worker Processes at once rather than in a rolling fashion. All Worker Processes stop before the API Process restarts, and then the API Process starts them again.
Some general guidelines to keep in mind:
* Most UI interactions, such as changing the order of Functions in a Pipeline or changing the order of Routes, do not require restarts.
* Some configuration changes in the **Settings** UI do require restarts. These prompt you for confirmation before restarting.
* All direct edits of configuration files in `$CRIBL_HOME/local/` require restarts.
* Edge Nodes might temporarily disappear from the Leader's **Workers** or **Edge Nodes** tab while restarting.
* A `git commit` command on the Leader Node's host (using a freestanding `git` client not embedded in Cribl's CLI or UI) requires either a reload or restart.
* When using the [Cribl App for Splunk](/stream/deploy-splunkapp), changes to Splunk configuration files might or might not require restarts. Please check current [Splunk docs](http://docs.splunk.com).
############################################################
# SOURCE: https://docs.cribl.io/edge/deploy-add-second-leader.md
############################################################
# Leader High Availability/Failover
To handle unexpected outages in on-prem Distributed deployments, Cribl Edge supports configuring standby Leaders for failover.
In this High Availability (HA) scenario, if the primary Leader goes down,
one of the other Leaders, called standby Leaders, take its place.
This ensures continuity of operation, including functions that require the Leader,
such as Collectors and Collector-based Sources, which can ingest data without interruption.
For scheduling Collector jobs, [High-availability Collectors](/stream/collectors-ha) complements Leader HA. An eligible Fleet can elect a temporary Worker Captain to orchestrate jobs when the Leader is unreachable, without shared Network File System (NFS).
> High-availability Collectors does not replace standby Leaders (or a Cribl.Cloud-managed Leader) when you need full control plane resilience. It only covers Collector job orchestration inside the Fleet. While the Leader is down, you still lack an active Leader for anything that requires one, including the Leader UI and API, new commit and deploy changes, ad hoc Collector runs started from the Leader, license updates to restarted nodes, Leader-side metrics aggregation, and notifications generated on the Leader. During a Leader outage, those capabilities still rely on Leader HA (self-hosted) or on Cribl.Cloud operating the Leader for you.
>
{.box .info}
Cribl.Cloud handles High Availability and configuring standby Leaders automatically,
requiring no configuration or action on your part.
> For license tiers that support configuring standby Leaders, see [Cribl Pricing](https://cribl.io/pricing/stream/).
>
{.box .info}
## Primary and Standby Leaders
Only one Leader Node is the elected primary at any given time, and only the primary Leader owns Leader state.
A standby Leader (or Leaders) takes over as primary in the event of failover, when the primary Leader becomes unavailable.
Leader state backed by SQLite, such as Worker connection data, is always written to local disk, because SQLite file locking is not compatible with NFS.
With Leader replication enabled, the active git repository is also on local disk and is replicated to the failover volume rather than being read and written across NFS.
All changes to configurations and git commits are replicated to a shared failover NFS volume,
and from it to the standby Leaders.
The shared volume coordinates the Leaders and holds replicated configuration, git data, and shared Leader state. It is not where you install Cribl Edge.
See [High Availability Requirements](ha-requirements) for the distinction between the installation directory and the NFS failover volume.
If the primary Leader becomes unavailable, a standby Leader takes its place.
The standby then pulls the required configuration and the latest git commits from the failover volume.
All Edge Nodes will connect to the new primary Leader, which retains the state and metrics of the old primary Leader.
By default, a standby Leader reports a standby status from the `GET /health` endpoint, so the load balancer routes client requests only to the primary Leader.
When you enable [Active-Active (Proxy) Mode for HA Leaders](ha-requirements#ha-proxy), the load balancer can route requests to either Leader, and the standby proxies incoming requests to the primary.
{border="false"}
### Git-Backed Configuration and Leader Replication {#git-branch-ha}
In Leader High Availability (HA) deployments with GitOps enabled, Leader replication aligns branches. The local Git branch for the active Leader Node matches the branch on the failover (HA) volume repository.
For more information about `CRIBL_GIT_REMOTE`, `CRIBL_GIT_BRANCH`, and related bootstrap behavior, see [usage notes for `CRIBL_GIT_REMOTE`](environment-variables#cribl_git_remote-bootstrapforce-sync).
### Leader Settings in High Availability Setups {#leader-settings}
When you first configure Leader High Availability, Cribl Edge will create a new [`leader.yml`](leaderyml) file in the local `$CRIBL_HOME/local/cribl` directory and will upload it to the failover volume. Configuration stored in the `leader.yml` file on the failover volume will take precedence over what is stored in the local `instance.yml` file.
The `leader.yml` file replicates most of the content of the local `instance.yml`, but leaves out the failover configuration.
While running in failover mode, when you change **Settings > Distributed Settings** via the UI, Cribl Edge applies those changes to `leader.yml` in the failover directory.
See [Configure Standby Leader Nodes](configure-standby-leaders) for information on how to prepare standby Leaders for a failover scenario.
############################################################
# SOURCE: https://docs.cribl.io/edge/ha-requirements.md
############################################################
# High Availability Requirements
Before adding a standby Leader, ensure that you fulfill all requirements in this section.
## Disk Space Requirements {#disk-requirements}
Ensure that all Leaders, both the primary and the standby ones, have enough disk space available.
You need enough space to hold the contents of the failover volume (except the `state` directory),
all your configuration files, plus at least 10 GB in reserve for future use.
## Leader Health Check Timeouts {#timeouts}
Configure adequate timeout for Leader health checks
to ensure the Leader has enough time to pull all the configuration and the git repository from the failover volume.
Too low timeout could results in the Leader being terminated prematurely.
The recommended timeout scales with the number of Worker Groups and Fleets the Leader manages.
The exact value depends on the size of the git repository,
size of the `groups` directory, and the total number of configuration files.
Approximate recommendations are:
- 2 minutes for smaller deployments
- 5 minutes for deployments with > 50 Worker Groups/Fleets
- 10 minutes for deployments with > 100 Worker Groups/Fleets
## Git Configuration Requirements
If you have git timeout configured in `local/cribl.yml`,
ensure that it is equal or greater than the system default of 10 minutes.
## Auth Token Requirements
All Leaders must have matching auth tokens.
See [Create and Manage Authentication Tokens](securing-auth-token) for more information about creating and managing authentication tokens.
## NFS Requirements
> ##### The failover volume is not the installation directory
>
> The NFS failover volume is separate from each Leader's installation directory.
> Install Cribl Edge on fast local disk on every Leader, both primary and standby.
> Network filesystems (NFS, EFS, and similar) are not supported for the installation directory.
> Use NFS only for the failover volume, which coordinates Leaders and holds replicated configuration, git data, and shared Leader state.
> A running Leader in failover mode resolves the `$CRIBL_HOME` variable to a path that depends on its replication state.
> Read this requirement as applying to the installation directory rather than to `$CRIBL_HOME`.
> See [OS and System Requirements](/stream/requirements) and [Leader High Availability/Failover](deploy-add-second-leader).
>
{.box .warning}
- On all Leader Nodes, use the latest version of the NFS client. **NFSv4 is required.**
- Ensure that the NFS volume has at least 100 GB available disk space.
- Ensure that the NFS volume's IOPS (Input/Output Operations per Second) is ≥ 200. (Lower IOPS values can cause excessive latency.)
- Ensure that ping/latency between the Leader Nodes and NFS is < 50 ms.
- Ensure your NFS system supports updating `mtime`.
> You can validate the NFS latency using a tool like `ioping`. Navigate to the NFS mount, and enter the following command:
>
> ```shell
> ioping .
> ```
>
> For details on this particular option, see the [ioping docs](https://github.com/koct9i/ioping/).
>
{.box .info}
## NFS Mount Option Requirements
The Leader Node will access large numbers of files whenever you use the UI or deploy configurations to Edge Nodes. When this happens, NFS's default behavior is to synchronize access time updates for those files, often across multiple availability zones and/or regions. To avoid the problematic latency that this can introduce, Cribl recommends that you add one of the following NFS mount options:
1. `relatime`: Update the access time only if it is more than 24 hours ago, or if the file is being created or modified. This allows you to track general file usage without introducing significant latency in Cribl Edge. To do the same for folders, add the `reldiratime` option.
1. `noatime`: Never update the access time. (Cribl Edge does not need access times to be updated to operate correctly.) This is the most performant option - but you will be unable to see which files are being accessed. To do the same for folders, add the `nodiratime` option.
## Recommended NFS Configuration
If you are on AWS, we recommend the following:
- Use Amazon's Elastic File System (AWS EFS) for your NFS storage.
- Ensure that the user running Cribl Edge has read/write access to the mount point.
- Configure the EFS **Throughput mode** to `Enhanced` > `Elastic`.
- For details on NFS mount options, see [Recommended NFS mount options](https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-nfs-mount-settings.html).
For best performance, place your Leader Nodes in the same geographic region as the NFS storage. If the Leader and NFS are distant from each other, you might run into the following issues:
- Latency in UI and/or API access.
- Missing metrics between Leader restarts.
- Slower performance on data Collectors.
## Load Balancer Requirements {#loadbalancers}
Configure all Leaders behind a load balancer.
- [Port `4200`](ports#leader) must be exposed via a network load balancer.
- [Port `9000`](ports#leader) can be exposed via an application load balancer or network load balancer.
### Active-Active (Proxy) Mode for HA Leaders {#ha-proxy}
By default, a standby Leader returns a standby status (HTTP `420`) from `GET /health`, so a load balancer that routes only to Leaders returning `200` sends client requests to the primary Leader. UI/API requests that reach the standby on port `9000` are proxied to the primary.
You can enable Active-Active (Proxy) Mode to simplify load balancer routing. In this mode, both primary and standby Leaders report healthy (`200`), allowing the load balancer to route client requests to either Leader. The standby proxies incoming requests to the elected primary, which preserves single-writer semantics because only the primary owns Leader state.
To enable proxy mode, set `connection_proxy.procs` to a nonzero value on each Leader, either in [`service.yml`](serviceyml) or on the Leader's **Service Processes** > **Services** settings page.
The standby must be able to reach the primary over port `9000` to forward UI/API traffic. With proxy mode enabled, control-plane connections on port `4200` are also proxied. Ensure firewall rules allow Leader-to-Leader communication on these paths.
### Load Balancer Health Checks {#load-balancer-health-checks}
To ensure reliable load balancing for your Leader Nodes, configure health checks using the `GET /health` endpoint.
Load balancer health checks are supported only on port `9000`. [Configure TLS](securing-configure-tls) to use `https` for health checks. Otherwise, use `http`.
Load balancers that support such health checks include:
- Amazon Web Services (AWS) [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) (NLB). Suitable for TCP, UDP, and TLS traffic.
- AWS [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) (ALB). Application-aware, suitable for HTTP/HTTPS traffic.
- [HAProxy](https://docs.haproxy.org/).
- [NGINX Plus](https://docs.nginx.com/nginx/).
For optimal performance and to avoid potential throttling, adhere to the following guidelines:
- **Polling frequency**: Set your load balancer to check the health endpoint every `60` seconds. This interval aligns with Cribl.Cloud's internal monitoring and prevents excessive API calls.
- **Avoid overly frequent checks**: Polling more often than every `60` seconds (for example, every `5` seconds) can overload the Leader's API Process.
- **Load balancer routing**: Configure your load balancer to direct traffic exclusively to Leader Nodes that return a `200` status code from the health check.
For detailed information on query and response formats for the `GET /health` endpoint, refer to the code example for [querying the health endpoint](/cribl-as-code/query-health-endpoint).
> ##### AWS Network Load Balancers
>
> If you need to access the same target through a Network Load Balancer, use an IP-based target group and deactivate client IP preservation. For details, see:
> - [Why can an instance in a target group not reach itself via NLB?](https://repost.aws/questions/QU_9Dcea7NR3W3MzaZKVkcBA/why-can-an-instance-in-a-target-group-not-reach-itself-via-nlb)
> - [Why can't a target behind my Network Load Balancer connect to its own Network Load Balancer?](https://repost.aws/knowledge-center/target-connection-fails-load-balancer)
>
{.box .warning}
### Source-Level Health Checks
For many HTTP-based Sources, you can enable a Source-level health check endpoint in the **Advanced Settings** tab. Load balancers can send periodic test requests to these endpoints, and a `200 OK` response indicates that the Source is healthy.
Frequent requests to Source-level health check endpoints can trigger throttling settings. In such cases, Cribl will return a `503 Service Unavailable` response, which can be misinterpreted as a service failure. A `503` response may indicate that the health checks are running too frequently instead of an actual service failure.
############################################################
# SOURCE: https://docs.cribl.io/edge/configure-standby-leaders.md
############################################################
# Configure Standby Leader Nodes
To prepare your deployment for High Availability,
in the configuration for all Leaders, both primary and standby,
you must specify the same failover volume - a shared Network File System (NFS) volume.
You can configure standby Leader Nodes in the following ways. These configuration options are similar to configuring the primary [Leader Node](setting-up-leader-and-worker-nodes):
- [Using the UI](#ui)
- [Updating the YAML config file](#yaml)
- [Using the Command Line](#cli)
- [Using Environment Variables](#env-vars)
> Remember, the `$CRIBL_VOLUME_DIR` environment variable overrides `$CRIBL_HOME`.
>
{.box .success}
## Use the UI {#ui}
For all Leader Nodes, both primary and standby, configure the following:
1. In **Settings** > **Global** > **Distributed Settings** > **General Settings**, select **Mode**: `Leader`.
1. Next, on the **Leader Settings** left tab, select **Resiliency**: `Failover`. This exposes several additional fields.
1. In the **Failover volume** field, enter the NFS directory that the configurations and git commits will be replicated to. This directory must be outside of the installation directory. One valid solution is to use `CRIBL_DIST_LEADER_FAILOVER_VOLUME=` (alternatively `CRIBL_DIST_MASTER_FAILOVER_VOLUME`). See [Use Environment Variables](#env-vars) for more information.
1. Optionally, adjust the **Lease refresh period** setting, which determines how often the primary Leader tries to refresh its hold on the Lease file,
and the **Missed refresh limit**, which determines how many Lease refresh periods elapse before standby Nodes attempt to promote themselves to primary.
Increase these two values to avoid unnecessary failovers if your NFS has higher latency or is occasionally slow.
1. Select **Save** to restart.
> When you save the **Resiliency**: `Failover` setting, further **Distributed Settings** changes via the UI will lock for both the primary and backup Leader. (This prevents errors in [bootstrapping Workers](/stream/deploy-workers) due to incomplete token synchronization between the two Leaders.) However, you can still update each Leader's distributed settings by modifying its configuration files.
>
{.box .warning}
## Use the YAML Config File {#yaml}
For all Leader Nodes, both primary and standby, configure the following
in `$CRIBL_HOME/local/_system/instance.yml`, under the `distributed` section:
1. Set `resiliency` to `failover`.
1. Specify a volume for the NFS disk to automatically add to the Leader Failover cluster and trigger automated data migration.
```yaml {title="$CRIBL_HOME/local/_system/instance.yml"}
distributed:
mode: master
master:
host:
port: 4200
resiliency: failover
failover:
volume: /path/to/nfs
```
> Note that `instance.yml` configs are local, not on the shared NFS volume.
>
{.box .info}
### Use the Command Line {#cli}
For all Leader Nodes, both primary and standby, use a command of this form:
`./cribl mode-master -r failover -v /mnt/cribl`
For all options, see the [CLI Reference](cli-mode-master).
## Use Environment Variables {#env-vars}
For all Leader Nodes, both primary and standby, configure environment variables listed in [Environment Variables Reference: Adding Fallback Leaders](environment-variables#fallback-leaders).
## Address HA Migration Timeouts {#timeouts}
When enabling Leader High Availability (HA) on a production system with a lot of data,
or when booting up new Leader Nodes,
the process of moving data can sometimes take longer than expected.
If the system runs out of time during this move,
it might only copy part of the data, which can cause problems when the system restarts.
To mitigate the risk of incomplete data transfers,
temporarily increase the system's transfer time prior to enabling HA.
You can do this by adjusting a setting called `TimeoutSec` in the `cribl.service` configuration file.
Locate the **Service** section and add or modify the `TimeoutSec` directive.
For example, set it to `600` seconds (10 minutes):
```
[Service]
TimeoutSec=600
```
A successful data transfer will be confirmed by specific messages in the system's log, such as:
`{"time":"2025-02-17T22:01:46.295Z","cid":"api","channel":"ResiliencyConfigs","level":"info","message":"failover migration is complete"}`
A failed migration due to timeout might show logs ending with a system shutdown, like this:
`{"time":"2025-02-06T15:29:15.416Z","cid":"api","channel":"ShutdownMgr","level":"info","message":"Starting shutdown ...","reason":"Got SIGTERM","timeout":0}`
> After the HA setup is complete, revert the `TimeoutSec` setting to its original value.
> Leaving the timeout extended indefinitely can lead to prolonged Leader downtime
> if a problem occurs during a future failover.
> Make sure to document the original setting before making changes to facilitate easy reversion.
{.box .warning}
############################################################
# SOURCE: https://docs.cribl.io/edge/manage-ha.md
############################################################
# Manage High Availability Deployment
Manage your High Availability deployment by monitoring Leaders, performing an upgrade,
and disabling standby Leader Nodes.
## Monitor Primary and Standby Leader Nodes
To view the status of your Leader Nodes:
1. On the top bar, select **Products**, and then select **Stream**.
1. In the sidebar, select **Monitoring**, then **System**, then **Leaders**.
1. In the **Leaders** table, you can see the status of all your Leader, including the roles they play (primary or standby).
> This page is only accessible when you have a standby Leader configured.
{.box .info}
{border="true"}
## Upgrade a High Availability Deployment
> Upgrading the Leader Node through the UI is not supported for Distributed deployments with a standby Leader.
> Instead, [use the command line (CLI) to upgrade](upgrading#cli).
>
{.box .warning}
Follow this upgrade order:
1. Stop the standby Leader first.
1. Upgrade the standby Leader.
1. Start the standby Leader.
1. Stop the primary Leader; the upgraded standby will take over automatically through the HA failover mechanism.
1. Upgrade the now-stopped former primary Leader for your [Stream](/stream/upgrading/) or [Edge](/edge/upgrading/) deployment.
1. Start the former primary Leader. It will rejoin and assume the standby role (since the other Leader is currently active).
1. Upgrade Edge Nodes in rolling batches; do not upgrade them to a version newer than the Leader.
## Disable a Standby Leader
Cribl recommends that you maintain a standby Leader to ensure continuity in your on-prem Distributed deployment. Should you decide to disable it, contact support to assist you.
############################################################
# SOURCE: https://docs.cribl.io/edge/deploy-antivirus-exceptions.md
############################################################
# Anti-Virus Exceptions
Configure your antivirus to prevent accidental blockage of Cribl Edge.
---
If you are running anti-virus software on an Edge instance's host OS, here are general guidelines for minimizing accidental blockage of Edge's normal operation.
Your overall goals are to prevent the anti-virus software from locking any files while Edge needs to write to them, and from triggering any changes that Edge would detect as needing to be committed.
First, if [persistent queues](persistent-queues) are enabled on any Destinations, exclude any directories that these Destinations write to. This is especially relevant if you're writing queues to any custom locations outside of `$CRIBL_HOME`.
Next, for any non-streaming Destinations that you've configured, exclude their staging paths.
Next, exclude these subdirectories of `$CRIBL_HOME`:
- `state/`
- `log/`
- `.git/` (usually only exists on Leader Nodes)
- `groups/` (on Leader Nodes)
- `local/` (on Edge or Leader)
Finally, avoid scanning any processes. Except for the queueing/staging directories already listed above, Edge runs everything in memory, so scanning process memory will slow down Edge's processing and reduce throughput.
## Stop Endpoint Security from Blocking Edge on Windows {#endpoint-sec}
Antivirus or other endpoint security tools on Windows (such as Windows Defender) may block the `cribl` process.
To tell Windows Defender to exclude the Cribl Edge process from its security scan,
you can run these commands on the appropriate Windows Edge Nodes:
```bash
Add-MpPreference -ExclusionProcess "C:\Program Files\Cribl\bin\nssm.exe"
Add-MpPreference -ExclusionProcess "C:\Program Files\Cribl\bin\cribl.exe"
Add-MpPreference -ExclusionPath "C:\ProgramData\Cribl"
```
############################################################
# SOURCE: https://docs.cribl.io/edge/ports.md
############################################################
# Ports
Learn which ports need to be open for Cribl Edge and its integrations to function.
---
Cribl Edge requires certain ports to be open, and additional ports are needed if you intend to use specific integrations or options to work.
## Leader
In a Distributed deployment, the following ports must be open on the Leader Node.
Ensure that the Leader is reachable on those ports from **all** Edge Nodes.
Default Port | Protocol | Purpose | Direction
--- | --- | --- | ---
`9000` | HTTP/S | Cribl Edge UI. | In
`9000` | HTTP/S | Bootstrapping Fleets from Leader (on-prem). | In
`443` | HTTP/S | Bootstrapping Fleets from Leader (Cribl.Cloud). | In
`4200` | TCP | Heartbeat/Metrics/Leader requests/notifications to clients (for example: live captures, teleporting, status updates, config bundle notifications, and so on). This traffic is separate from the [Notifications](notifications) feature and its external Notification targets such as email, PagerDuty, Slack, webhooks, or AWS SNS. | In
`4200` | HTTP/S | Software upgrade (via [path](upgrading#package-source), not CDN). | In
**External Notification targets:** If you configure external [Notification targets](notifications-targets) (for example, email, PagerDuty, webhooks, Slack, or AWS SNS), the Leader must be able to initiate outbound network connections to those services. Ensure your firewall rules allow outbound traffic from the Leader to each target endpoint over the ports and protocols required by that service (commonly HTTPS/443, or SMTP ports for email). Because these ports are target-specific, they are not listed individually here.
## Outpost
The following ports are used by [Cribl Outpost](outpost).
Default Port | Protocol | Purpose | Direction
--- | --- | --- | ---
`9000` | TCP | Cribl Outpost UI. Login is disabled, but the [Health endpoint](/cribl-as-code/query-health-endpoint) is available. | In
`443` | HTTP/S | Optional port for config bundle downloads from S3 endpoints for Leader-initiated upgrades. When this port is not open, Outpost falls back to bundle downloads from the Leader over HTTP/S 4200. | Out
`4200` | TCP | Communication between the Outpost and the Leader. | Out
`4200` | TCP | Communication between the Outpost and Edge Nodes. | In
`4200` | HTTP/S | Config bundle downloads from the Leader. | Out
`4200` | HTTP/S | Receiving config bundle requests from Edge Nodes. | In
## Edge Nodes
The following ports are used by Edge Nodes.
| Default Port | Protocol | Purpose | Direction |
| --- | --- | --- | --- |
| `9420` | TCP | Cribl Edge UI. | In |
| `9000` | HTTP/S | Communication with the Leader for bootstrapping (on-prem). | Out
| `443` | HTTP/S | Communication with the Leader for bootstrapping (hybrid deployment), and with `https://cdn.cribl.io` to download configurations from CDN. | Out |
| `4200` | TCP | Heartbeat/Metrics/Leader requests/notifications to clients (for example: live captures, teleporting, status updates, config bundle notifications, and so on). | Out |
| `4200` | HTTP/S | Config bundle downloads from the Leader. | Out |
## Other Ports
### Common Ports
| Default Port | Protocol | Purpose | Direction |
| --- | --- | --- | --- |
| `53` | UDP | DNS lookups. | Out |
| `389` | TCP | LDAP Auth (non-TLS). | Out |
| `443` | HTTP/S | OIDC Auth (TLS). | Out |
| `636` | TCP | LDAP Auth (TLS). | Out |
### Integrations and Apps
Integrations with specific services via Sources and Destinations or apps may require opening dedicated ports on Edge Nodes.
The defaults are listed below, but when configuring each Source or Destination you can choose another port.
| Default Port | Protocol | Purpose | Direction |
| --- | --- | --- | --- |
| `162` | UDP | [SNMP Trap](sources-snmp-traps) collection (non-TLS). The preconfigured SNMP Trap Source listens on port `9162`. | In |
| `162` | UDP | [SNMP Trap](destinations-snmp-traps) Destination (non-TLS). | Out |
| `443` | HTTP/S | Collection from and output to multiple HTTPS-based Sources and Destinations. | In / Out |
| `4317` | TCP | Collection from [OpenTelemetry](sources-otel) (gRPC). | In |
| `5986` | HTTP/S | [Windows Event Forwarder](sources-wef) Source. | In |
| `8081` | TCP | Kafka Schema Registry. | Out |
| `8088` | TCP | [Splunk HEC](sources-splunk-hec) input and output. | In/Out |
| `8089` | TCP | [Splunk Search](sources-splunk-search). | In |
| `8125` | TCP/UDP | Output to [StatsD](destinations-statsd), [StatsD Extended](destinations-statsd-extended), and [Graphite](destinations-graphite) (non-TLS). | Out |
| `9090` | TCP | Collection/discovery from [Prometheus Scraper](sources-prometheus). | Out |
| `9092` | TCP | Output to [Confluent Cloud](destinations-confluent) or [Kafka](destinations-kafka), used when no port is provided. | Out |
| `9093` | TCP | Output to [Azure Event Hubs](destinations-azure-event-hubs). | Out |
| `9200` | HTTP/S | [Elasticsearch API](sources-elastic) Source. | In |
| `9514` | TCP/UDP | [Syslog](sources-syslog) Source. | In |
| `9997` | TCP | [Splunk TCP](sources-splunk) Source. | Out |
| `10060` | TCP | [TCP (Raw)](sources-tcp-raw) data. | In |
| `10070` | TCP | [TCP JSON](sources-tcp-json) data. | In |
| `10080` | TCP | Collection from [HTTP JSON](sources-https) Sources. | In |
| `10200` | HTTP/S | [Cribl HTTP](destinations-cribl-http) Destination. | In |
| `10300` | TCP | [Cribl TCP](destinations-cribl-tcp) Destination. | In |
### Cribl.Cloud
Cribl.Cloud provides a set of ports linked to Sources enabled by default for your Workspace. To view them:
1. From your Cribl.Cloud Organization's top bar, select **Products**.
1. Then from the sidebar, select **Cribl** > **Workspace**, and then **Data Sources**.
Additionally, Cribl.Cloud makes the `20000` - `20010` port range available for configuring other Sources.
| Available Ports | Protocol | Purpose | Direction |
| --- | --- | --- | --- |
| `443` | TCP | Mapped to `10443` internally. Available for TCP-based Sources that require port `443`. | In |
| `20000` - `20010` | TCP | Additional Sources in Cribl.Cloud. | In |
#### Reusing Port 443 for Other Services {#port-443-reuse}
External port `443` is transparently mapped by the load balancer to internal port `10443` on the Fleet. You can use this port for TCP-based services that require port `443` (such as DocuSign).
To use port `443` for a service:
1. Configure a Source to listen on port `10443` internally.
2. External clients connect to your Cribl.Cloud endpoint on port `443`, which the load balancer routes to port `10443` on the Edge Node.
If multiple services share this port, differentiate traffic using a pre-processing Pipeline. For example, inspect `__headers` or other request metadata to identify the data source.
> No other custom ports can be opened for Cribl-managed Worker Groups in Cribl.Cloud beyond port `443` and ports `20000`-`20010`.
{.box .info}
### Billy Port {#cribl-copilot-port}
To use [Billy](/copilot/copilot-chat), your Cribl Edge deployment must be able to establish a connection to `ai.cribl.cloud` on port `443`.
############################################################
# SOURCE: https://docs.cribl.io/edge/configure-env-vars.md
############################################################
# Configure Environment Variables
Configure environment variables to control Cribl Edge behavior.
---
The procedure for configuring environment variables depends on the operating system the Cribl Edge Node is deployed to.
Whenever you change environment variables, you must restart Cribl Edge on the affected Nodes.
> For a full list of available variables, see [Environment Variables](environment-variables).
{.box .info}
## Configure Environment Variables on Linux
### Set Variables with systemd
If Cribl Edge runs as a systemd service, add environment variables to the systemd override file
(for example, `/etc/systemd/system/cribl-edge.service.d/override.conf`) so they persist across restarts and upgrades.
1. Open the override file for editing:
```shell
sudo systemctl edit cribl-edge
```
1. Add the variables under the `[Service]` section:
``` {title="override.conf"}
[Service]
Environment=CRIBL_NOEXEC=1
Environment=HTTP_PROXY=http://proxy.example.com:3128
```
1. Reload the systemd daemon and restart Cribl Edge:
```shell
sudo systemctl daemon-reload
sudo systemctl restart cribl-edge
```
For details on persisting overrides, see [Persisting Overrides](deploy-boot-start#persisting-overrides).
### Set Variables Without systemd
If you are not using systemd, add `export` lines to a shell profile file
(such as `/etc/environment` or the Cribl user's `~/.bashrc`) so they load on login:
```shell
export CRIBL_NOEXEC=1
export HTTP_PROXY=http://proxy.example.com:3128
```
## Configure Environment Variables on Windows
### Set Variables During Installation
You can configure environment variables while installing Cribl Edge on Windows
using either the interactive wizard or a parameter for the MSI installer.
- **Wizard**: In the setup wizard, in the **Set Environment Variables** modal,
enter one variable per line. See [Install Edge on Windows Using the Wizard](deploy-windows#wizard) for more details.
- **MSI installer**: When installing via MSI, use the `SETENVVARS` parameter with pipe-delimited key-value pairs.
The entire string must be quoted:
```shell
msiexec /qn /i cribl--