As your Cribl deployment grows, maintaining consistency across multiple Worker Groups and environments becomes a significant operational challenge. To solve this challenge, use Pack-based configuration management to templatize your entire end-to-end Cribl data streams into reusable units that you can port to other Worker Groups in your deployment. This approach is helpful when you need standardized configuration management workflows to manage multiple Worker Groups, instances, or environments (such as development, testing, and production).

Pack-based configuration management has several advantages over the standard Leader-to-Worker Group configuration management workflow:

• Composable management: Define end-to-end data streams as a single unit. Package the configurations for a Source, its associated processing logic, and the final Destination into a reusable Pack that you can deploy across Cribl Stream to different Worker Groups and environments.

• Git version-control: You can version-control Packs in an external Git repository to make your Cribl configurations auditable and to flexibly adapt to your company’s change management policies.

• Environmental portability: Move configurations easily between environments, such as from dev to prod or more. Packs allow you to bind configuration fields to variables, so that you can apply the same configuration components to multiple Worker Groups and then customize the configuration details that are specific to the local context of a given Worker Group.

• Scalability: Onboard new data streams or spin up additional Worker Groups quickly by deploying Packs that have been validated in one context to another.

When To Use Pack-Based Configuration Management

​

Pack-based configuration management is not required to manage deployments of Cribl Stream. For many deployments, the default Leader-to-Worker configuration management system is sufficient.

However, this workflow is particularly beneficial for organizations that require:

• The ability for multiple configuration content developers or teams to collaborate on configurations. Use this workflow if you need developers to work on configurations in parallel. Each content developer can work on isolated feature branches, merge changes systematically, and resolve conflicts before they impact production.
• Two- or three-branch environment promotion and segregation workflows. This workflow supports the classic development three-branch model where changes flow from dev to test to prod branches and environments in a controlled manner. You could also use a simpler two-branch model flow from dev to prod instead.
• Formal configuration approval from a central Change Advisory Board (CAB). Git workflows keep configuration changes visible so that stakeholders can review and approve them. An external Git repository can provide additional change traceability and a clear audit trail of who approved what and when.
• The ability to roll back configurations quickly. While this is already possible in Cribl Stream, the Pack-based configuration management workflow extends and complements the Leader-to-Worker Group control plane system by allowing you to track changes in an external Git repository. Git’s version control capabilities make rollbacks straightforward so that you can easily revert to a previous, stable version. Connecting to an external repository provides additional visibility and auditability that integrates with your company’s overall change management system.

Prior experience with Git is helpful for this workflow. For users familiar with Git, this workflow can streamline version control, change management, and CI/CD for your Cribl environments.

Key Components

​

The recommended Pack-based configuration management workflow for Cribl Stream uses these components:

• Packs: Packs contain the actual configuration content that you will track. This includes dependencies such as the Knowledge objects or (optionally) the sensitive resources (such as certificates or secrets) needed to run the Pack.
• Environments: A logical grouping of Cribl Stream resources that process data, depending on your Worker Group placement strategy. An environment is typically defined by a specific set of Worker Groups.
• Variables: Dynamic placeholders within a Pack configuration that map to a specific values defined at either the Pack or Worker Group level. Variables allow you to port the Pack to another environment, then make small adjustments to the logic to allow for specific differences that are unique to that Worker Group.
• External Git repository: A centralized Git repository serves as the single source of truth, tracking all changes to your Pack configuration. Each repository should track one individual Pack only.

Packs

​

The core building block of every Cribl Stream configuration management system is the Pack. A Pack contains a bundle of configurations that you want to develop, test, version control, and distribute to multiple Organizations or Worker Groups. Packs can contain configurations for many types of resources:

• Sources
• Destinations
• Event Breakers
• Pipelines
• Routes
• Knowledge Objects (such as lookup files)
• Sample data files

In other words, you can create Packs that contain a full, end-to-end Cribl Stream configuration. Think of a Pack as a configuration object that you can version control and export/import to other environments. It works well in workflows where multiple developers work in a sandbox development environment. Developers can review each other’s work and merge their changes to a testing or production environment using Git workflows.

Environments

​

An Environment is a logical boundary that defines how data is processed, managed, and secured within your architecture. The logical boundary is defined by your Worker Group placement strategy, such as geographic regions or data proximity. It could also refer to content development environments that use the two- or three-branch development strategy, such as dev, staging, and prod.

While typically defined by a specific set of Worker Groups, an environment in the Pack-based workflow encompasses the unique variables and configurations required for that specific environment. This means a single Pack can be deployed across multiple environments while automatically adapting to the new context.

Variables

​

Variables allow you to copy or import that Pack to multiple Worker Groups, making small adjustments to the logic to adapt to the specific environment context for that Worker Group. For example, you could adjust the Worker Group-level variables that account for regional differences (such as different AWS regions), VPC endpoints, or local port assignments. This ensures that the core data processing business logic remains intact while adapting to the local requirements of its new Worker Group context. It also allows you to easily make updates to Packs and deploy those updates with minimal overhead.

External Git Repositories

​

Once you’ve built an end-to-end configuration of Cribl Stream inside a Pack, you can then integrate it with an external Git repository for version control, parallel development, and automated deployment with CI/CD.

Cribl Stream is compatible with all major Git platforms, including GitHub, GitLab, and Bitbucket.

Additional Configuration Management Components

​

This table explains how standard configuration management components map to Cribl Stream:

Considerations

​

When working with Packs, keep these considerations in mind:

• Packs cannot be cloned or imported from one Cribl product to another. For example, you cannot clone a Pack from Cribl Stream to Cribl Edge and vice versa.
• Sources and Destinations created in Packs add the Pack name to the __inputId and __outputId fields of their events. For example: datagen:My_Pack.my_datagen_source, or cribl_http:My_Pack.my_http_destination.