On This Page

Home / Edge/ Packs/Pack-Based Configuration Management

Pack-Based Configuration Management

As your Cribl deployment grows, maintaining consistency across multiple Fleets and environments becomes a significant operational challenge. To solve this challenge, use Pack-based configuration management to templatize your entire end-to-end Cribl data streams into reusable units that you can port to other Fleets in your deployment. This approach is helpful when you need standardized configuration management workflows to manage multiple Fleets, instances, or environments (such as development, testing, and production).

Pack-based configuration management has several advantages over the standard Leader-to-Fleet configuration management workflow:

  • Composable management: Define end-to-end data streams as a single unit. Package the configurations for a Source, its associated processing logic, and the final Destination into a reusable Pack that you can deploy across Cribl Edge to different Fleets and environments.

  • Git version-control: You can version-control Packs in an external Git repository to make your Cribl configurations auditable and to flexibly adapt to your company’s change management policies.

  • Environmental portability: Move configurations easily between environments, such as from dev to prod or more. Packs allow you to bind configuration fields to variables, so that you can apply the same configuration components to multiple Fleets and then customize the configuration details that are specific to the local context of a given Fleet.

  • Scalability: Onboard new data streams or spin up additional Fleets quickly by deploying Packs that have been validated in one context to another.

When To Use Pack-Based Configuration Management
​

Pack-based configuration management is not required to manage deployments of Cribl Edge. For many deployments, the default Leader-to-Worker configuration management system is sufficient.

However, this workflow is particularly beneficial for organizations that require:

  • The ability for multiple configuration content developers or teams to collaborate on configurations. Use this workflow if you need developers to work on configurations in parallel. Each content developer can work on isolated feature branches, merge changes systematically, and resolve conflicts before they impact production.
  • Two- or three-branch environment promotion and segregation workflows. This workflow supports the classic development three-branch model where changes flow from dev to test to prod branches and environments in a controlled manner. You could also use a simpler two-branch model flow from dev to prod instead.
  • Formal configuration approval from a central Change Advisory Board (CAB). Git workflows keep configuration changes visible so that stakeholders can review and approve them. An external Git repository can provide additional change traceability and a clear audit trail of who approved what and when.
  • The ability to roll back configurations quickly. While this is already possible in Cribl Edge, the Pack-based configuration management workflow extends and complements the Leader-to-Fleet control plane system by allowing you to track changes in an external Git repository. Git’s version control capabilities make rollbacks straightforward so that you can easily revert to a previous, stable version. Connecting to an external repository provides additional visibility and auditability that integrates with your company’s overall change management system.

Prior experience with Git is helpful for this workflow. For users familiar with Git, this workflow can streamline version control, change management, and CI/CD for your Cribl environments.

Key Components
​

The recommended Pack-based configuration management workflow for Cribl Edge uses these components:

  • Packs: Packs contain the actual configuration content that you will track. This includes dependencies such as the Knowledge objects or (optionally) the sensitive resources (such as certificates or secrets) needed to run the Pack.
  • Environments: A logical grouping of Cribl Edge resources that process data, depending on your Fleet placement strategy. An environment is typically defined by a specific set of Fleets.
  • Variables: Dynamic placeholders within a Pack configuration that map to a specific values defined at either the Pack or Fleet level. Variables allow you to port the Pack to another environment, then make small adjustments to the logic to allow for specific differences that are unique to that Fleet.
  • External Git repository: A centralized Git repository serves as the single source of truth, tracking all changes to your Pack configuration. Each repository should track one individual Pack only.

Packs
​

The core building block of every Cribl Edge configuration management system is the Pack. A Pack contains a bundle of configurations that you want to develop, test, version control, and distribute to multiple Organizations or Fleets. Packs can contain configurations for many types of resources:

  • Sources
  • Destinations
  • Event Breakers
  • Pipelines
  • Routes
  • Knowledge Objects (such as lookup files)
  • Sample data files

In other words, you can create Packs that contain a full, end-to-end Cribl Edge configuration. Think of a Pack as a configuration object that you can version control and export/import to other environments. It works well in workflows where multiple developers work in a sandbox development environment. Developers can review each other’s work and merge their changes to a testing or production environment using Git workflows.

Environments
​

An Environment is a logical boundary that defines how data is processed, managed, and secured within your architecture. The logical boundary is defined by your Fleet placement strategy, such as geographic regions or data proximity. It could also refer to content development environments that use the two- or three-branch development strategy, such as dev, staging, and prod.

While typically defined by a specific set of Fleets, an environment in the Pack-based workflow encompasses the unique variables and configurations required for that specific environment. This means a single Pack can be deployed across multiple environments while automatically adapting to the new context.

Variables
​

Variables allow you to copy or import that Pack to multiple Fleets, making small adjustments to the logic to adapt to the specific environment context for that Fleet. For example, you could adjust the Fleet-level variables that account for regional differences (such as different AWS regions), VPC endpoints, or local port assignments. This ensures that the core data processing business logic remains intact while adapting to the local requirements of its new Fleet context. It also allows you to easily make updates to Packs and deploy those updates with minimal overhead.

External Git Repositories
​

Once you’ve built an end-to-end configuration of Cribl Edge inside a Pack, you can then integrate it with an external Git repository for version control, parallel development, and automated deployment with CI/CD.

Cribl Edge is compatible with all major Git platforms, including GitHub, GitLab, and Bitbucket.

Additional Configuration Management Components
​

This table explains how standard configuration management components map to Cribl Edge:

ElementExplanationHow it Applies in Cribl
Automation agent or runnerA process or tool that pulls from Git and applies the config to the target environment.Handled inside Cribl Edge by committing and deploying imported Packs to Fleets. Can also be handled using a CI/CD pipeline tool in external repositories, including GitHub Actions, GitLab CI, and Jenkins.
Change workflowConfiguration changes go through a review and approval process.Handled in an external Git repository linked to Cribl Edge in compliance with your internal change management policies.
Target system(s)Where the configuration is ultimately applied.Instances of Cribl Edge.
Secret managementSecrets or encrypted files.Handled inside Cribl Edge and mapped to variables in external Git repositories.
Environment mappingDirectories or branches per environment.Handled with variables inside Cribl Edge and mapped to variables in external Git repositories.

Considerations
​

When working with Packs, keep these considerations in mind:

  • Packs cannot be cloned or imported from one Cribl product to another. For example, you cannot clone a Pack from Cribl Stream to Cribl Edge and vice versa.
  • Sources and Destinations created in Packs add the Pack name to the __inputId and __outputId fields of their events. For example: datagen:My_Pack.my_datagen_source, or cribl_http:My_Pack.my_http_destination.