GitOps

With Cribl Stream’s GitOps features, you can manage Cribl Stream configuration with standard version-control systems and CI/CD flow. You can separate development from production configurations, and thus, safely build and continuously deploy your observability pipelines. The production environment will be read-only, and thus strongly protected against unauthorized or unintended changes. This intentionally restrictive approach is a good fit for some but not all organizations.

The GitOps workflow described here assumes that your development, staging, and production environments are identical or nearly identical. If, on the other hand, your development environment differs significantly from production (in its branching structure, for example, or if users have wide latitude for experimentation), you should consider solutions other than GitOps. If you have questions about how GitOps compares to various alternatives, please contact Cribl Support.

GitOps requires an Enterprise license and a distributed Cribl Stream deployment.

Cribl.Cloud does not support GitOps.

How to Use This Doc​

This doc has three sections.

• Work through all three sections, in order, once.
• Once you have done that, you are finished with the first two sections, Preparing Your Git Repo and Preparing Cribl Stream.
• Next, incorporate the HTTP request from the third section (GitOps Workflow) into your CI/CD tool infrastructure.
• From then on, your normal production routine will be to just repeat the GitOps Workflow (that is, the third section) again and again.

You’ll perform setup steps in the Cribl Stream UI, in your Git management system, and at the command line. Screenshots are in light mode for Cribl Stream, and in dark mode for GitHub examples.

Environment and branch names have been chosen to help you keep track of what you’re doing – if you decide to use different ones, exercise care.

Procedures begin with instructions that are agnostic to the Git management system you want to use, whether that is free GitHub, GitHub Enterprise, GitLab, Bitbucket, or a self-hosted Git server. The detailed instructions that follow are for use with free GitHub. If you’re using one of the other Git management systems, you’ll need to substitute appropriate instructions for the ones that begin “To do this in GitHub …”

Check out Cribl’s free GitOps sandbox for a “deep dive” and to work through a practical example.

If you encounter either of the following situations, see the linked appendix:

• You need to upgrade your GitOps-enabled Cribl Stream deployment.
• You are not using systemd, and need to override the defaults using environment variables.

Preparing Your Git Repo​

The procedure in this section only needs to be done once. When you’ve completed it, you can prepare Cribl Stream for GitOps.

Creating SSH Public and Private Keys​

You’ll use SSH to connect to your Git management system; Cribl does not recommend using basic authentication for GitOps.

At the command line, generate public and private SSH keys, specifying Ed25519 as the key type:

ssh-keygen -t ed25519 -C "your_email@example.com"

Verify that the ~/.ssh/ directory contains both of the newly-generated keys:

• id_ed25519 (the private key that Cribl Stream will use).
• id_ed25519.pub (the public key that your Git management system will use).

Copy the public SSH key to your clipboard:

cat ~/.ssh/id_ed25519.pub

Associate the public key with your Git management account.

To do this in GitHub:

1. If you do not have a GitHub account, create one.
2. Sign in to your GitHub account.
3. On the SSH and GPG keys page, click New SSH key.
4. Paste the key in the Key field.
5. Click Add SSH key.

Creating a Private Git Repo​

You need to create a dedicated, private Git repo – your Development environment will write to it, and your Production environment will sync to it. Create a private repo named cribl that contains one branch, named main, now.

To do this in GitHub:

1. In your GitHub profile page, click Go to your personal profile.
2. On the resulting page, select the Repositories tab.
3. Click New to create a new repository.
4. Name the repo cribl.
5. Change the repo from Public to Private.
6. Click Create repository. The new repo’s landing page will open.
7. In the Quick setup box, click creating a new file to open the text editor.

   This step and the next are required because they establish the commit history that procedures in the next section depend on.

8. Enter some text, enter a name for the file, and click Commit changes (once in the text editor and again in the pop-up).

By default, the repo will contain one branch named main.

Setting Up the prod Branch​

We’ve chosen names that always make it clear whether we’re talking about environments or about branches:

• Development and Production will be your Cribl Stream environments.
• dev and prod will be the branches in your Git repo.

In this section, we’ll set up prod; later on, we’ll set up dev.

Recall that your Git repo has one branch, named main. Now, rename main as prod.

To do this in GitHub:

1. In your GitHub repo’s landing page (e.g., https://github.com/<your_Git_username>/cribl), select the 1 branch tab. (That’s what the branches tab will be named, assuming that the sole branch is main).

   

   

2. In the main branch’s row, click the pencil icon at far right, and rename main as prod.

Preparing Cribl Stream​

The procedure in this section only needs to be done once. When you’ve completed it, you can move on to testing and using GitOps.

Setting Up Your Environments​

In your distributed Cribl Stream deployment, set up two separate environments, one named Development and the other named Production. Cribl strongly recommends that you use systemd to manage the cribl service.

Complete the following steps in both environments:

• Create one Cribl Leader and two Cribl Workers.
• On the Leader, install a Cribl Enterprise License.
• On the Leader, install git using the following command (or the equivalent for your OS):
  sudo yum install git -y

Then, in the Production environment:

• On the Leader, install jq – the GitOps workflow instructions below use jq.
  • Run this command (or the equivalent for your OS):
    sudo yum install jq -y
• Note the Production Leader’s IP address for use later on.

Configuring Your Production Leader​

First, you’ll ensure that your Production Leader and your Git repo can communicate; then, you’ll put your Production Leader into read-only mode.

Creating the gitops User​

1. Create a new local user called gitops.
2. On the Production Leader’s local filesystem, temporarily record the password for the gitops user; we’ll refer to it as GITOPS_USER_PASSWORD.
3. Associate the default gitops Role with the gitops user.
   • This Role’s only permission is to POST to the /version/sync endpoint of the Cribl API (in the API Reference, see versioning).

Adding a Secret to Git​

Your Git repo needs to be able to communicate with your Production Leader. This requires that you store a secret in your Git management system. We’ll use GITOPS_USER_PASSWORD as the secret.

From the Production Leader, upload the GITOPS_USER_PASSWORD to the Git management system (or perform an equivalent operation) now.

To do this in GitHub:

1. In the GitHub site for your repo, navigate to Settings > Secrets and variables.
2. Click New repository secret and follow the prompts.

When done, you should see a GITOPS_USER_PASSWORD row under Actions secrets:

Connecting Your Production Environment to Your Git Repo​

In Cribl Stream, in the UI for the Production Leader, select global ⚙️ Settings (lower left) > Git Settings.

In the Remote tab:

1. In Remote URL, enter the URL for your private Git repo.
   • On GitHub, the Remote URL will look like this:
     git@github.com:<your_GitHub_username>/cribl.git
2. Make sure that in the Authentication type drop–down, SSH (the default) is selected.
3. In SSH private key, copy and paste the private key (~/.ssh/id_ed25519).

   The key will paste in with an appended newline below it. Do not delete this newline before you save the Remote Settings, or else you will trigger an error of the form: fatal: Could not read from remote repository.

4. Toggle SSH strict host key checking off. This will allow the remote repository to accept your host key information. If you miss this step, you will trigger an error.
5. Click Save.
6. Toggle SSH strict host key checking on again.
7. Click Save (again!).
8. Commit and Push.

In the General tab:

1. Set Branch to prod.
2. Leave GitOps Workflow set to None.
3. Click Save.

Check your private Git repo – in the prod branch, you should now see your commit.

Making the Production Leader Read-Only​

In a terminal on the Production Leader, become root:

sudo su -

Add systemctl overrides to set environment variables that will (a) make the Production Leader read-only upon startup, and (b) set the branch to prod.

1. Use systemctl to open the override.conf file in an editor:

   systemctl edit cribl 

   Copy

2. When prompted, add the following overrides:

   [Service]
   Environment=CRIBL_GIT_OPS=push
   Environment=CRIBL_GIT_BRANCH=prod

   Copy

3. Restart:

   systemctl daemon-reload && systemctl restart cribl

   Copy

Creating the dev Branch​

Recall that your Git repo has one branch, named prod. Now, create a new branch named dev, from your existing prod branch.

To do this in GitHub:

1. In your GitHub repo’s landing page (e.g., https://github.com/<your_Git_username>/cribl), select the 1 branch tab. (That’s what the branches tab will be named, assuming that the sole branch is prod).

   

   

2. Click New branch to open the Create a branch modal.

3. Make sure that Source is set to prod.

4. In the New branch name field, enter dev.

   

   

5. Click Create new branch to close the modal.

6. Back in the Branches tab, click Overview to refresh the tab – you should now see both the prod and dev branches listed.

Configuring Your Development Leader​

First, you’ll point your Development Leader to the dev branch in the remote repo. Then, you’ll ensure that one or more users exist that can push configs to that repo.

Pointing Your Development Leader to Its Branch​

In Cribl Stream, in your Development Leader’s UI, select global ⚙️ Settings (lower left) > Git Settings.

In the Remote tab:

1. In Remote URL, enter the URL for your private Git repo – and if the URL includes a Git username, this should be gitops, the one you created above.
   • On GitHub, the Remote URL will look like this:
     git@github.com:<your_GitHub_username>/cribl.git
2. Make sure that in the Authentication type drop–down, SSH (the default) is selected.
3. In SSH private key, copy and paste the private key (~/.ssh/id_ed25519).

   The key will paste in with an appended newline below it. Do not delete this newline before you save the Remote Settings, or else you will trigger an error of the form: fatal: Could not read from remote repository.

4. Toggle SSH strict host key checking off. This will allow the remote repository to accept your host key information. The remote host’s information is stored in ~/.ssh/known_hosts. If you miss this step, you will trigger an error.
5. Click Save.
6. Toggle SSH strict host key checking on again.
7. Click Save (again!), but do not Commit or Push yet.

In the General tab:

1. Set Branch to dev.
2. Leave GitOps Workflow set to None.
3. Click Save, but do not Commit or Push yet.

Restart the Development Leader. When it comes back up, it will be able to write to your Git repo.

Setting Up Users to Push to the Remote Repo​

You will need one or more users capable of pushing configs to the remote repo. There is more than one way to achieve this. Your choice will depend on the needs of your organization.

• Any user with the admin role already has ability to push to the remote repo. If this is your preferred approach, skip to the next section.
• If you prefer to allow one or more regular users to push to the remote repo, without having the power of the admin role, you can create such users by completing the procedure below.

First, edit the roles.yml file to make a new push_to_remote role. (We’re using the command line here because you cannot perform this operation in the UI.)

1. SSH into the Leader node.

2. cd into your $CRIBL_HOME/local/cribl directory.

3. Open or create the roles.yml file in a text editor.

4. Without changing the existing text in the file, paste in the following snippet:

   push_to_remote:
     policy:
       - POST /version/sync
       - POST /version/commit
       - POST /version/push
       - GET /system/settings/git-settings
     description: Members with this role can push configs to a remote repo.

   Copy

5. Restart Cribl Stream.

Next, back in the UI, create a new config_pusher user.

1. In global ⚙️ Settings (lower left) > Access Management > Local Users, click Add User to open the user creation modal.
2. Name the new local user config_pusher.
3. In addition to the usual configuration steps, use the Roles drop-down to associate the push_to_remote role with the config_pusher user.
4. Click Save.
5. Commit and Push.

Verify That Your New User Can Push to the Remote Repo​

1. Log out, then log back in as config_pusher.
2. The Version Control drop-down should now display a Push button.

Later, when you are ready to push new configs to the remote repo, click this new Push button.

GitOps Workflow​

If you’ve completed the above preparatory sections, your GitOps workflow is ready go. Here’s an overview, with links to detailed procedures.

• Work in the Development Environment: All your commits and testing happen in the Development environment first; you push changes to the dev branch in your Git repo.
• Merge Changes from dev to prod: When you are satisfied that your changes are ready for production, you create a pull request that brings the prod branch up-to-date with dev. This is all in your Git management system, and changes nothing in your Cribl Stream Production environment.
• Sync Your Production Leader to Git: For your changes to take effect in production, you will send an HTTP request to the /version/sync endpoint of the Cribl API, on the Production Leader. Once you are in production, it will be your CI/CD infrastructure – not you – that sends the request. This will cause the Production Leader to pull in the latest changes from your Git repo’s prod branch – without violating the Production Leader’s read-only status.

The procedures include a harmless test task so that you can validate your setup and acquire a feel for the workflow.

Working in the Development Environment​

In this step, you’ll make changes in the Development environment, and subsequently propagate them to Production. To test this with a minimal unit of work, start by adding a Route.

In Cribl Stream, in the Development Leader’s UI, navigate to Routing > Data Routes.

1. Click Add Route.

2. Configure the new Route as follows:

   Name	Filter	Pipeline	Output
   test-gitops	true	devnull	devnull:devnull

3. Click Commit as shown in the screenshot below.

   

   

4. When the Git Changes modal opens, add a commit message and click Commit and Deploy as shown in the screenshot below.

   

   

5. Finally, push the changes to your remote repo via the Global Config dialog, whose appearance will vary depending on how you’ve configured Git Settings > Collapse Actions. Make sure the dialog says that the Git branch you’re pushing to is dev, as configured above.

Merging Changes from dev to prod​

In your Git management system’s dev branch, you should see the new route.yml appearing in the /groups/<group_name>/local/cribl/ directory.

Merge the changes from the dev branch to prod now.

To do this in GitHub:

1. Click Compare & pull request. Be sure that the base drop-down says prod and the compare drop-down says dev.

   

   

2. Add some comments.

3. Click Create pull request. The UI should say that you want to merge 1 commit into prod from dev.

4. If GitHub detects conflicts, fix them.

5. Have the appropriate person on your team review the pull request.

6. You (or the appropriate person) should now click Merge pull request, then Confirm merge.

If the merge is successful, GitHub will display a confirmation message. Your committed changes have now been merged from dev to prod.

Syncing Your Production Leader to Git​

Your changes so far have all been done in the Development environment. Meanwhile, you have committed to the dev branch, and pushed to the prod branch in your Git management system – but your Production environment has not changed.

Since, for safety’s sake, you made the Production environment read-only, you can neither write to it nor update it via a UI. Instead, the following must happen:

• You will send an HTTP request to the /version/sync endpoint of the Cribl API, on the Production Leader (in the API Reference, see versioning). Once you are in production, it will be your CI/CD infrastructure – not you – that sends the request.

• This will cause the Production Leader to pull in the latest changes from your Git repo’s prod branch – without violating the Production Leader’s read-only status.

Below you’ll see a script you can use to send the HTTP request.

• This script is generic, and will require you to adapt it to work with the Git management system you’re using.
• Once you run the script, you should see the changes you made in your Development environment (in this example, the new Route) in the Production environment, too.

Syncing to Git: Generic Script​

Substitute your Production Leader’s IP address and the gitops user’s password for the placeholders.

#!/bin/bash

# Authenticate and Save the Token.
TOKEN=$(curl http://<Production_Leader_IP_address>:9000/api/v1/auth/login \
     -H 'Content-Type: application/json' \
     -d '{"username":"gitops","password":"$GITOPS_USER_PASSWORD"}' 2>/dev/null | \
     jq -r .token)

# Set up the Authentication Header
export AUTH_HEAD="Authorization: Bearer $TOKEN"

# Make the notification call
curl -X POST "http://<Production_Leader_IP_address>:9000/api/v1/version/sync" -H "accept: application/json" -H "${AUTH_HEAD}" -d "ref=prod&deploy=true"

Making the HTTP Request from CI/CD​

Now that you have run one of the above scripts to prove that your GitOps workflow works, you should incorporate the script’s HTTP request into your CI/CD setup. Once that is done, you’re in production with GitOps! To develop, test, and push changes to production, just keep running through the whole GitOps workflow.

Appendix: Upgrading​

Upgrading a Cribl Stream GitOps deployment is essentially a variation on the day-to-day GitOps workflow described in the previous section, with the key difference that you take the Production environment out of, then put it back into, read-only mode.

Once you have begun the upgrade, do not make any changes unrelated to the upgrade process, in either the Development or Production environment, until the upgrade process is complete. For example, do not start creating any new Routes or Pipelines.

First, in your Development environment:

1. Validate that Cribl Stream is working as expected, and that the latest changes have been pushed to the dev branch in your Git repo.
2. Stop the Cribl Stream Leader and back up $CRIBL_HOME (the Cribl install directory).
3. Use the UI to upgrade the Leader and Workers.
4. Create a PR to merge the dev branch into prod. Merge the PR.

Next, in your Production environment:

1. Stop the Cribl Stream Leader and back up $CRIBL_HOME (the Cribl install directory).
2. Take your Production environment out of read-only mode according to the instructions below.
3. Use the UI to upgrade the Leader and Workers – but do not commit changes after the upgrade.
4. Put the Leader back into read-only mode according to the instructions above.
5. Sync the Production environment to the prod branch in your Git repo.

Finally, confirm that all Workers in both environments have the latest configuration.

Taking the Production Leader Out of Read-Only Mode​

In a terminal on the Production Leader, become root:

sudo su -

Comment out the systemctl overrides that set environment variables that (a) had made the Production Leader read-only upon startup, and (b) had set the branch to prod.

1. Use systemctl to open the override.conf file in an editor:

   systemctl edit cribl

   Copy

2. When prompted, comment out the Git-related overrides as shown:

   [Service]
   # Environment=CRIBL_GIT_OPS=push
   # Environment=CRIBL_GIT_BRANCH=prod

   Copy

3. Restart:

   systemctl daemon-reload && systemctl restart cribl

   Copy

Appendix: Environment Variables​

If you are not using systemd to manage the cribl service, you can use environment variables to override the defaults in the UI.