Version Control
Tracking, backing up, and restoring configuration changes for single-instance and distributed deployments
Cribl Stream integrates with Git clients and remote repositories to provide version control of Cribl Stream’s configuration. This integration offers backup and rollback for single-instance and distributed deployments.
These options are separate from the Git repo responsible for version control of Worker configurations, located on the Leader Node in distributed deployments. We cover all these options and requirements below.
Cribl.Cloud deployments do not currently support integration with external Git clients or remote repos.
Git Installation (Local or Standalone/Single-Instance)
To verify that git
is available, run:
git --version
The minimum version that Cribl Stream requires is: 1.8.3.1. If you don’t have git
installed, see the installation links here.
Git Required for Some Features
Git is a hard requirement for certain Cribl Stream features.
Distributed Deployments
For distributed deployments, git
must be installed and available locally on the host running the Leader Node.
All configuration changes must be committed before they are deployed. The Leader notifies Workers that a new configuration is available, and Workers pull the new configuration from the Leader Node.
Licensing Dashboard
Even on single-instance deployments, the Monitoring > Licensing dashboard will display configuration change markers only if you have git
installed.
Committing Changes
Once Git is installed, you can commit configuration changes using the git
CLI. You can also commit changes interactively, using Cribl Stream’s UI.
Pending commits have a red dot indicator, as shown below. Click Commit to proceed.
Next, in the resulting Commit Changes modal, you can verify the diff’ed configuration changes. Other options here include clearing individual files’ check boxes to exclude them from the commit (as shown below), and clicking Undo to reverse the changes instead of committing them.
When you’re ready to commit to your commit, click Commit. Look for a Commit successful confirmation message at the lower right.
Reverting Commits
Once Git is installed, you can revert to a previous commit using the git
CLI. You can also restore a Worker Group’s previous commit using Cribl Stream’s UI:
Select the commit from the Config Version drop-down, as shown below. (Note that you’re selecting not a single commit to roll back, but the commit point to roll back to.)
Then, in the resulting Commit modal, verify the diff’ed configuration changes and click Revert.
Finally, confirm permission for Cribl Stream to restart.
Support For Remote Repositories
Git remote repositories are supported – but not required – for version control of all configuration changes.
This feature requires a Cribl Stream Enterprise or Standard license.
You can configure a Leader Node with Git remote push capabilities through the Cribl Stream CLI, or through the Cribl Stream UI (via Settings > Global Settings > Distributed Settings > Git Settings).
To create a repo, see these tutorials:
- Setting Up a Repository (CLI instructions, host-agnostic, from Atlassian).
- Creating a New Repository (specific to GitHub’s Web UI).
- Create a Repo (longer GitHub-specific tutorial, also covers committing changes).
Remote Formats Supported
Remote URI schema patterns should match this regex:(?:git|ssh|ftps?|file|https?|git@[-\w.]+):(\/\/)?(.*?)(\.git\/?)?$
.
You can find a list of supported formats here.
For example:
- GitHub or other providers:
<protocol>://git@example.com/<username>/<reponame>.git
- Local Git servers:
git://<host.xyz>:<port>/<user>/path/to/repo.git
Securing Local Configuration and Remote Repos
Make a local backup of your whole
$CRIBL_HOME
directory before you set a remote repo. This safeguards your Cribl Stream configuration against being overwritten by a misconfigured remote.Create each new remote as an empty repo, with no readme file and no commit history. (See the GitHub and GitLab examples below.)
Some files that are used by Cribl Stream (both Leader and Worker Groups) contain sensitive keys; examples are
cribl.secret
and...auth/ssh/git.key
. These will be pushed to the remote repo as part of the entire directory structure under version control. Ensure that this repo is secured appropriately.
Connecting to a Remote Repo over HTTPS (Recommended)
Cribl recommends connecting to a remote repo over HTTPS. The examples below show a token-based and a password-based HTTPS connection to GitHub.
Example: Connecting to GitHub over HTTPS with a Personal Access Token
Create a new GitHub repository.
For best results, create a new empty repo, with no readme file and no commit history. (This will preventgit push
errors.) Note the username and email associated with your login on the repo provider.Create a personal access token with repo scope.
Copy the token to your clipboard.
In Cribl Stream, go to Settings > Global Settings > System > Git Settings.
In the Remote tab, select Authentication type: None.
Fill in the Remote URL field with your username and repo name. Use the format below:
https://<username>:<token>@github.com/<owner>/<repository>.git
For additional details, see GitHub’s Creating a Personal Access Token tutorial.
For GitHub repos specifically, use only personal access tokens in the Remote URL field. GitHub announced its end of support for plaintext passwords as of August 13, 2021.
Example: Connecting to GitHub over HTTPS with Username and Password
Create a new GitHub repository.
For best results, create a new empty repo, with no readme file and no commit history. (This will preventgit push
errors.) Note the username and email associated with your login on the repo provider.In Cribl Stream, go to Settings > Global Settings > System > Git Settings.
In the Remote tab, select Authentication type: Basic.
Enter your credentials in the User and Password fields.
Connecting to a Remote with SSH
You can set up SSH keys from the CLI, or upload keys via the UI. If you have a passphrase set, this functionality is available only through the CLI – see Encryption: Configuring Keys with the CLI. The example below outlines the UI steps.
Example: Connecting to GitHub with SSH
Create a new GitHub repository.
For best results, create a new empty repo, with no readme file and no commit history. (This will preventgit push
errors.) Note the user name and email associated with your login to the repo provider.Add an SSH public key to your GitHub account.
In Cribl Stream, go to global Settings > Global Settings > System > Git Settings > Remote.
Fill in the Remote [repo] URL. In the generic example below, replace
<username>
with your user name on the repo provider:
Remote URL:<protocol>://git@github.com:<username>/<reponame>.git
For GitHub specifically, the URL/protocol format must be:
Remote URL: git@github.com:<user>/<reponame>.git
A specific (fictitious) GitHub example:
Remote URL: git@github.com:taylorswift/leadsheets.git
Paste in the SSH private key.
The key will paste in with an appended newline below it. Do not delete this newline before you save the Remote Settings, or else you will trigger an error of the form:
fatal: Could not read from remote repository.
As the user running Cribl Stream, run this command to add the GitHub keys to
known_hosts
:
ssh-keyscan -H github.com >> ~/.ssh/known_hosts
For additional details, see GitHub’s Connecting to GitHub with SSH tutorial.
GitLab Notes
For repos hosted on GitLab, Cribl’s general recommendations are:
Create a new empty (blank) project, with no readme file and no commit history. (Clear the Initialize repository with a README check box.)
This will prevent
git push
errors by creating a project with no branch, not evenmain
. When you later push Cribl Stream’smaster
branch, that will create a correspondingmaster
branch on the GitLab remote with no conflicts.Create a GitLab project access token for authentication. See GitLab’s documentation, which also covers project bot conventions.
With project bots, the first token’s username is set to
project_{project_id}_bot
. The password is the alphanumeric token.Create the token with
write_repository
scope.Specify a remote URL in HTTPS format – e.g.:
https://localgitlab.<yourdomain.ext>/<yourusername>/cribl.git
If you do see a
main
branch in Cribl Stream’s UI, do not select it. Leave the branch atmaster
, then use the Stream UI’s Commit and Push commands. This will create the desiredmaster
branch on GitLab’s remote.
GitLab’s Repository Settings > Push Rules section includes these two settings of interest:
As needed, enable Check whether author is a GitLab user.
Understand the consequences of of enabling Prevent committing secrets to Git. This blocks commits of
.pem
and.key
files. If you have certificates or SSH keys configured, this will break commits from Cribl Stream, throwing only a genericAPI Error
in the UI. Check yourgit
CLI client for more-specific diagnostics.
Additional Git Settings
On the Git Settings > General tab, you can modify the following configurations, all of which are optional.
- Branch, GitOps workflow: These drop-downs are available to configure GitOps.
See the GitOps documentation linked above for details and precautions.
- Collapse actions: Combine multiple Git buttons to one, to reduce repetitive clicks. See Collapse Actions below.
- Default commit message: Enter a placeholder message to apply to all commits from Cribl Stream. This also reduces clicks.
- Git timeout: Maximum time (in milliseconds) to wait for Git processes to complete before killing them. If a Leader instance hangs, try lowering this value from the default
60000
(60 seconds). However, avoid very low values (like20
ms) – which will time out even when Git processes are working properly. Enter0
to wait indefinitely.
On the Git Settings > Remote tab, you can change the Authentication Type from its SSH default to Basic authentication. This displays two additional fields:
- User: Username on the repo.
- Password: Authentication password (e.g., a GitHub personal access token).
GitHub (specifically) does not support Basic authentication.
On the Git Settings > Scheduled Actions tab, you can schedule a Commit, Push, or Commit & Push action to occur on a predefined interval.
For the selected action type, you can define a cron schedule, and a commit message distinct from the General tab’s Default Commit Message. Then click Save.
You can schedule only one type of action. To swap to a different type, select it from the Scheduled global actions drop-down, and resave. To turn off scheduled Git commands, select None from the drop-down, and resave.
Pushing Configs to a Remote Repo
Once you’ve configured a remote, a Git Push button appears in the Global Config dialog. You can use this to copy committed configuration changes to your remote.
The branch indicator will normally read
master
, as shown above, unless you have enabled GitOps with an appropriate license.
Collapse Actions
If you enabled the Git Settings > Collapse Actions option, you will be offered a combined Commit & Push button in the Global Config dialog.
At the Group level, enabling Collapse Actions adds a combined Commit & Deploy option to the modal you see after clicking Commit for the Group’s config.
Enabling Collapse Actions with a remote repo simplifies the Commit Changes confirmation dialog. It substitutes just a commit Message text box, with OK and Cancel buttons – omitting the diff view. Don’t enable this option if you prefer to inspect configuration changes before committing.
Backing Up Configs Out of Band
Once you’ve configured your remote repo, changes to all Cribl Stream config files under the $CRIBL_HOME
directory will normally be backed up to your remote whenever you click the above Git Push or Commit & Push UI buttons. The exceptions are:
- Any configuration files you’ve chosen to store outside
$CRIBL_HOME
. - Any configuration files residing in paths you’ve added to
.gitignore
.
To back up these files to your remote, you’ll need to either:
- Change the above paths/settings, or
- Use an external
git
client to manually push them.
Troubleshooting Push Errors
To resolve errors commonly encountered when pushing to a remote repo, see:
Troubleshooting Resources
Cribl University offers a Troubleshooting Criblet on Commit & Deploy best practices and error recovery. To follow the direct course link, first log into your Cribl University account. (To create an account, click the Sign up link. You’ll need to click through a short Terms & Conditions presentation, with chill music, before proceeding to courses – but Cribl’s training is always free of charge.) Once logged in, check out other useful Troubleshooting Criblets and Advanced Troubleshooting short courses.
Restoring Leader from a Remote Repo
If a remote repo is configured and has the latest known good Leader configuration, this section outlines the general steps to restore the config from that repo.
Let’s assume that either the entire $CRIBL_HOME
directory of the Leader is corrupted, or you’re replicating the remote repo’s config onto a fresh instance. Let’s also assume that the remote repo has the form:
git@github.com:<username>/<reponame>.git
.
- Important: In a directory of choice, untar the same Cribl Stream version that you’re trying to restore, but do not start it.
Make sure you download the matching Cribl Stream version. Cribl’s Download page foregrounds Cribl Stream’s current version, but provides a link to this archive of prior releases.
- Change directory into
$CRIBL_HOME
and initializegit
:
# git init
- If you are using SSH key authentication, specify the key using the following command:
GIT_SSH_COMMAND='ssh -i </path/to/github/repo>.key -o IdentitiesOnly=yes'
As an alternative to executing
GIT_SSH_COMMAND
on the fly, you can set your key in$CRIBL_HOME/.git/config
with:git config core.sshCommand "ssh -i /path/to/key"
Or set it globally with:
git config --global core.sshCommand "ssh -i /path/to/key"
- Ensure that you have proper access to the remote repo:
# git ls-remote git@github.com:<username>/<reponame>.git
56331fabb4822eaec4ca0ffd008d6e9974c1e419f HEAD
5631fabb4822eaec4ca0ffd008d6e9974c1e419f refs/heads/master
- Next, add/configure the remote:
# git remote add origin git@github.com:<username>/<reponame>.git
- Now set up your local branch to exactly match the remote branch:
# git fetch origin
# git reset --hard origin/master
- Finally, to confirm that the commits match, run this command while in
$CRIBL_HOME
:
# git show --abbrev-commit
You should see output indicating that HEAD
points to both master
and origin/master
, as in this example:
commit 5631fab (HEAD -> master, origin/master)
Author: First Last <email@example.com>
Date: Fri Jan 31 10:16:07 2020 -0500
admin: Last commit before failure/crash
......
Step 6 above pulls in all the latest configs from the remote repo. Step 7 confirms the local repo matches the remote. You should now be able to start the Leader as normal. Once up and running, Workers should start checking in after about 60 seconds.
Verify cribl.secret
The
cribl.secret
file – located at$CRIBL_HOME/local/cribl/auth/cribl.secret
– contains the secret key that is used to encrypt sensitive settings on configuration files (e.g., AWS Secret Access Key, etc.). Make sure this file is properly restored on the new Leader, because it is required to make encrypted conf file settings usable again.
.gitignore File
A .gitignore
file specifies files that git
should ignore when tracking changes. Each line specifies a pattern, which should match a file path to be ignored. Cribl Stream ships with a .gitgnore
file containing a number of patterns/rules, under a section of the file labeled CRIBL SECTION
.
# Do NOT REMOVE CRIBL and CUSTOM header lines!
# DO NOT REMOVE rules under the CRIBL section as they may be reintroduced on update.
# You can ONLY comment out rules in the CRIBL section.
# You can add new rules in the CUSTOM section.
### CRIBL SECTION -- DO NOT REMOVE ###
default/ui/**
default/data/ui/**
bin/**
log/**
pid/**
data/uploads/**
diag/**
**/state/**
#### CUSTOM SECTION -- DO NOT REMOVE ###
<User-defined patterns/rules go here>
CRIBL Section
Do Not Remove CRIBL SECTION or CUSTOM SECTION Headers
Under the
CRIBL SECTION
header, Cribl Stream specifies the files that Git will ignore by default. Further files for Git to ignore should be listed under theCUSTOM SECTION
header.Do not add any lines to the
CRIBL SECTION
block, because the next update will just overwrite them anyway. Do not remove any lines either. The only modifications that will survive updates are commented lines.Do not remove any section header lines, or unexpected behavior might occur on update.
CUSTOM Section
User-defined, custom patterns/rules can be safely defined under the CUSTOM SECTION
. Cribl Stream will not modify the contents of CUSTOM SECTION
.
Good candidates to add here include large lookup files – especially large binary database files. For details, see Git Push Errors: Large Files Detected.
Files skipped with .gitignore
If you have files that you’ve set .gitgnore
to skip, you will need to back them up and restore them by means other than Git. For example, you can periodically copy/rsync them to a backup destination, and then restore them to their original locations after you complete the steps above.
Files specified in .gitignore
are not only excluded from pushes to the remote repo, but are also excluded from Worker Group config bundles. When Workers load a new config that references a skipped (and missing) file, this can produce unexpected results, and usually errors.
For example, if you add **/auth/certs/leader*
to .gitignore
, this will prevent the leader.crt
and leader.key
files from leaking to Worker Nodes in config deployments.
Configs that you .gitignore
in this way on the Leader will not persist on Workers after they receive their next deploy. However, these configs will be available in the next few rolling backup bundles in state/
.
Commit and Deploy via the API
You can automate commit and deploy commands by using Cribl Stream API requests. The following examples show how to do so for a Worker Group’s configuration.
About the Examples
The API calls below include Worker Group names as path parameters.
The
curl
commands assume that you have set the$token
environment variable to match the value of a bearer token. See Authentication for alternative approaches; adapt the example commands to suit your chosen approach.
Commit Changes
To commit pending changes to your configured repo, adapt and run the following API call:
POST /api/v1/version/commit
{"message":"a descriptive commit message","group":"<worker_group_name>"}
You will receive a JSON response with some details about the commit:
{"items":[{"branch":"master","commit":"abcd1234","summary":{"changes":"1","insertions":"4","deletions":0}}],"count":1}
From that response, you’ll need to extract the commit ID (abcd1234
) to use in the second deploy API call below.
Commit Example
Here, let’s commit all pending changes to the default Worker Group:
curl -X POST -H "Authorization: Bearer $token" -H "Content-Type: application/json" "https://stream:9000/api/v1/version/commit" -d '{"message":"automation@cribl: commit","group":"default"}'
Deploy Changes
To deploy your committed config changes to Worker Groups, adapt and run the following API call. As the version
value, you’ll need the commit ID you received in the commit response above:
PATCH /api/v1/master/groups/<worker_group_name>/deploy
{"version":"abcd1234"}
A successful response payload looks like this:
{"items":[{"description":"Default Worker Group","tags":"default","configVersion":"5b8f42a","id":"default"}],"count":1}
Deploy Example
Here, let’s deploy the previously committed configuration to the default
Worker Group:
curl -X PATCH -H "Authorization: Bearer $token" -H "content-type: application/json" "https://stream:9000/api/v1/master/groups/default/deploy" -d '{"version":"abcd1234"}'
Selectively Commit Changes
If you want to commit only a subset of configuration changes, adapt and use the following payload:
POST /api/v1/version/commit
{
"message":"descriptive commit message",
"group":"default",
"files":["groups/default/data/samples/sample-g0aT.json","groups/default/local/cribl/samples.yml"]
}
Selective Commit Example
Let’s selectively commit a sample data file, and the updated YAML listing of all sample files, to the default
Worker Group:
curl -X POST -H "Authorization: Bearer $token" -H "Content-Type: application/json" "https://stream:9000/api/v1/version/commit" -d '{"message":"automation@cribl: commit","group":"default","files":["groups/default/data/samples/sample-g0aT.json","groups/default/local/cribl/samples.yml"]}'