Packs

Packs enable Cribl Stream administrators and developers to pack up and share complex configurations and workflows across multiple Worker Groups, or across organizations.

Packs = Portability

With a Cribl Stream deployment of any size, using Packs can simplify and accelerate your work. Packs can also accelerate internal troubleshooting, and accelerate working with Cribl Support, because they facilitate quickly replicating your Cribl Stream environment.

For example, where a Pipeline’s configuration references Lookup file(s), Cribl Stream will import the Pipeline only if the Lookups are available in their configured locations. A Pack can consolidate this dependency, making the Pipeline portable across Cribl Stream instances. If you import the Lookups into the Pack, you can develop and test a configuration, and then port it from development to production instances, or readily deploy it to multiple Worker Groups.

We don’t claim to have brokered world peace here, but we do modestly hope to promote a stable, prosperous Pax Criblatica for the Cribl Stream ecosystem.

What Is a Pack?

Packs are implemented as a user interface (described on this page) and as a .crbl file format.

What’s in a Pack?

Currently, a Pack can pack up everything between a Source and a Destination:

  • Routes (Pack-level)
  • Pipelines (Pack-level)
  • Functions (built-in and custom)
  • Sample data files
  • Knowledge objects (Lookups, Parsers, Global Variables, Grok Patterns, and Schemas)
A Pack with internal Routes & Pipelines; no Knowledge or samples
A Pack with internal Routes & Pipelines; no Knowledge or samples

As the above list suggests, a Pack can encapsulate a whole set of infrastructure for a given use case.

Packs don’t have access to Knowledge objects (Lookups, Global Variables, etc.) that are defined outside the Pack. Knowledge objects defined within a Pack are not available to Cribl Stream resources outside the Pack. In either scenario, you must explicitly duplicate the Knowledge objects in the opposite scope.

What’s Not in a Pack?

Sources, Collectors, and Destinations are external to Packs, so you can’t specify them within a Pack. This excludes a few other things:

  • Routes configured within a Pack can’t specify a Destination.
  • Packs can’t include Event Breakers, which are associated with Sources. (However, you can instead use the Event Breaker Function in Packs’ contained Pipelines.)

You connect a Pack with a Source and Destination by attaching it to a Route (see below), just as you’d attach a Pipeline.

Where Can I Get Some Packs?

Easy now. See The Cribl Packs Dispensary™ below.

Using Packs

These instructions cover using predefined Packs, as well as creating and modifying Pack configurations.

Version Compatibility

Packs created or modified in Cribl Stream 4.0.x cannot be used in any pre-4.0 version of Cribl Stream. (If you try, you’ll see a should NOT have additional properties error.) To avoid this problem, Cribl recommends that you upgrade to Cribl Stream 4.0.x or later.

For compatibility questions about any individual Packs, please contact us in Cribl Community Slack’s #packs channel.

Where Can I Use Packs?

Wherever you can reference a Pipeline, you can specify a Pack:

  • In Sources, where you attach pre-processing Pipelines.
  • In Destinations, where you attach post-processing Pipelines.
  • In Routes, in the Routing table’s Pipeline/Output column.

This expanded view shows how a Pack can replace a Pipeline in a Route:

A Pack snaps into Cribl Stream like an enhanced Pipeline
A Pack snaps into Cribl Stream like an enhanced Pipeline

Packs are distinguished in the display with a PACK badge, as you can see here in the Routing table:

PACKs badged in Routing table’s Pipeline column
PACKs badged in Routing table’s Pipeline column

The PACK badge is also displayed when you click into a resource – shown here on one of the Routes from the above table:

PACK badge on a Pack connected to a Route
PACK badge on a Pack connected to a Route

Cribl Stream’s Monitoring page includes a Packs link where you can monitor Packs’ throughput.

Accessing Packs

You access Packs differently, depending on your deployment type.

Single-Instance Deployment

In a single-instance deployment, Packs are global. From Cribl Stream’s top-level navigation, just select Processing > Packs. On the filesystem, Packs (including those that you add) are stored at $CRIBL_HOME/default/.

Packs, single-instance navigation
Packs, single-instance navigation

Distributed Deployment

In a distributed deployment, Packs are associated with (and installed within) Worker Groups. Select Manage, and then click into the Worker Group you want to manage (for example, default). Next, from that Worker Group’s top nav, select Processing > Packs (Stream) or More > Packs (Edge).

Stream Group > Manage Packs page
Stream Group > Manage Packs page
Edge Fleet > Manage Packs page
Edge Fleet > Manage Packs page

Each Group’s Packs are stored at at $CRIBL_HOME/groups/<group-name>/default/. (In a typical installation, you’ll find the starter HelloPacks Pack at /opt/cribl/groups/default/default/.)

By design, you can readily share Packs across Worker Groups by exporting/importing them (both operations covered below).

To unpack Packs, use the above instructions (per deployment type) to navigate to the HelloPacks example Pack shipped with Cribl Stream. On the Manage Packs page, click this Pack’s row to see its configuration.

Click Pipelines on the Pack’s submenu, and you’ll see that the Pack includes devnull, main, and passthru Pipelines, corresponding to the default Pipelines provided at Cribl Stream’s global level. This Pack also includes an Apache-specific sample Pipeline – click it to unpack that, too.

Pipelines within example Pack
Pipelines within example Pack

Click Routes on the Pack’s submenu, and you’ll see that this Pack also provides both a default and an Apache-specific Route.

Configuring a Pack

Once loaded, each Pack displays a submenu with familiar links: Routes, Pipelines, Knowledge, and Settings on the left pane, along with Sample Data, and Preview Simple on the right. The Pack’s submenu is a subset of Cribl Stream’s top nav.

Configuring a Pack
Configuring a Pack

The left pane’s submenu links give you access to configuration objects specific to this Pack.

Sample Data

The right pane defaults to displaying all sample data files available on your Cribl Stream instance. If you prefer to filter only sample files internal to the Pack, toggle In Pack only to the right.

If you add sample data files via this Pack UI, they will be internal to that Pack. Each sample file here displays its own In Pack toggle on its row, which works as follows:

A light-blue toggle is locked, meaning that this sample file is internal to the Pack. It will export with the Pack. If you want to make this sample available across Cribl Stream, you’ll need to also add it via the global right preview pane (accessed from Routing > Data Routes or Processing > Pipelines).

A grayed-out or dark-blue toggle means that this sample file is global to Cribl Stream. It is available to this Pack. Toggle this to Yes (dark blue) if you want the sample file to export along with the Pack.

Sample file in Pack
Sample file in Pack

Basically, you can manipulate all the options here as you’d work with their big sister or brother in Cribl Stream’s global navigation.

Importing or Upgrading a Pack

To import a new Pack, or an updated version of an existing Pack, from your filesystem:

  1. Navigate to the Manage Packs page.
  2. Click Add Pack at the upper right.
  3. Select your desired Add/Import source: Dispensary, File, URL, or Git repo.
  4. Follow the above links to details on each of these options.
Importing a Pack
Importing a Pack
Custom Functions

Packs can include Pipelines containing custom functions, which can (in turn) run arbitrary JavaScript. Before you install a Pack, make sure it comes from a provider you trust, such as the Cribl Packs Dispensary or your own organization.

As an additional protection layer, all Pack import modals provide an Allow custom functions toggle. In the toggle’s default No position, if Cribl Stream detects custom functions in the specified Pack, it will block the import with an error message. If you trust the Pack’s provider,set the toggle to Yes, and the import will proceed normally.

The Cribl Packs Dispensary™

You might be wondering, “Where can I find a reliable source of Packs that add useful features to Cribl Stream, vetted for safety?”

Well, Cribl is proud to point you to the Cribl Packs Dispensary™. Here, Cribl’s own solutions engineers have seeded several strains of high-productivity Cribl Stream configurations. Because the Packs Dispensary™ is a place to share good stuff, we expect many new hybrids to sprout from the community. Cribl will test and curate submissions to ensure the quality of the repo’s contents.

You can install Dispensary Packs directly through Cribl Stream’s UI, as outlined in Add from Packs Dispensary below.

Cribl Packs Dispensary™ (as displayed in Cribl Stream&rsquo;s <strong>Add</strong> drawer)
Cribl Packs Dispensary™ (as displayed in Cribl Stream’s Add drawer)

Interested in publishing your own Packs on the Cribl Packs Dispensary™? See Publishing a Pack.

Add from Packs Dispensary

To add a Pack from the Cribl Packs Dispensary™ sharing site:

  1. From the Manage Packs page’s New Pack submenu, select Add from Dispensary.
  2. The Packs Dispensary will open in a drawer, as shown in the screenshot above.
  3. Using the drawer’s controls, browse or search for the Pack(s) you want. (You can use the check boxes at the left to filter by data type, use case, and technology.)
  4. Click any Pack’s tile to display its details page with its README. This will typically outline the Pack’s purpose, compatibility, requirements, and installation.
  5. To proceed, click Add Pack on this page.
  6. That’s it! You’ll see a banner confirming that the Pack is now installed.

Import from File

To import a Pack (.crbl file) from your local filesystem:

  1. From the New Pack submenu, select Import from File.
  2. From the resulting File Open dialog, select the file to import.
  3. Optionally, give the pack an explicit, unique New Pack ID. (For details about this option, see Upgrading an Existing Pack below.)
  4. Where appropriate (see just above), enable Allow custom functions.
  5. Click OK to confirm the import.
Importing from a file
Importing from a file

Import from URL

To import a Pack from a known, public or internal, URL:

  1. From the New Pack submenu, select Import from URL.
  2. Enter a valid URL for the Pack’s source. (This field’s input is validated for URL format, but not for accuracy, before you submit the modal.)
  3. Optionally, give the pack an explicit, unique New Pack ID. (See Upgrading an Existing Pack.)
  4. Where appropriate, enable Allow custom functions. (See Custom Functions.)
  5. Click OK to confirm the import.
Confirming file import from URL
Confirming file import from URL

To import a Pack from a public URL, Cribl Stream’s Leader Node (or single instance) requires Internet access. A distributed deployment’s Leader can then deploy the Pack to Workers even if the Workers lack Internet access.

Import from Git Repos

To import a Pack from a known public or private Git repo:

  1. From the New Pack submenu, select Import from Git.

  2. Enter the source repo’s valid URL.
    This field’s input is validated for URL format, but not for completeness or accuracy, before you submit the modal. When targeting a private repo, use the format: https://<username>:<token/password>:<repo‑address>. Public repos need only https://<repo‑address>, as shown in the example below.

  3. Optionally, give the pack an explicit, unique New Pack ID. (See Upgrading an Existing Pack.)

  4. Optionally, enter a Branch or tag to filter the import source using the repo’s metadata. You can specify a branch (such as master) or a tag (such as a release number: 0.5.1, etc.).

  5. Where appropriate (see Custom Functions), enable Allow custom functions.

  6. Click OK to confirm the import.

Importing from a Git repo
Importing from a Git repo

To import a Pack from a public repo, Cribl Stream’s Leader Node (or single instance) requires Internet access. A distributed deployment’s Leader can then deploy the Pack to Workers even if the Workers lack Internet access.

Dispensary GitHub Repo

One authoritative public repo is the Cribl Pack Dispensary on GitHub. (This is the precursor to the Cribl-hosted Cribl Packs Dispensary™ site.)

You can install Dispensary Packs directly through Cribl Stream’s UI, as outlined in Import from Git Repos above. However, if you prefer, you can click through to any Dispensary repo’s release page, download the corresponding .crbl file, and then upload the file into Cribl Stream.

Downloading a <code>.crbl</code> file from the Cribl Pack Dispensary&rsquo;s Web UI
Downloading a .crbl file from the Cribl Pack Dispensary’s Web UI

If you’ve posted completed Packs to our GitHub repo, we encourage you to now submit them to our new Cribl Packs Dispensary™ site. See Publishing a Pack.

Upgrading an Existing Pack

Each Pack that is installed within a given Worker Group (or single-instance deployment) must have a unique ID. The ID is based on the Pack’s internal configuration – not its container’s file name, nor on its Display name.

If you import a Pack whose internal ID matches an installed Pack – whether an update, or just a duplicate – you’ll be prompted to assign a unique New Pack ID to import it as a separate Pack.

Renaming a Pack on import
Renaming a Pack on import

You’ll also have the option to Overwrite the installed Pack, reusing the same ID.

If you toggle this option to Yes, the imported Pack will completely overwrite your existing Pack’s configuration.

Each Pack within a Cribl Stream instance must have a unique Pack ID, so you cannot share an ID between two (or more) installed Packs.

To explicitly upgrade an existing Pack, you can instead click the menu icon on its row and select Upgrade.

Upgrading an existing Pack
Upgrading an existing Pack

If you’ve modified an installed Pack, Cribl Stream will block the overwrite of the Pack, to prevent deletion of your locally created resources.

When upgrading a Pack, Cribl recommends:

  • Import the updated Pack under a new name that includes the version number (e.g., cribl‑syslog‑input‑120). This allows you to review and adjust new functionality against currently–deployed configurations.
  • Do a side–by–side comparison of the previous and new versions of the Pack – remember to review all comments in the new Pack. Doing this side-by-side comparison allows you to copy Function expressions and other settings from the current version into the same fields in the new version.
  • Enable or disable any Functions in the new Pack as necessary.
  • Update any Routes, Pipelines, Sources, or Destinations that use the previous Pack version to reference the new Pack.
  • If the Pack includes any user–modified Knowledge objects (e.g., Lookup files), be sure to copy the modified files locally for safe keeping before upgrading the Pack. After installing the upgrade, copy those files over the Pack upgrade’s default versions.
  • Test, test, and test!
  • Commit and Deploy.

Creating a Pack

You can create a new Pack from scratch, to consolidate and export multiple Cribl Stream configuration objects:

  1. Navigate to the Manage Packs page.

  2. Click Add Pack.

  3. From the submenu, select Create Pack.

  4. In the resulting New Pack modal, fill in a unique Pack ID and other details.

    • Each Pack within a Cribl Stream instance must have a separate Pack ID, but you can assign arbitrary Display names.
    • Version is a required field identifying the Pack’s own versioning.
    • Minimum Stream version is an optional field specifying the lowest compatible version of Cribl Stream software.
    • Description and Author are optional identifiers.
    • Data type, Use cases, and Technologies are optional combo boxes. You can insert one or multiple keywords to help users filter Packs that you post publicly on the Cribl Packs Dispensary™.
    • Tags are optional, arbitrary labels that you can use to filter/search and organize Packs.
  5. Click Save.

Creating a Pack
Creating a Pack
  1. On the Manage Packs page, click the new Pack’s row to open the Pack.
Manage Packs page
Manage Packs page
  1. Use the standard Cribl Stream controls (see above) to configure and save the infrastructure you want to pack up. As you save changes in the UI, they’re saved to the Pack.

If you’d like to share your Pack with the community of Cribl users, you can publish it on the Cribl Packs Dispensary™.

The Cribl Packs Dispensary™ site is designed for sharing completed Packs. If you want to collaborate with others on iteratively developing a Pack, Cribl recommends relying on our Dispensary GitHub Repo for the development phase.

If you have a Cribl.Cloud account, you can also collaborate there by inviting team members to your Cribl.Cloud Organization. See Inviting and Managing Other Users.

Once your Pack is ready to share, we encourage you to submit it to the Cribl Packs Dispensary™ site. If you already have completed Packs on our GitHub repo, bring them over here!

Modifying Pack Settings

You can update a Pack’s metadata (Version, Description, Author, etc.) and display settings. If you’re developing a new Pack to share, you’ll want to use this interface to populate the Pack’s README and display logo.

  1. From the Pack’s submenu, select Pack Settings. The left README tab will gain focus.
  2. To populate the Pack’s README file, toggle View to Edit, replace the placeholder markdown content, and Save.
Editing Pack&rsquo;s README
Editing Pack’s README
  1. To update other metadata, click the left Settings tab.
Editing Pack&rsquo;s metadata
Editing Pack’s metadata
  1. To add a Pack logo, click the Pack’s Settings > Display left tab.

    Cribl recommends adding a logo to each custom Pack, to visually distinguish the Pack’s UI from the surrounding Cribl Stream UI (as well as from other Packs). You can upload a .png or .jpg/.jpeg file, up to a maximum size of 2MB and 350x350px. Cribl recommends a transparent image, sized approximately 280x50px.

Exporting a Pack

To export a newly created or modified Pack, click its menu icon on the Packs page and select Export.

Exporting a Pack
Exporting a Pack

The resulting Export Pack modal provides the following options.

Export Mode

Select one of these three buttons:

  • Merge safe: Attempt to safely merge local modifications into the Pack’s default layer (original configuration), then export.

  • Merge: Force-merge local modifications into the Pack’s original configuration, then export.

    Merge is the only export mode available when you’ve selected Groups as your Export target.

  • Default only: Export only the Pack’s original configuration, without local modifications.

The Merge safe option is conservative, and will block the export where Cribl Stream can’t readily merge conflicting, modified contents with the Pack’s original contents:

<strong>Merge safe</strong> error
Merge safe error

If you encounter an error like the example shown above, use the Merge or Default only export mode instead.

Export Target

The options here are:

  • File (the default): You’ll be prompted to confirm a file name and destination after you click OK.
    (In Cribl Stream 3.5 and higher, the default file name automatically includes the Pack’s version number.)

  • Groups: Selecting this displays a Groups control, prompting you to select one or multiple existing Worker Groups/Fleets to export the Pack to. (The current Worker Group is automatically omitted from the options.)

Exporting Multiple Packs

You can export multiple Packs in one operation, by selecting their check boxes and then clicking Export multiple Packs. This option comes with a few constraints:

  • You can export multiple Packs only to Groups (not to Files).
  • Therefore, this option is available only in distributed deployments.
  • The only export mode available is Merge.
  • The Exported Pack ID field is disabled, and hidden.
Exporting multiple packs at once
Exporting multiple packs at once

A status modal will list any Packs that failed to export.

Exported Pack ID

Defaults to the Pack’s current ID, with the version number appended. This is an opportunity to change the Pack’s ID if you’re exporting it as you develop a new version.

Managing Packs via API

You can perform Pack operations by running Cribl Stream API calls on the command line. This is required if you plan to automate Pack operations, e.g., in a CI/CD pipeline.

In this section, we’ll walk through one scenario where running API calls on the command line works well: exporting a Pack from one Worker Group and installing it into another. The two Worker Groups/Fleets do not need to have the same Leader Node.

About the Following Examples
  • The API calls here include Worker Group names as path parameters.
  • The curl commands assume that you have set the $token environment variable to match the value of a bearer token. Of course, this is just one option for authentication. See the Authentication topic for others; adapt the example commands to suit your chosen approach.

Export via API

Adapt and run this Export pack API call, using the export mode of your choice:

GET /api/v1/m/<worker_group_name>/packs/<pack_name>/export?mode=merge

Export Example

Let’s export a Pack named goat-herd from the default Worker Group, and use the > redirect to write the exported Pack to a file named goat-herd.crbl:

curl -X GET -H "Authorization: Bearer $token" 'https://stream:9000/api/v1/m/default/packs/goat-herd/export?mode=merge' > goat-herd.crbl

This request returns an octet-stream attachment which is downloaded as a .crbl file. And voilà, you have exported your Pack.

Install via API

Installing the exported Pack in a different Worker Group is a two-step process: First upload, then actually install.

Install via API – Step 1

Adapt and run this Upload pack API call, referencing the exported Pack file:

PUT /api/v1/m/<new_worker_group>/packs?filename=<pack_name>.crbl

Install Example – Step 1

We’ll use our target Worker Group name (in this example, it’s group420). Then we need to specify the exported Pack contents as a file payload, using the --data-binary option to upload the binary data without modification. The @ prefix tells curl that goat-herd.crbl is the path to the file, not the data itself.

curl -X PUT -H "Authorization: Bearer $token" 'https://stream:9000/api/v1/m/group420/packs?filename=goat-herd.crbl' --data-binary "@goat-herd.crbl"

This request returns a JSON object of the following form:

{"source":"pack_name.random_id.crbl"}

Install via API – Step 2

Adapt and run this Install pack API call:

POST /api/v1/m/<new_worker_group>/packs

Meanwhile, remember that this API call will need a payload – the JSON object returned by the previous API call.

Install Example – Step 2

We’ll use the curl -d option to specify the JSON object payload. We’ll add a new element to the object, whose key is id, and whose value is the Pack’s new name in the new Worker Group.

Here, the goat-herd Pack is renamed as billys_pack. (If you do not wish to rename the Pack, just omit the id element – but keep the source element.)

curl -X POST -H "Authorization: Bearer $token" -H "Content-Type: application/json" 'https://stream:9000/api/v1/m/group420/packs' -d '{"source":"goat-herd.987654321.crbl", "id":"billys_pack"}'

Copy via API

To bulk-copy Packs between Worker Groups/Fleets, adapt and run this Clone pack API call, referencing a source Worker Group, destination Worker Group(s)/Worker Group(s), and Pack(s).

POST /api/v1/packs/__clone__

{
  "srcGroup": "copy_from_this_worker_group_id",
  "dstGroups": [
    "destination_group_1",
    "destination_group_2",
    ...
  ],
  "packs": [
    "pack_id_1",
    "pack_id_2",
    ...
  ]
}

Copy Example

For example, to copy the Palo Alto Networks and Cisco ASA Packs from the default to dc1-logs and dc2-logs Worker Groups:

curl -X POST -H "Authorization: Bearer $token" -H "Content-Type: application/json" 'https://stream:9000/api/v1/packs/__clone__' -d '{"srcGroup":"default","dstGroups":["dc1-logs","dc2-logs"],"packs":["PAN","cribl_cisco_asa_cleanup"]}'