Cribl LogStream ‚Äď Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF ‚Äď v.3.1.2

Packs

About Packs

Packs, introduced in LogStream 3.0, enable LogStream administrators and developers to pack up and share complex configurations and workflows across multiple Worker Groups, or across organizations.

Packs = Portability

With a LogStream deployment of any size, using Packs can simplify and accelerate your work. Packs can also accelerate internal troubleshooting, and accelerate working with Cribl Support, because they facilitate quickly replicating your LogStream environment.

For example, where a Pipeline's configuration references Lookup file(s), LogStream will import the Pipeline only if the Lookups are available in their configured locations. A Pack can consolidate this dependency, making the Pipeline portable across LogStream instances. You can develop and test a configuration, and then port it from development to production instances, or readily deploy it to multiple Worker Groups.

We don't claim to have brokered world peace here, but we do modestly hope to promote a stable, prosperous Pax Criblatica for the LogStream ecosystem.

What Is a Pack?

Packs are implemented as a user interface (described on this page) and as a .crbl file format.

What's in a Pack?

Currently, a pack can pack up everything between a Source and a Destination:

  • Routes (Pack-level)
  • Pipelines (Pack-level)
  • Functions (built-in and custom)
  • Sample data files
  • Knowledge objects (Lookups, Parsers, Global Variables, Grok Patterns, and Schemas)
A Pack with internal Routes & Pipelines; no Knowledge or samplesA Pack with internal Routes & Pipelines; no Knowledge or samples

A Pack with internal Routes & Pipelines; no Knowledge or samples

As the above list suggests, a Pack can encapsulate a whole set of infrastructure for a given use case.

What's Not in a Pack?

Sources, Collectors, and Destinations are external to Packs, so you can't specify them within a Pack. This excludes a few other things:

  • Routes configured within a Pack can't specify a Destination.
  • Packs can't include Event Breakers, which are associated with Sources.

You connect a Pack with a Source and Destination by attaching it to a Route (see below), just as you'd attach a Pipeline.

Where Can I Get Some Packs?

Easy now. See The Cribl Pack Dispensary‚ĄĘ below.

Using Packs

These instructions cover using predefined Packs, as well as creating and modifying Pack configurations.

Where Can I Use Packs?

Wherever you can reference a Pipeline, you can specify a Pack:

  • In Sources, where you attach pre-processing Pipelines.
  • In Destinations, where you attach post-processing Pipelines.
  • In Routes, in the Routing table's Pipeline/Output column.
A Pack snaps into LogStream like an enhanced PipelineA Pack snaps into LogStream like an enhanced Pipeline

A Pack snaps into LogStream like an enhanced Pipeline

Packs are distinguished in in the display with a PACK badge, as you can see here in the Routing table:

PACKs badged in Routing table's Pipeline columnPACKs badged in Routing table's Pipeline column

PACKs badged in Routing table's Pipeline column

The PACK badge is also displayed when you click into a resource ‚Äď shown here on one of the Routes from the above table:

PACK badge on a Pack connected to a RoutePACK badge on a Pack connected to a Route

PACK badge on a Pack connected to a Route

LogStream's Monitoring page includes a Packs link where you can monitor Packs' throughput.

Accessing Packs

You access Packs differently, depending on your deployment type.

Single-Instance

In a single-instance deployment, Packs are global. From LogStream's top-level navigation, just click Packs.

Packs, single-instance navigationPacks, single-instance navigation

Packs, single-instance navigation

Distributed/Default Worker Group

In a distributed deployment with the default single Worker Group (Leader mode), select Configure from the left nav, then Packs from the resulting top nav.

Packs, Leader modePacks, Leader mode

Packs, Leader mode

Distributed/Mutliple Worker Groups

In a distributed deployment with multiple Worker Groups (Leader mode), Packs are associated with (and installed within) Worker Groups. Navigate to the parent Worker Group, then select Packs from that Group's top nav.

Worker Group > Manage Packs pageWorker Group > Manage Packs page

Worker Group > Manage Packs page

ūüĎć

As the top nav adds more controls on narrower browsers, Packs and other right-side links can move onto the ‚ÄĘ‚ÄĘ‚ÄĘ overflow menu, as shown above.

By design, you can readily share Packs across Worker Groups by exporting/importing them (both covered below).

Getting Started with Packs

To unpack Packs, use the above instructions (per deployment type) to navigate to the HelloPacks example Pack shipped with LogStream. On the Manage Packs page, click this Pack's row to see its Pack's configuration.

Manage Packs page with example PackManage Packs page with example Pack

Manage Packs page with example Pack

Click Pipelines on the Pack's submenu, and you'll see that the Pack includes devnull, main, and passthru Pipelines, corresponding to the default Pipelines provided at LogStream's global level. This¬†Pack also includes an Apache-specific sample Pipeline ‚Äď click it to unpack that, too.

Click Routes on the Pack's submenu, and you'll see that this Pack also provides both a default and an Apache-specific Route.

Pack Configuration

Once loaded, each Pack displays a submenu with familiar links ‚Äď a subset of LogStream's top nav above it: Routes, Pipelines, Knowledge, and Settings on the left pane, along with Sample¬†Data, and Preview Simple on the right.

Configuring a PackConfiguring a Pack

Configuring a Pack

The left pane's links give you access to configuration objects specific to this Pack. In the right pane, you can toggle between displaying All sample files available on your LogStream instance, versus samples internal to the the Pack Only.

Basically, you can manipulate all the options here as you'd work with their big sister or brother in LogStream's global navigation.

Importing or Upgrading a Pack

To import a new Pack, or an updated version of an existing Pack, from your filesystem:

  1. Navigate to the Manage Packs page.
  2. Click + Add New.
  3. Select your desired Import from source: File, URL, or Git repo.
Importing a PackImporting a Pack

Importing a Pack

ūüöß

Custom Functions

Packs can include Pipelines containing custom functions, which can (in turn) run arbitrary JavaScript. Before you install a Pack, make sure it comes from a provider you trust, such as the Cribl Pack Dispensary or your own organization.

LogStream 3.1.2 and higher provide an additional layer of protection: All Pack import modals provide an Allow custom functions slider. In the slider's default No position, if LogStream detects custom functions in the specified Pack, it will block the import with an error message. If you trust the Pack's provider, toggle the slider to Yes, and the import will proceed normally.

Import from File

To import a Pack (.crbl file) from your local filesystem:

  1. From the + Add New submenu, select Import from File.
  2. From the resulting File Open dialog, select the file to import.
  3. Optionally, give the pack an explicit, unique New Pack ID. (For details about this option, see Upgrading an Existing Pack below.)
  4. Where appropriate (see just above), enable Allow custom functions.
  5. Click OK to confirm the import.
Importing from a fileImporting from a file

Importing from a file

Import from URL

To import a Pack from a known, public or internal, URL:

  1. From the + Add New submenu, select Import from URL.
  2. Enter a valid URL for the Pack's source. (This field's input is validated for URL format, but not for accuracy, before you submit the modal.)
  3. Optionally, give the pack an explicit, unique New Pack ID. (See Upgrading an Existing Pack.)
  4. Where appropriate, enable Allow custom functions. (See Custom Functions.)
  5. Click OK to confirm the import.
Confirming file import from URLConfirming file import from URL

Confirming file import from URL

ūüĎć

To import a Pack from a public URL, LogStream's Leader Node (or single instance) requires Internet access. A distributed deployment's Leader can then deploy the Pack to Workers even if the Workers lack Internet access.

Import from Git Repos

To import a Pack from a known public or private Git repo:

  1. From the + Add New submenu, select Import from Git.

  2. Enter the source repo's valid URL.

    This field's input is validated for URL format, but not for completeness or accuracy, before you submit the modal. When¬†targeting a private repo, use the format: https://<username>:<token/password>:<repo‚ÄĎaddress>. Public repos need only https://<repo‚ÄĎaddress>, as shown in the example below.

  3. Optionally, give the pack an explicit, unique New Pack ID. (See Upgrading an Existing Pack.)

  4. Optionally, enter a Branch or tag to filter the import source using the repo's metadata. You can specify a branch (such as master) or a tag (such as a release number: 0.5.1, etc.).

  5. Where appropriate (see Custom Functions), enable Allow custom functions.

  6. Click OK to confirm the import.

Importing from a Git repoImporting from a Git repo

Importing from a Git repo

ūüĎć

To import a Pack from a public repo, LogStream's Leader Node (or single instance) requires Internet access. A distributed deployment's Leader can then deploy the Pack to Workers even if the Workers lack Internet access.

The Cribl Pack Dispensary‚ĄĘ

You might be wondering, "This Import from Git option is nice, but how do I discover a reliable repo from which to pull Packs that add useful features to LogStream?"

For starters, Cribl is proud to point you to the Cribl Pack Dispensary‚ĄĘ. Here, Cribl's own engineers have seeded several strains of high-productivity LogStream configurations. Because this repo is a place to share good stuff, we expect many new hybrids to sprout from the community. Cribl will test and curate submissions to ensure the quality of the repo's contents.

You can install Dispensary Packs directly through LogStream's UI (see Import from Git Repos above). However, if you prefer, you can click through to any Dispensary repo's release page, download the corresponding .crbl file, and then upload the file into LogStream.

Downloading a `.crbl` file from the Cribl Pack Dispensary's Web UIDownloading a `.crbl` file from the Cribl Pack Dispensary's Web UI

Downloading a .crbl file from the Cribl Pack Dispensary's Web UI

Upgrading an Existing Pack

Each Pack that is installed within a given Worker Group (or single-instance deployment) must have a unique ID. The ID is based on the Pack's internal configuration ‚Äď not its container's file name, nor on its Display¬†name.

If you import a Pack whose internal ID matches an installed Pack ‚Äď whether an update, or just a duplicate ‚Äď you'll be prompted to assign a unique New¬†Pack ID to import it as a separate Pack.

Renaming a Pack on importRenaming a Pack on import

Renaming a Pack on import

You'll also have the option to Overwrite the installed Pack, reusing the same ID.

‚ĚóÔłŹ

If you toggle this option to Yes, the imported Pack will completely overwrite your existing Pack's configuration.

Each Pack within a LogStream instance must have a unique Pack ID, so you cannot share an ID between two (or more) installed Packs.

To explicitly upgrade an existing Pack, you can instead click the Upgrade button on its row.

Upgrading an existing PackUpgrading an existing Pack

Upgrading an existing Pack

ūüĎć

If you've modified an installed Pack, LogStream will block overwriting the Pack, to prevent deletion of your locally created resources.

Creating a Pack

You can create a new Pack from scratch, to consolidate and export multiple LogStream configuration objects:

  1. Navigate to the Manage Packs page.
  2. Click + Add New.
  3. From the submenu, select Create Pack.
  4. In the resulting New Pack modal, fill in a unique Pack ID and other details.
    Each Pack within a LogStream instance must have a separate Pack ID, but you can assign arbitrary Display names.
  5. Click OK to save the Pack.
Creating a PackCreating a Pack

Creating a Pack

  1. On the Manage Packs page, click the new Pack's row to open the Pack.
Manage Packs pageManage Packs page

Manage Packs page

  1. Use the standard LogStream controls (see above) to configure and save the infrastructure you want to pack up. As you save changes in the UI, they're saved to the Pack.

Modifying Pack Settings

You can update a Pack's metadata (Version, Description, Author, etc.) and display settings. If you're developing a new Pack to share, you'll want to use this interface to populate the Pack's README and display logo.

  1. From the Pack's submenu, select Settings.
Pack SettingsPack Settings

Pack Settings

  1. To populate the Pack's README file, toggle View to Edit, replace the placeholder markdown content, and Save.
Editing Pack's READMEEditing Pack's README

Editing Pack's README

  1. To update other metadata, click the left Settings tab.
Editing Pack's metadataEditing Pack's metadata

Editing Pack's metadata

  1. To add a Pack logo, click the Pack's Settings > Display left tab.

    Cribl recommends adding a logo to each custom Pack, to visually distinguish the Pack's UI from the surrounding LogStream UI (as well as from other Packs). You can upload a .png or .jpg/.jpeg file, up to a maximum size of 2MB and 350x350px. Cribl recommends a transparent image, sized approximately 280x50px.

Editing Pack's display (logo) settingsEditing Pack's display (logo) settings

Editing Pack's display (logo) settings

Exporting a Pack

To export a newly created or modified Pack, click its Export button on the Packs page.

Exporting a PackExporting a Pack

Exporting a Pack

The resulting Export Pack modal provides the following options.

Export Mode

Select one of these three buttons:

  • Merge safe: Attempt to safely merge local modifications into the Pack's default layer (original configuration), then export.

  • Merge: Force-merge local modifications into the Pack's original configuration, then export.

  • Default only: Export only the Pack's original configuration, without local modifications.

The Merge safe option is conservative, and will block the export where conflicting modified contents can't be readily merged with the Pack's original contents:

**Merge safe** error**Merge safe** error

Merge safe error

If you encounter this error, use the Merge or Default only export mode instead.

Export Target

The options here are:

  • File (the default): You'll be prompted to confirm a file name and destination after you click OK.

  • Group: Selecting this displays a Group drop-down, prompting you to select an existing Worker¬†Group to export the Pack to. (The current Worker¬†Group is automatically omitted from the options.)

Updated 17 days ago

Packs


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.