Cribl LogStream ā€“ Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF ā€“ v.3.1.1

Pipelines

What Are Pipelines

Data matched by a given Route is delivered to a Pipeline. Pipelines are the heart of LogStream processing. Each Pipeline is a list of Functions that work on the data.

šŸ“˜

As with Routes, the order in which the Functions are listed matters. AĀ Pipeline's Functions are evaluated in order, topā€‘>down.

Accessing Pipelines

Select Pipelines from LogStream's global top nav (single-instance deployments) or from a WorkerĀ Group's top nav (distributed deployments). Next, click any displayed Pipeline to see or reconfigure its contained Functions.

Adding Pipelines

To create a new Pipeline, or to import an existing Pipeline to a different LogStream instance, click +Ā Pipeline at the upper right. The resulting menu offer three options:

  • CreateĀ Pipeline: Configure a new Pipeline from scratch, by adding Functions in LogStream's graphical UI.
  • Import from File: Import an existing Pipeline from a .json file on your local filesystem.
  • Import from URL: Import an existing Pipeline from .json file at a remote URL. (This must be a public URL ending in .json ā€“ the import option doesn't pass credentials to private URLs ā€“ and the target file must be formatted as a valid Pipeline configuration.)
Creating or importing a PipelineCreating or importing a Pipeline

Creating or importing a Pipeline

šŸ‘

To export a Pipeline, see Advanced Mode (JSONĀ Editor).

To import or export a Pipeline along with broader infrastructure (like KnowledgeĀ Objects and/or sample data files), see Packs.

How Do Pipelines Work

Events are always delivered to the beginning of a Pipeline via a Route. The data in the Stats column shown below are for the last 15 minutes.

Pipelines and Route inputsPipelines and Route inputs

Pipelines and Route inputs

šŸ“˜

You can pressĀ the ] (right-bracket) shortcut key to toggle between the Preview pane and an expanded Pipelines display. (This shortcut works when no field has focus.)

In the condensed Pipelines display above, you can also hover over any Pipeline's Functions column to see a horizontal preview of the stack of Functions contained in the Pipeline:

Preview on hovering over the bottom Pipeline (highlighted in gray)Preview on hovering over the bottom Pipeline (highlighted in gray)

Preview on hovering over the bottom Pipeline (highlighted in gray)

Within the Pipeline, events are processed by each Function, in order. A Pipeline will always move events in the direction that points outside of the system. This is on purpose, to keep the design simple and avoid potential loops.

Pipeline FunctionsPipeline Functions

Pipeline Functions

šŸ“˜

You can streamline the above display by organizing related Functions into FunctionĀ groups.

Pipeline Settings

Click the gear button at top right to open the Pipeline's Settings. Here, you can attach the Pipeline to a Route. InĀ the Settings' Async function timeout (ms) field, you can enter a buffer to adjust for Functions that might take much longer to execute than normal. (An example would be a Lookup Function processing a large lookup file.)

Pipeline SettingsPipeline Settings

Pipeline Settings

Advanced Mode (JSONĀ Editor)

Once you've clicked the gear button to enter PipelineĀ Settings, you can click EditĀ asĀ JSON at upper right to edit the Pipeline's definition in a JSON text editor. In this mode's editor, you can directly edit multiple values. You can also use the Import and Export buttons here to copy and modify existing Pipeline configurations, as .json files.

Advanced Pipeline EditingAdvanced Pipeline Editing

Advanced Pipeline Editing

Click EditĀ in GUI at upper right to return to the graphical PipelineĀ Settings page; then click BackĀ to to restore the graphical Pipeline editor.

Pipeline Actions

Click a Pipeline's Actions (...) menu to display options for copying or deleting the Pipeline.

Pipeline > Actions menuPipeline > Actions menu

Pipeline > Actions menu

Copying a Pipeline displays the confirmation message and the (highlighted) Paste button shown below.

Paste button for copied PipelinePaste button for copied Pipeline

Paste button for copied Pipeline

Pasting prompts you to confirm, or change, a modified name for the new Pipeline. The result will be an exact duplicate of the original Pipeline in all but name.

Saving/renaming a pasted PipelineSaving/renaming a pasted Pipeline

Saving/renaming a pasted Pipeline

Types of Pipelines

You can apply various Pipeline types at different stages of data flow. All Pipelines have the same basic internal structure (a series of Functions) ā€“ the types below differ only in their position in the system.

Pre-processing, processing, and post-processing PipelinesPre-processing, processing, and post-processing Pipelines

Pre-processing, processing, and post-processing Pipelines

Pre-Processing Pipelines

These are Pipelines that are attached to a Source to condition (normalize) the events before they're delivered to a processing Pipeline. They're optional.

Typical use cases are event formatting, or applying Functions to all events of an input. (E.g., to extract a message field before pushing events to various processing Pipelines.)

You configure these Pipelines just like any other Pipeline, by selecting Pipelines from the top menu. You then attach your configured Pipeline to individual Sources, using the Source's Preā€‘Processing > Pipeline drop-down.

Fields extracted using pre-processing Pipelines are made available to Routes.

Processing Pipelines

These are "normal" event processing Pipelines, attached directly to Routes.

Post-Processing Pipelines

These Pipelines are attached to a Destination to normalize the events before they're sent out. AĀ post-processing Pipeline's Functions apply to all events exiting to the attached Destination.

Typical use cases are applying Functions that transform or shape events per receiver requirements. (E.g., to ensure that a _time field exists for all events bound to a Splunk receiver.)

You configure these Pipelines as normal, by selecting Pipelines from the top menu. You then attach your configured Pipeline to individual Destinations, using the Destination's Postā€‘Processing > Pipeline drop-down.

You can also use a Destination's Postā€‘Processing options to add SystemĀ Fields like cribl_input, identifying the LogStream Source that processed the events.

Best Practices for Pipelines

Functions in a Pipeline are equipped with their own filters. Even though filters are not required, we recommend using them as often as possible.

As with Routes, the general goal is to minimize extra work that a Function will do. The fewer events a Function has to operate on, the better the overall performance.

For example, if a Pipeline has two Functions, f1 and f2, and if f1 operates on source 'foo' and f2 operates on source 'bar', it might make sense to apply source=='foo' versus source=='bar' filters on these two Functions, respectively.

Updated 19 days ago


What's Next

Functions
Packs

Pipelines


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.