Cribl LogStream โ€“ Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF โ€“ v.3.1.1

Basic Concepts

Notable features and concepts to get a fundamental understanding of Cribl LogStream

As we describe features and concepts, it helps to have a mental model of Cribl LogStream as a system that receives events from various sources, processes them, and then sends them to one or more destinations.

Sources, LogStream, destinationsSources, LogStream, destinations

Sources, LogStream, destinations

Let's zoom in on the center of the above diagram, to take a closer look at the processing and transformation options that LogStream provides internally. The basic interface concepts to work with are Routes, which manage data flowing through Pipelines, which consist of Functions.

Routes, Pipelines, FunctionsRoutes, Pipelines, Functions

Routes, Pipelines, Functions

Routes

Routes evaluate incoming events against filter expressions to find the appropriate Pipeline to send them to. Routes are evaluated in order. Each Route can be associated with only one Pipeline and one output (configured as a LogStream Destination).

By default, each Route is created with its Final flag set to Yes. With this setting, a Route-Pipeline-Destination set will consume events that match its filter, and that's that.

However, if you disable the Route's Final flag, one or more event clones will be sent down the associated Pipeline, while the original event continues down LogStream's Routing table to be evaluated against other configured Routes. This is very useful in cases where the same set of events needs to be processed in multiple ways, and delivered to different destinations. For more details, see Routes.

Pipelines

A series of Functions is called a Pipeline, and the order in which you specify the Functions determines their execution order. Events are delivered to the beginning of a Pipeline by a Route, and as they're processed by a Function, the events are passed to the next Function down the line.

Pipelines attached to Routes are called processing Pipelines. You can optionally attach pre-processing Pipelines to individual LogStream Sources, and attach post-processing Pipelines to LogStream Destinations. All Pipelines are configured through the same UI โ€“ these three designations are determined only by a Pipeline's placement in LogStream's data flow.

Pipelines categorized by positionPipelines categorized by position

Pipelines categorized by position

Events only move forward โ€“ toward the end of a Pipeline, and eventually out of the system. Forย more details, see Pipelines.

Functions

At its core, a Function is a piece of code that executes on an event, and that encapsulates the smallest amount of processing that can happen to that event. For instance, a very simple Function can be one that replaces the term foo with bar on each event. Another one can hash or encrypt bar. Yet another function can add a field โ€“ say, dc=jfk-42 โ€“ to any event with source=*us-nyc-application.log.

Functions stacked in a PipelineFunctions stacked in a Pipeline

Functions stacked in a Pipeline

Functions process each event that passes through them. To help improve performance, Functions can optionally be configured with filters, to limit their processing scope to matching events only. For more details, see Functions.

A Scalable Model

You can scale LogStream up to meet enterprise needs in a distributed deployment. Here, multiple LogStream Workers (instances) share the processing load. But as you can see in the preview schematic below, even complex deployments follow the same basic model outlined above.

Distributed deployment architectureDistributed deployment architecture

Distributed deployment architecture

Updated 4 months ago


Basic Concepts


Notable features and concepts to get a fundamental understanding of Cribl LogStream

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.