Functions

When events enter a Pipeline, they’re processed by a series of Functions. At its core, a Function is code that executes on an event, and it encapsulates the smallest amount of processing that can happen to that event.

The term “processing” means a variety of possible options: string replacement, obfuscation, encryption, event-to-metrics conversions, etc. For example, a Pipeline can be composed of several Functions – one that replaces the term foo with bar, another one that hashes bar, and a final one that adds a field (say, dc=jfk-42) to any event that matches source=='us-nyc-application.log'.

How Do They Work

Functions are atomic pieces of JavaScript code that are invoked on each event that passes through them. To help improve performance, Functions can be configured with filters to further scope their invocation to matching events only.

You can add as many Functions in a Pipeline as necessary, though the more you have, the longer it will take each event to pass through. Also, you can enable and disable Functions within a Pipeline as necessary. This lets you preserve structure as you optimize or debug.

Functions stack in a Pipeline
Functions stack in a Pipeline

You can reposition Functions up or down the Pipeline stack to adjust their execution order. Use a Function’s left grab handle to drag and drop it into place.

The Final Toggle

The Final toggle in Function settings controls what happens to the results of a Function.

When Final is toggled to No (default), events will be processed by this Function, and then passed on to the next Function below.

When Final is toggled to Yes, the Function “consumes” the results, meaning they will not pass down to any other Function below.

A flag in Pipeline view indicates that a Function is Final.

Functions in a Pipeline, some of them have Final flags.
The Eval and first Drop Functions bearing the Final flag

Functions and Shared-Nothing Architecture

Cribl Stream is built on a shared-nothing architecture, where each Node and its Worker Processes operate separately, and process events independently of each other. This means that all Functions operate strictly in a Worker Process context – state is not shared across processes.

For example, consider two events that meet a Pipeline’s criteria to be aggregated. If these two events arrive on separate Workers or Worker Processes, Cribl Stream will not aggregate them together.

This is particularly important to understand for certain Functions that might imply state-sharing, such as Aggregations, Rollup Metrics, Dynamic Sampling, Sampling, Suppress, etc.

If you have a large number of Worker Processes, consider implementing a distributed caching tier, such as Redis, to aggregate events across Workers. (See our Redis Function topic.)

Out-of-the-Box Functions

Cribl Stream ships with several Functions out-of-the-box, and you can chain them together to meet your requirements. For more details, see individual Functions, and the Use Cases section, within this documentation.

Accessing Event Fields with __e

The special variable __e represents the (context) event inside a JavaScript expression. Using __e with square bracket notation, you can access any field within the event object, for example, __e['hostname']. Functions use __e extensively. You also must use this notation for fields that contain a special character, like -, ., or @.

Supported JavaScript Standard

Cribl Stream supports the ECMAScript® 2015 Language Specification.

Very Large Integer Values

Cribl Stream’s JavaScript implementation can safely represent integers only up to the Number.MAX_SAFE_INTEGER constant of about 9 quadrillion (precisely, {2^53}‑1). Cribl Stream Functions will round down any integer larger than this, in Data Preview and other contexts. Trailing 0’s might indicate such rounding down of large integers.

What Functions to Use When

For more usage examples, download Cribl’s Stream Cheat Sheet (Quick Reference Card).

Function Groups

A Function group is a collection of consecutive Functions that can be moved up and down a Pipeline’s Functions stack together. Groups help you manage long stacks of Functions by streamlining their display. They are a UI visualization only: While Functions are in a group, those Functions maintain their global position order in the Pipeline.

Function groups work much like Route groups.

To build a group from any Function, click the Function’s ••• (Options) menu, then select Group Actions > Create Group.

Creating a group
Creating a group

You’ll need to enter a Group Name before you can save or resave the Pipeline. Optionally, enter a Description.

Once you’ve saved at least one group to a Pipeline, other Functions’ ••• (Options) > Group Actions submenus will add options to Move to Group or Ungroup/Ungroup All.

You can also use a Function’s left grab handle to drag and drop it into, or out of, a group. A saved group that’s empty displays a dashed target into which you can drag and drop Functions.

Drag-and-drop target
Drag-and-drop target