How Do They Work

​

Functions are atomic pieces of JavaScript code that are invoked on each event that passes through them. To help improve performance, Functions can be configured with filters to further scope their invocation to matching events only.

You can add as many Functions in a Pipeline as necessary, though the more you have, the longer it will take each event to pass through. Also, you can enable and disable Functions within a Pipeline as necessary. This lets you preserve structure as you optimize or debug.

You can reposition Functions up or down the Pipeline stack to adjust their execution order. Use a Function’s left grab handle to drag and drop it into place.

The Final Toggle

​

The Final toggle in Function settings controls what happens to the results of a Function.

When Final is toggled to No (default), events will be processed by this Function, and then passed on to the next Function below.

When Final is toggled to Yes, the Function “consumes” the results, meaning they will not pass down to any other Function below.

A flag in Pipeline view indicates that a Function is Final.

Functions and Shared-Nothing Architecture

​

Cribl Stream is built on a shared-nothing architecture, where each Node and its Worker Processes operate separately, and process events independently of each other. This means that all Functions operate strictly in a Worker Process context – state is not shared across processes.

For example, consider two events that meet a Pipeline’s criteria to be aggregated. If these two events arrive on separate Workers or Worker Processes, Cribl Stream will not aggregate them together.

This is particularly important to understand for certain Functions that might imply state-sharing, such as Aggregations, Rollup Metrics, Dynamic Sampling, Sampling, Suppress, etc.

If you have a large number of Worker Processes, consider implementing a distributed caching tier, such as Redis, to aggregate events across Workers. (See our Redis Function topic.)

Out-of-the-Box Functions

​

Cribl Stream ships with several Functions out-of-the-box, and you can chain them together to meet your requirements. For more details, see individual Functions, and the Use Cases section, within this documentation.

Accessing Event Fields with __e

​

The special variable __e represents the (context) event inside a JavaScript expression. Using __e with square bracket notation, you can access any field within the event object, for example, __e['hostname']. Functions use __e extensively. You also must use this notation for fields that contain a special character, like -, ., or @.

Supported JavaScript Standard

​

Cribl Stream supports the ECMAScript® 2015 Language Specification.

Very Large Integer Values

​

Cribl Stream’s JavaScript implementation can safely represent integers only up to the Number.MAX_SAFE_INTEGER constant of about 9 quadrillion (precisely, {2^53}‑1). Cribl Stream Functions will round down any integer larger than this, in Data Preview and other contexts. Trailing 0’s might indicate such rounding down of large integers.

What Functions to Use When

​

• Add, remove, update fields: Eval, Lookup, Regex Extract

• Find & Replace, including basic sed-like, obfuscate, redact, hash, etc.: Mask, Eval

• Add GeoIP information to events: GeoIP

• Extract fields: Regex Extract, Parser

• Extract timestamps: Auto Timestamp

• Drop events: Drop, Regex Filter, Sampling, Suppress, Dynamic Sampling

• Sample events (e.g, high-volume, low-value data): Sampling, Dynamic Sampling

• Suppress events (e.g, duplicates, etc.): Suppress

• Serialize events to CEF format (send to various SIEMs): CEF Serializer

• Serialize / change format (e.g., convert JSON to CSV): Serialize

• Convert JSON arrays into their own events: JSON Unroll, XML Unroll

• Flatten nested structures (e.g., nested JSON): Flatten

• Aggregate events in real-time (i.e., statistical aggregations): Aggregations

• Convert events to metrics format: Publish Metrics, Prometheus Publisher (beta)

• Transforms dimensional metrics events into the OTLP (OpenTelemetry Protocol) format: OTLP Metrics

• Resolve hostname from IP address: Reverse DNS (beta)

• Extract numeric values from event fields, converting them to type number: Numerify

• Send events out to a command or a local file, via stdin, from any point in a Pipeline: Tee

• Convert an XML event’s elements into individual events: XML Unroll

• Duplicate events in the same Pipeline, with optional added fields: Clone

• Break events within, instead of before they reach, a Pipeline: Event Breaker

• Add a text comment within a Pipeline’s UI, to label steps without changing event data: Comment

For more usage examples, download Cribl’s Stream Cheat Sheet (Quick Reference Card).

Function Groups

​

A Function group is a collection of consecutive Functions that can be moved up and down a Pipeline’s Functions stack together. Groups help you manage long stacks of Functions by streamlining their display. They are a UI visualization only: While Functions are in a group, those Functions maintain their global position order in the Pipeline.

Function groups work much like Route groups.

To build a group from any Function, click the Function’s ••• (Options) menu, then select Group Actions > Create Group.

You’ll need to enter a Group Name before you can save or resave the Pipeline. Optionally, enter a Description.

Once you’ve saved at least one group to a Pipeline, other Functions’ ••• (Options) > Group Actions submenus will add options to Move to Group or Ungroup/Ungroup All.

You can also use a Function’s left grab handle to drag and drop it into, or out of, a group. A saved group that’s empty displays a dashed target into which you can drag and drop Functions.