When events enter a pipeline they're processed by a series of functions. At its core, a function is code that executes on an event and it encapsulates the smallest amount of processing that can happen to that event. The term "processing" means a variety of possible options; from string replacement, to obfuscation, encryption, event to metrics conversions etc. For example, a pipeline can be composed of several functions, one that replaces the term
bar, another one that hashes
bar and a last one that adds a field, say,
dc=jfk-42 to any event that matches
Functions are atomic pieces of JS code that are invoked on each event that passes thru them. To help improve performance, functions can be configured with filters to further scope their invocation on matching events only. You can add as many functions in a pipeline as necessary, though the more you have the longer it will take each event to pass thru. In addition, you can turn functions On/Off inline as necessary.
Similar to the
Final toggle in routes, the
Final toggle here controls the flow of events at the function level.
Off (default): means that matching events processed by this function will be passed down to the next function.
On: means that this function is the last one that the matching events will be applied to. All others coming down the pipeline will be skipped.
Cribl LogStream ships with several functions out of the box and you can chain them together to meet your requirements. Expand the list of Functions on the left and the Use Cases section for more details.
At the time of this custom functions are not yet supported.
Find & Replace, including basic
sed-like, obfuscate, redact, hash etc.
Add GeoIP information to events
Suppress events (e.g, duplicates etc.)
Serialize / change format (e.g., convert JSON to CSV)
Flatten nested structures (e.g., nested JSON)
Aggregate events in real-time (i.e. statistical aggregations)
Resolve hostname from IP address
Reverse DNS (beta)
Updated about a month ago