What are Pipelines

After the data has been matched by a route it gets delivered to a pipeline. A pipeline is a list of functions that work on the data. Similar to routes, the order in which the functions are listed matters.

How do Pipelines Work

Events are always delivered at the beginning of a pipeline via a route and they are processed by each function, in order. A pipeline will always move events in the direction that points outside of the system. This is on purpose so as to keep the design simple and avoid potential loops.

Types of Pipelines

---

Input Pipelines

These are pipelines that are attached to a Source for the purposes of conditioning the events before they're delivered to a Processing Pipeline. They are optional and typical use cases are event formatting or when applying functions to all events of that input . E.g., extract a message field before pushing events to various processing pipelines.

Processing Pipelines

These are "normal" event processing pipelines.

Output Pipelines

These pipelines are attached to a Destination for the purposes of conditioning the events before they're sent out. Typical use cases are applying functions that transform or shape events per receiver requirements. E.g., ensure that a _time field exists for all events bound to a Splunk receiver.

Considerations

Functions in a pipeline are equipped with their own filters. Even though they're not required, it is advised that they're used as often as possible. Similar to routes, the general goal is to minimize extra work that a function will do; the fewer events a function has to operate on the better the overall performance. For example, if a pipeline has two functions, f1--f2 and if f1 operates on source 'foo' and f2 that operates on source 'bar' it may make sense to apply source=='foo' and source=='bar' filters on each one respectively.