What are Pipelines
After the data has been matched by a route it gets delivered to a pipeline. A pipeline is a list of functions that work on the data. Similar to routes, the order in which the functions are listed matters.
Functions in a pipeline are evaluated in order, top down.
How do Pipelines Work
Events are always delivered at the beginning of a pipeline via a route and they are processed by each function, in order. A pipeline will always move events in the direction that points outside of the system. This is on purpose so as to keep the design simple and avoid potential loops.
Input Pipelines
These are pipelines that are attached to a Source for the purposes of conditioning the events before they're delivered to a Processing Pipeline. They are optional and typical use cases are event formatting or when applying functions to all events of that input . E.g., extract a message field before pushing events to various processing pipelines.
Processing Pipelines
These are "normal" event processing pipelines.
Output Pipelines
These pipelines are attached to a Destination for the purposes of conditioning the events before they're sent out. Typical use cases are applying functions that transform or shape events per receiver requirements. E.g., ensure that a _time field exists for all events bound to a Splunk receiver.
Pre-conditioning Pipeline
Use with caution. When selected, this will use the selected pipeline as a preprocessor. The pipeline will be then processed PRIOR to any ingestion through subsequent routes.
Considerations
Functions in a pipeline are equipped with their own filters. Even though they're not required, it is advised that they're used as often as possible. Similar to routes, the general goal is to minimize extra work that a function will do; the fewer events a function has to operate on the better the overall performance. For example, if a pipeline has two functions, f1--f2 and if f1 operates on source 'foo' and f2 that operates on source 'bar' it may make sense to apply source=='foo' and source=='bar' filters on each one respectively.
Updated 11 days ago