All data processing in Cribl LogStream is based on discrete data entities commonly known as events. An event is generally defined as a collection of key/value pairs (fields). Some Sources deliver events directly while others may deliver bytestreams that need to be broken up by Event Breakers. Events travel from a source through pipelines's functions and to destinations.
The internal representation of a Cribl LogStream event (loosely based on Splunk's) is as follows:
{
"_raw": "<body of non-JSON parse-able event>",
"_time": "<timestamp in UNIX epoch format>",
"__inputId": "<Id/Name of Source that delivered the event>",
"__other1": "<Internal field1>",
"__other2": "<Internal field2>",
"__otherN": "<Internal fieldN>",
"key1": "<value1>",
"key2": "<value2>",
"keyN": "<valueN>",
"...": "..."
}- Fields that start with a double-underscore are known as internal fields and each Source can add one or many to each event. For example, Syslog adds both a
__inputIdand a__srcIpPortfield. Internal fields are used only within Cribl LogStream and are not passed down to Destinations. - Upon arrival from a Source if an event cannot be JSON parsed, all of its content will be assigned to
_raw. - If a timestamp is not configured to be extracted, current time in UNIX epoch format wiill be assigned to
_time.
Using Capture
One way to see what an event looks like as it travels the system is to use the Capture feature. While in Preview (right pane) click on Start Capture. In the following popover enter a Filter Expression to narrrow down the events of interest, click Capture ... then select the desired Where to capture option. There four options:
- As They Come In - capture events right after they're delivered by the respecitive Input.
- After Input Pipeline - capture events right after the Input Conditioning Pipeline, before they go down the Routes.
- After Processing Pipeline - capture events right after the Processing Pipeline that actually handled them.
- After Output Pipeline - capture events right after the Output Conditioning Pipeline, before they go out to the configured Destination.
Updated about a month ago