Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up)
Download entire manual as PDF - v2.3.3

Parser

Description

The Parser Function can be used to extract fields out of events, or to reserialize (rewrite) events with a subset of fields. Reserialization will maintain the format of the events.

For example: If an event contains comma-delimited fields, and fieldA and fieldB are filtered out, those fields' positions will be set to null, but not deleted completely.

Parser cannot remove fields that it did not create. A subsequent Eval Function can do so.

Usage

Filter: Filter expression (JS) that selects data to be fed through the Function. Defaults to true, meaning that all events will be evaluated.

Description: Simple description about this Function. Defaults to empty.

Final: If true, stops data from being fed to the downstream Functions. Defaults to No.

Operation mode: Extract will create new fields. Reserialize will extract, filter fields, and then reserialize.

Type: Parser/Formatter type to use. Options:

  • CSV
  • Extended Log File Format (ELFF)
  • Common Log Format (CLF)
  • K=V Pairs
  • JSON
  • Delimited Values

Setting Type to Delimited Values displays the following extra options:

  • Delimiter: Delimiter character to split value. Defaults to comma (,). You can also specify pipe (|) or tab characters.
  • Quote char: Character used to quote literal values. Defaults to ".
  • *Escape char: Character used to escape delimiter or quote characters. Defaults to \.
  • Null value: Field value representing the null value. These fields will be omitted. Defaults to -.

Library: Select an option from the Parsers Library.

Source field: Field that contains text to be parsed. Not usually needed in Serialize mode.

Destination field: Name of field in which to add extracted and serialized fields. (Extract and Serialize modes only.)

Clean fields: This option appears for Type: K=V Pairs. Toggle to Yes to clean field names by replacing non-alphanumeric characters with _. This will also strip leading and trailing " symbols.

List of fields: Fields expected to be extracted, in order. If not specified, Parser will auto-generate fields.

Fields to keep: List of fields to keep. Supports wildcards (*). Takes precedence over Fields to remove. Nested addressing supported.

Fields to remove: List of fields to remove. Supports wildcards (*). Cannot remove fields matching Fields to keep. Nested addressing supported.

📘

Negated terms are supported in both Fields to remove and Fields to keep. When you use negated terms, the list is order-sensitive. E.g., !foobar, foo* means "All fields that start with foo, except foobar." However, !foo*, * means "All fields, except for those that start with foo."

Fields filter expression: Expression to evaluate against {index, name, value} context of each field. Return truthy to keep, falsy to remove field. Index is zero-based.

How Fields Settings Interact

The Fields to keep, Fields to remove, and Fields filter expression settings interact as follows:

  • Order of evaluation: Fields to keep > Fields to remove > Fields filter expression.

  • If a field is in both Fields to keep and Fields to remove, Fields to keep takes precedence.

  • If a field is in both Fields to remove and Fields filter expression, Fields to remove takes precedence.

Example 1

Insert the following sample, using Preview > Add a Sample > Paste a Sample:
2019/06/24 05:10:55 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008

Create the following test Parser Function (or import this Pipeline: https://raw.githubusercontent.com/weeb-cribl/cribl-samples/master/parser/functions/parser/parser_1.json).

Parser Function initial configuration

First, set the Parser type to Key=Value Pairs.

Scenario A:

Keep fields a, b, c. Drop the rest.

Expected result: a, b, c

  • Fields to Keep: a, b, c
  • Fields to Remove: *
  • Fields Filter Expression:

Result: The event will gain four new fields and values, as follows.

  • a: 000
  • b: 001
  • c: 002
  • cribl_pipe: parser2

Scenario A result

You can check your stats by clicking the Preview pane’s Basic Statistics (chart) button. In the resulting pop-up, the Number of Fields should have incremented ty four.

Now that you have the hang of it, try out the other simple scenarios below.

Scenario B:

Keep fields a, b, those that start with g. Drop the rest.

Expected result: a, b, g1, g2, g3

  • Fields to keep: a, b
  • Fields to remove: [empty]
  • Fields filter expression: name.startsWith('g')

Scenario C:

Keep fields a, b, those that start with g but only if value is 007. Drop the rest.

Expected result: a, b, g2

  • Fields to keep: a, b
  • Fields to remove: [empty]
  • Fields filter expression: name.startsWith('g') && value=='007'

Scenario D:

Keep fields a, b, c, those that start with g, unless it's g1. Drop the rest.

Expected result: a, b, c, g2, g3

  • Fields to keep: a, b, c
  • Fields to remove: g1
  • Fields filter expression: name.startsWith('g')

Scenario E:

Keep fields a, b, c, those that start with g but only if index is greater than 6. Drop the rest.

Expected result: a, b, c, g2, g3

  • Fields to keep: a, b, c
  • Fields to remove: [empty]
  • Fields filter expression: name.startsWith('g') && index>6

📘

The index refers to the location of a field in the array of all fields extracted by this Parser. It is zero-based. In the case above, g2 and g3 have index values of 7 and 8, respectively.

Example 2

Assume we have a JSON event that needs to be reserialized, given these requirements:

  1. Remove the level field only if it's set to info.
  2. Remove the startTime field, and all fields in the values.total. path that end in Cxn.

Parser Function configuration:

Parser Function configuration for Example 2

JSON event after being processed by the Function:

Example 2 event transformation

Example 3

Insert the following sample, using Preview > Add a Sample > Paste a Sample:

2019/06/24 15:25:36 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008,

For all scenarios below, first create a Parser Function to extract all fields, by setting the Parser type to Key=Value Pairs. Then add a second Parser Function with the configuration shown under Parser 2.

Scenario A:

Serialize fields a, b, c, d in CSV format.

Expected result: _raw field will have this value 000,001,002,003

Parser 2:

  • Operation mode: Serialize
  • Source field: [empty]
  • Destination field: [empty]
  • Type: CSV
  • List of fields: a, b, c, d (needed for positional formats)

Scenario B:

Serialize fields a, b, c in JSON format, under a field called bar.

Expected result: bar field will be set to: {"a":"000","b":"001","c":"002","d":"003"}

Parser 2:

  • Operation mode: Serialize
  • Source field: [empty]
  • Destination field: bar
  • Type: JSON
  • List of fields: [empty]
  • Fields to keep: a, b, c, d

Updated a day ago

Parser


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.