Description

The Parser Function can be used to extract fields out of events, or to reserialize (rewrite) events with a subset of fields. Reserialization will maintain the format of the events.

For example: If an event contains comma-delimited fields, and fieldA and fieldB are filtered out, those fields' positions will be set to null, but not deleted completely.

Parser cannot remove fields that it did not create. A subsequent Eval Function can do so.

Usage

Filter: Filter expression (JS) that selects data to be fed through the Function. Defaults to true, meaning that all events will be evaluated.

Description: Simple description about this Function. Defaults to empty.

Final: If true, stops data from being fed to the downstream Functions. Defaults to No.

Operation mode: Extract will create new fields. Reserialize will extract, filter fields, and then reserialize.

Type: Parser/Formatter type to use. Options:

• CSV
• Extended Log File Format (ELFF)
• Common Log Format (CLF)
• K=V Pairs
• JSON
• Delimited Values

Setting Type to Delimited Values displays the following extra options:

• Delimiter: Delimiter character to split value. Defaults to comma (,). You can also specify pipe (|) or tab characters.
• Quote char: Character used to quote literal values. Defaults to ".
• *Escape char: Character used to escape delimiter or quote characters. Defaults to \.
• Null value: Field value representing the null value. These fields will be omitted. Defaults to -.

Library: Select an option from the Parsers Library.

Source field: Field that contains text to be parsed. Not usually needed in Serialize mode.

Destination field: Name of field in which to add extracted and serialized fields. (Extract and Serialize modes only.)

Clean fields: This option appears for Type: K=V Pairs. Toggle to Yes to clean field names by replacing non-alphanumeric characters with _. This will also strip leading and trailing " symbols.

List of fields: Fields expected to be extracted, in order. If not specified, Parser will auto-generate fields.

Fields to keep: List of fields to keep. Supports wildcards (*). Takes precedence over Fields to remove. Nested addressing supported.

Fields to remove: List of fields to remove. Supports wildcards (*). Cannot remove fields matching Fields to keep. Nested addressing supported.

📘

Negated terms are supported in both Fields to remove and Fields to keep. When you use negated terms, the list is order-sensitive. E.g., !foobar, foo* means "All fields that start with foo, except foobar." However, !foo*, * means "All fields, except for those that start with foo."

Fields filter expression: Expression to evaluate against {index, name, value} context of each field. Return truthy to keep, falsy to remove field. Index is zero-based.

How Fields Settings Interact

The Fields to keep, Fields to remove, and Fields filter expression settings interact as follows:

• Order of evaluation: Fields to keep > Fields to remove > Fields filter expression.

• If a field is in both Fields to keep and Fields to remove, Fields to keep takes precedence.

• If a field is in both Fields to remove and Fields filter expression, Fields to remove takes precedence.

Example 1

Insert the following sample, using Preview > Add a Sample > Paste a Sample:
2019/06/24 05:10:55 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008

Create the following test Parser Function (or import this Pipeline: https://raw.githubusercontent.com/weeb-cribl/cribl-samples/master/parser/functions/parser/parser_1.json).

First, set the Parser type to Key=Value Pairs.

Scenario A:

Keep fields a, b, c. Drop the rest.

Expected result: a, b, c

• Fields to Keep: a, b, c
• Fields to Remove: *
• Fields Filter Expression:

Result: The event will gain four new fields and values, as follows.

• a: 000
• b: 001
• c: 002
• cribl_pipe: parser2

You can check your stats by clicking the Preview pane’s Basic Statistics (chart) button. In the resulting pop-up, the Number of Fields should have incremented ty four.

Now that you have the hang of it, try out the other simple scenarios below.

Scenario B:

Keep fields a, b, those that start with g. Drop the rest.

Expected result: a, b, g1, g2, g3

• Fields to keep: a, b
• Fields to remove: [empty]
• Fields filter expression: name.startsWith('g')

Scenario C:

Keep fields a, b, those that start with g but only if value is 007. Drop the rest.

Expected result: a, b, g2

• Fields to keep: a, b
• Fields to remove: [empty]
• Fields filter expression: name.startsWith('g') && value=='007'

Scenario D:

Keep fields a, b, c, those that start with g, unless it's g1. Drop the rest.

Expected result: a, b, c, g2, g3

• Fields to keep: a, b, c
• Fields to remove: g1
• Fields filter expression: name.startsWith('g')

Scenario E:

Keep fields a, b, c, those that start with g but only if index is greater than 6. Drop the rest.

Expected result: a, b, c, g2, g3

• Fields to keep: a, b, c
• Fields to remove: [empty]
• Fields filter expression: name.startsWith('g') && index>6

📘

The index refers to the location of a field in the array of all fields extracted by this Parser. It is zero-based. In the case above, g2 and g3 have index values of 7 and 8, respectively.

Example 2

Assume we have a JSON event that needs to be reserialized, given these requirements:

1. Remove the level field only if it's set to info.
2. Remove the startTime field, and all fields in the values.total. path that end in Cxn.

Parser Function configuration:

JSON event after being processed by the Function:

Example 3

Insert the following sample, using Preview > Add a Sample > Paste a Sample:

2019/06/24 15:25:36 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008,

For all scenarios below, first create a Parser Function to extract all fields, by setting the Parser type to Key=Value Pairs. Then add a second Parser Function with the configuration shown under Parser 2.

Scenario A:

Serialize fields a, b, c, d in CSV format.

Expected result: _raw field will have this value 000,001,002,003

Parser 2:

• Operation mode: Serialize
• Source field: [empty]
• Destination field: [empty]
• Type: CSV
• List of fields: a, b, c, d (needed for positional formats)

Scenario B:

Serialize fields a, b, c in JSON format, under a field called bar.

Expected result: bar field will be set to: {"a":"000","b":"001","c":"002","d":"003"}

Parser 2:

• Operation mode: Serialize
• Source field: [empty]
• Destination field: bar
• Type: JSON
• List of fields: [empty]
• Fields to keep: a, b, c, d