Description

---

The Parser function can be used to extract fields out of events or reserialize (re-write) events with a subset of fields. Reserialization will maintain the format of the event. For example, if an event contains comma delimited fields and fieldA and fieldB are filtered out, their positions will be set to null and not deleted completely.

Usage

---

Filter: Filter expression (JS) that selects data to be fed through the function. Defaults to empty - all events will be evaluated.
Description: Simple description about this function. Defaults to empty.
Final: If true, stops data from being fed to the downstream functions. Defaults to No.

Parser Mode: Operating mode. Extract creates new fields. Reserialize will extract, filter fields and then reserialize. Serialize will put fields in a certain format. Defaults to Extract.
Source Field: Field which contains text to be parsed. Not usually needed in Serialize Mode.
Destination Field: Field name where to add extracted and serialized fields to. Extract and Serialize Mode only.
Type: Parser/Formatter type to use. Options: CSV, JSON, K=V Pairs, Extended Log File Format (ELFF), Common Log Format (CLF)
Library: Browse Parser/Formatter library.
List of Fields: Fields expected to be extracted, in order. If not specified parser will auto-generate.
Fields to Keep: List of fields to keep, supports wildcards (*). Takes precedence over Fields to Remove. Nested addressing supported.
Fields to Remove: List of fields to remove, supports wildcards (*). Cannot remove fields matching Fields to Keep. Nested addressing supported.

• Note: Negated terms are supported in both Fields to Remove and Fields to Keep. List is order sensitive when negated terms are used. E.g., !foobar, foo* means "All fields that start with 'foo' except foobar". !foo*, * means "All fields except for those that start with 'foo'".

Fields Filter Expression: Expression evaluated against {index, name, value} context of each field. Return truthy to keep, falsy to remove field. Index is zero based.
Destination Field: Field where to add extracted fields to (Extract mode only).

How do Fields to Keep, Fields to Remove and Fields Filter Expression interact

---

Order or priority: Fields to Keep > Fields to Remove > Fields Filter Expression

If a field is in Fields to Keep and Fields to Remove, Fields to Keep takes precedence.
If a field is in Fields to Remove and in Fields Filter Expression, Fields to Remove takes precedence.

Example 1

---

Assume we have an event with KV pairs as below:
<timestamp> a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008, ...
To extract all fields we can select K=V Pairs from Parser Type.

Scenario A: Keep fields a, b, c. Drop the rest.
Expected result: a, b, c

• Fields to Keep: a, b, c
• Fields to Remove: *
• Fields Filter Expression: <empty>

Scenario B: Keep fields a, b, those that start with g. Drop the rest.
Expected result: a, b, g1, g2, g3

• Fields to Keep: a, b
• Fields to Remove: <empty>
• Fields Filter Expression: name.startsWith('g')

Scenario C: Keep fields a, b, those that start with g but only if value is 007. Drop the rest.
Expected result: a, b, g2

• Fields to Keep: a, b
• Fields to Remove: <empty>
• Fields Filter Expression: name.startsWith('g') && value=='007'

Scenario D: Keep fields a, b, c, those that start with g, unless it's g1. Drop the rest.
Expected result: a, b, c, g2, g3

• Fields to Keep: a, b, c
• Fields to Remove: g1
• Fields Filter Expression: name.startsWith('g')

Scenario E: Keep fields a, b, c, those that start with g but only if index is greater than 6. Drop the rest.
Expected result: a, b, c, g2, g3

• Fields to Keep: a, b, c
• Fields to Remove: <empty>
• Fields Filter Expression: name.startsWith('g') && index>6

Note: index refers to the location of a field in the array of all fields extracted by this parser. It is zero-based. In the case above, g2 and g3 have an index of 7 and 8 respectively.

Example 2

---

Assume we have a JSON event that needs to be reserialized given these requirements:

1. Remove the level field only if it's set to info
2. Remove the startTime field and all those that end in Cxn in the values.total. path

Parser Function Configuration:

JSON event after processed by the function:

Example 3

---

Assume we have an event with KV pairs as below:
<timestamp> a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008, ...

For all scenarios below, first create a Parser function to extract all fields by selecting K=V Pairs from Parser Type. Then proceed with another Parser function right below it.

Scenario A: Serialize fields a, b, c, d in CSV format
Expected result: _raw field will have this value 000,001,002,003
Parser 2

• Operation Mode: Serialize
• Source Field: <empty>
• Destination Field: <empty>
• Type: CSV
• List of Fields: a, b, c, d (needed for positional formats)

Scenario B: Serialize fields a, b, c in JSON format, under a field called bar
Expected result: bar field will be set to: {"a":"000","b":"001","c":"002","d":"003"}
Parser 2

• Operation Mode: Serialize
• Source Field: <empty>
• Destination Field: bar
• Type: JSON
• List of Fields: <empty>
• Fields to Keep: a, b, c, d