Description
The Parser Function can be used to extract fields out of events, or to reserialize (rewrite) events with a subset of fields. Reserialization will maintain the format of the events.
For example: If an event contains comma-delimited fields, and fieldA
and fieldB
are filtered out, those fields' positions will be set to null
, but not deleted completely.
Parser cannot remove fields that it did not create. A subsequent Eval Function can do so.
Usage
Filter: Filter expression (JS) that selects data to be fed through the Function. Defaults to true
, meaning that all events will be evaluated.
Description: Simple description about this Function. Defaults to empty.
Final: If true, stops data from being fed to the downstream Functions. Defaults to No
.
Operation mode: Extract will create new fields. Reserialize will extract, filter fields, and then reserialize.
Type: Parser/Formatter type to use. Options:
- CSV
- Extended Log File Format (ELFF)
- Common Log Format (CLF)
- K=V Pairs
- JSON
- Delimited Values
Setting Type to Delimited Values displays the following extra options:
- Delimiter: Delimiter character to split value. Defaults to comma (
,
). You can also specify pipe (|
) or tab characters. - Quote char: Character used to quote literal values. Defaults to
"
. - *Escape char: Character used to escape delimiter or quote characters. Defaults to
\
. - Null value: Field value representing the null value. These fields will be omitted. Defaults to
-
.
Library: Select an option from the Parsers Library.
Source field: Field that contains text to be parsed. Not usually needed in Serialize mode.
Destination field: Name of field in which to add extracted and serialized fields. If multiple new fields are created and this setting is configured then all new fields are created as elements of an array with the array name set to the name specified for this setting. If you want all new fields to be independent, rather than in an array, then specify them using List of fields below. (Extract and Serialize modes only.)
Clean fields: This option appears for Type: K=V Pairs. Toggle to Yes
to clean field names by replacing non-alphanumeric characters with _
. This will also strip leading and trailing "
symbols.
List of fields: Fields expected to be extracted, in order. If not specified, Parser will auto-generate fields.
Fields to keep: List of fields to keep. Supports wildcards (*
). Takes precedence over Fields to remove. Nested addressing supported.
Fields to remove: List of fields to remove. Supports wildcards (*
). Cannot remove fields matching Fields to keep. Nested addressing supported.
Negated terms are supported in both Fields to remove and Fields to keep. When you use negated terms, the list is order-sensitive. E.g.,
!foobar, foo*
means "All fields that start withfoo
, exceptfoobar
." However,!foo*, *
means "All fields, except for those that start withfoo
."
Fields filter expression: Expression to evaluate against {index, name, value}
context of each field. Return truthy to keep, falsy to remove field. Index is zero-based.
How Fields Settings Interact
The Fields to keep, Fields to remove, and Fields filter expression settings interact as follows:
-
Order of evaluation: Fields to keep > Fields to remove > Fields filter expression.
-
If a field is in both Fields to keep and Fields to remove, Fields to keep takes precedence.
-
If a field is in both Fields to remove and Fields filter expression, Fields to remove takes precedence.
Example 1
Insert the following sample, using Preview > Add a Sample > Paste a Sample:
2019/06/24 05:10:55 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008
Create the following test Parser Function (or import this Pipeline: https://raw.githubusercontent.com/weeb-cribl/cribl-samples/master/parser/functions/parser/parser_1.json).


Parser Function initial configuration
First, set the Parser type to Key=Value Pairs
.
Scenario A:
Keep fields a
, b
, c
. Drop the rest.
Expected result: a
, b
, c
- Fields to Keep:
a
,b
,c
- Fields to Remove:
*
- Fields Filter Expression:
Result: The event will gain four new fields and values, as follows.
- a:
000
- b:
001
- c:
002
- cribl_pipe:
parser2


Scenario A result
You can check your stats by clicking the Preview pane’s Basic Statistics (chart) button. In the resulting pop-up, the Number of Fields should have incremented ty four.
Now that you have the hang of it, try out the other simple scenarios below.
Scenario B:
Keep fields a
, b
, those that start with g
. Drop the rest.
Expected result: a
, b
, g1
, g2
, g3
- Fields to keep:
a
,b
- Fields to remove: [empty]
- Fields filter expression:
name.startsWith('g')
Scenario C:
Keep fields a
, b
, those that start with g
but only if value is 007
. Drop the rest.
Expected result: a
, b
, g2
- Fields to keep:
a
,b
- Fields to remove: [empty]
- Fields filter expression:
name.startsWith('g') && value=='007'
Scenario D:
Keep fields a
, b
, c
, those that start with g
, unless it's g1
. Drop the rest.
Expected result: a
, b
, c
, g2
, g3
- Fields to keep:
a
,b
,c
- Fields to remove:
g1
- Fields filter expression:
name.startsWith('g')
Scenario E:
Keep fields a
, b
, c
, those that start with g
but only if index is greater than 6
. Drop the rest.
Expected result: a
, b
, c
, g2
, g3
- Fields to keep:
a
,b
,c
- Fields to remove: [empty]
- Fields filter expression:
name.startsWith('g') && index>6
The
index
refers to the location of a field in the array of all fields extracted by this Parser. It is zero-based. In the case above,g2
andg3
haveindex
values of7
and8
, respectively.
Example 2
Assume we have a JSON event that needs to be reserialized, given these requirements:
- Remove the
level
field only if it's set toinfo
. - Remove the
startTime
field, and all fields in thevalues.total.
path that end inCxn
.
Parser Function configuration:


Parser Function configuration for Example 2
JSON event after being processed by the Function:


Example 2 event transformation
Example 3
Insert the following sample, using Preview > Add a Sample > Paste a Sample:
2019/06/24 15:25:36 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008,
For all scenarios below, first create a Parser Function to extract all fields, by setting the Parser type to Key=Value Pairs
. Then add a second Parser Function with the configuration shown under Parser 2.
Scenario A:
Serialize fields a
, b
, c
, d
in CSV format.
Expected result: _raw
field will have this value 000,001,002,003
Parser 2:
- Operation mode:
Serialize
- Source field: [empty]
- Destination field: [empty]
- Type:
CSV
- List of fields:
a
,b
,c
,d
(needed for positional formats)
Scenario B:
Serialize fields a
, b
, c
in JSON format, under a field called bar
.
Expected result: bar
field will be set to: {"a":"000","b":"001","c":"002","d":"003"}
Parser 2:
- Operation mode:
Serialize
- Source field: [empty]
- Destination field:
bar
- Type:
JSON
- List of fields: [empty]
- Fields to keep:
a
,b
,c
,d
Updated about a month ago