As data travels through a Cribl LogStream Pipeline, it is operated on by a series of Functions. Functions are fundamentally JavaScript code.

Functions that ship with Cribl LogStream are configurable via a set of inputs. Some of these configuration options are literals, such as field names, and others can be JavaScript expressions.

Expressions are valid units of code that resolve to a value. Every syntactically valid expression resolves to some value, but conceptually, there are two types of expressions: those that assign value to a variable (a.k.a., with side effects), and those that evaluate to a value.

Filters and Value Expressions

Filters

Filters are used in Routes to select a stream of the data flow, and in Functions to scope or narrow down the applicability of a Function. Filters are expressions that must evaluate to either true (or truthy) or false (or falsy). Keep this in mind when creating Routes or Functions. For example:

• sourcetype=='access_combined' && host.startsWith('web')
• source.endsWith('.log') || sourcetype=='aws:cloudwatchlogs:vpcflow'

Value Expressions

Value expressions are typically used in Functions to assign a value – for example, to a new field. For example:

• Math.floor(_time/3600)
• source.replace(/.{3}/, 'XXX')

Considerations and Best Practices for Creating Predictable Expressions

• In a value expression, ensure that the source variable is not null, undefined, or empty. For example, assume you want to have a field called len, to be assigned the length of a second field called employeeID. But you're not sure if employeeID exists. Instead of employeeID.length, you can use a safer shorthand, such as: (employeeID || '').length.

• If a field does not exist (undefined), and you're doing a comparison with its properties, then the boolean expression will always evaluate to false. For example, if employeeID is undefined, then both of these expressions will evaluate to false: employeeID.length > 10 , and employeeID.length < 10 .

• == means "equal to," while === means "equal value and equal type." For example, 5 == 5 evaluates to true, while 5 === "5" evaluates to false.

• A ternary operator is a very powerful way to create conditional values. For example, if you wanted to assign either minor or adult to a field groupAge, based on the value of age, you could do: (age >= 18) ? 'adult' : 'minor'.

Expressions Using Fields with Non-Alphanumeric Characters

If there are fields with non-alphanumeric characters – e.g., @timestamp or user-agent or kubernetes.namespace_name – you can access them using __e['<field-name-here>']. (Note the single quotes.) More details here. In any other place where the field is referenced – e.g., in the Eval function's field names – you should use a single-quoted literal, of the form: '<field-name-here>'.

Wildcard Lists

Wildcard Lists are used throughout the product, especially in various Functions, such as Eval, Mask, Publish Metrics, Parser, etc.

Wildcard Lists, as their name implies, accept strings with asterisks (*) to represent one or more terms. They also accept strings that start with an exclamation mark (!) to negate one or more terms.

Wildcard Lists are order-sensitive only when negated terms are used. This allows for implementing any combination of whitelists and blacklists.

For example: