Description

The Mask Function masks, or replaces, patterns in events. This is especially useful for redacting PII (personally identifiable information) and other sensitive data.

Usage

Filter: Filter expression (JS) that selects data to be fed through the Function. Defaults to true, meaning that all events will be evaluated.

Description: Simple description about this Function. Defaults to empty.

Final: If true, stops data from being fed to the downstream Functions. Defaults to No.

• Masking rules: Match Regex and Replace Expression pairs. Defaults to empty.

  • Match regex: Pattern to replace. Capture groups are supported. Use /g to replace all matches, e.g.: /foo(bar)/g
  • Replace expression: A JS expression or literal to replace all matching content.

Apply to fields: Fields on which to apply the masking rules. Defaults to _raw. Wildcards (*) and nested addressing are supported.

📘

Negated terms are also supported. When you negate field names, the fields list is order-sensitive. E.g., !foobar before foo* means "Apply to all fields that start with foo, except foobar." However, !foo* before * means "Apply to all fields, except for those that start with foo."

Evaluating the Replace Expression

The Replace expression field accepts a full JS expression that evaluates to a value, so you're not necessarily limited to what's under C.Mask. For example, you can do conditional replacement: g1%2==1 ? `fieldA="odd"` : `fieldA="even"`

The Replace expression can reference other event fields as event.<fieldName>. For example, `${g1}${event.source}` . Note that this is slightly different from other expression inputs, where event fields are referenced without event. Here, we require the event. prefix for the following reasons:

• We don't expect this to be a common case.
• Expanding the event in the replace context would have a high performance hit on the common path.
• There is a slight chance that there might be a gN field in the event.

Examples

Example 1: Transform a String

Here, we'll simply search for the string dfhgdfgj, and replace that value (if found) with Trans AM. This will help close America’s muscle-car gap:

Configure the Mask Function > Masking Rules as follows:

Match Regex: dfhgdfgj
Replace Expression: Trans AM

Result: Vroom vroom!

Example 2: Mask Sensitive Data

Assume that you're ingesting data whose _raw fields contain unredacted Social Security numbers in the Key=Value pattern social=#########.

You can use a Mask Function to run an md5 hash of the social keys' numeric values, replacing the original values with the hashed values. Configure the Masking Rules as follows:

Match Regex: (social=)(\d+)
Replace Expression: `${g1}${C.Mask.md5(g2)}`

In the first example everything in the Match regex field was replaced by the Replace Expression. However if that isn't desired then you can use capture groups in the Match Regex to define individual string components for manipulation or, alternatively, use string literals in the Replace expression for retaining any static text. Any content matching the Match Regex that is not inserted into the Replace expression will not be retained.

In this example, social= is assigned to capture group g1 for later reference. The value of social= will be hashed by referencing it as g2 in the md5 function. If we didn't make social= its own capture group (or specified social= as a literal in the Replace Expression) then we cannot reference it using g1 in the Replace expression, the value of social= would instead be assigned to g1, and the entire social=######### string would be replaced with a hash of the social security number, which probably isn't desired because no one would know the value being hashed without a field name preceding it.

Result: The sensitive values are replaced by their md5 hashes.

📘

In scenarios where you need to send unmodified values to certain Destinations (such as archival stores), you can narrow the Mask Function's scope by setting the associated Route's Output field.

For further masking examples, see Masking and Obfuscation.