On This Page

Home / Stream/ Using Integrations· Integrating with Other Services·Nightfall Integration

Nightfall Integration

Thanks to Nightfall for contributing this page and the corresponding Nightfall Pack, which you can install in Cribl Stream and customize as described below.

The Nightfall Pack relies on Nightfall’s Data Loss Prevention (DLP) engine, which uses machine learning to detect sensitive information in events streamed through Cribl. The Pack enables you to monitor your Pipelines in real time for PII, PHI, PCI, and other sensitive data. Nightfall discovers and automatically redacts the sensitive content before it can reach any Destination. Aside from redaction, you can use the Pack to automatically label problematic event data.

The risk of sensitive data spreading across data pipelines grows as organizations work with data from more sources. Nightfall can help ensure that sensitive data doesn’t accumulate in the first place, facilitating compliance with data-security regulations. This is important for Cloud-first organizations that must scale observability while complying with regional or industry-specific compliance regimes.

Configuring the Detection Engine
​

  1. Sign up for a free Nightfall Developer Platform account, if you have not already done so. This entitles you to scan up to 3GB of event data per month for free.

  2. Log in to the Nightfall Dashboard.

  3. Create an API Key.

  4. Create the Detection Rules and/or Policy that you want Nightfall to scan with.

    Detection Rules enable you to specify which Detectors will scan each event. A Detection Rule simply redacts sensitive information from events.

    A Policy governs alerts, which Nightfall can send via email, Slack, or webhook. (Assuming that you want to send data to a Cribl Stream Destination and/or a SIEM, use a webhook.) A Policy can include up to 20 previously defined Detection Rules. It can also turn redaction off, if desired.

Installing the Nightfall Pack
​

  1. From Cribl Stream’s top nav, select Processing > Packs.
  2. Click Add Pack, then select Add from Dispensary to open the Packs Dispensary drawer.
  3. Navigate to the DLP by Nighfall AI Pack’s tile.
Locating the Nightfall Pack
Locating the Nightfall Pack
  1. Click the Nightfall Pack’s tile to display its details page.
The Nightfall Pack details page
The Nightfall Pack details page
  1. Click Add Pack. Shortly, you’ll see a banner confirming that the Pack is installed.

Configuring the Nightfall Pack
​

  1. From the Manage Packs page, click the new Pack. This opens the Pack’s Routes tab.
  2. In the Pipeline column, click Nightfall.
  3. From the resulting Nightfall Pipeline configuration, expand the DLP Scanner with Nightfall AI Function.
Selecting the Nightfall Pipeline
Selecting the Nightfall Pipeline

  1. In the Nightfall API Key field, enter the API key you created in the Nightfall Dashboard.

  2. Set the Scan with a Policy or Multiple Detection Rules drop-down to match your detection engine configuration.

    Selecting Policy UUID exposes a corresponding field to enter the single UUID.

    Selecting Detection Rule UUID exposes a Add Rule. Click this to enter as many Detection Rule UUIDs as you want to specify.

  3. (Optionally:) Label sensitive events. If you toggle Label Log Lines with nightfall_violations to Yes, each event where Nightfall has detected sensitive information will have an added nightfall_violations: true field.

Advanced Features
​

The Nightfall Pipeline also includes three limiting Functions, described here. These are disabled by default, but you can enable them as needed.

Sampling
​

To reduce the data usage that’s billed against your Nightfall plan, you can enable the Sampling Function. This way, Nightfall will scan only a subset of each streamed file, instead of every event.

Log Batching
​

To reduce the number of requests that Cribl Stream makes to the Nightfall API, you can enable log batching. To do so, enable both the Aggregations and Unroll Functions, then configure the Aggregation Function’s Time window.

The Nightfall DLP plugin will automatically detect the aggregations, and unroll the batches, only after Cribl Stream sends them to the Detection Engine. Then, Nightfall will process each event separately, as it normally does.