Destination Backpressure Activated

​

Cribl Stream will generate a Notification when one of the following events occurs:

• The Destination’s Backpressure behavior is set to Block or Drop, and backpressure causes outgoing events to block or drop.
• The Destination’s Backpressure behavior is set to Persistent Queue, but its Queue‑full behavior is set to either Block or Drop new data, and a filled queue causes the Destination to to block or drop outgoing events.

The threshold for the Notification to trigger is: Cribl Stream detected a blocked or dropped state during ≥ 5% of the trailing Time window that you configure in the Configuring Destination Notifications.

Persistent Queue Usage

​

Cribl Stream will generate a Persistent Queue usage has surpassed <threshold>% Notification when the PQ accumulates files past the <threshold> percentage of capacity that you set in the Usage threshold field. This field appears only when you configure a Notification for Persistent Queue Usage.

Unhealthy Destination

​

Cribl Stream will generate a Destination <name> is unhealthy Notification when the Destination’s health has been in red status (as indicated on the UI’s Monitoring page) over the trailing Time window that you configure in Configuring Destination Notifications.

The algorithm has slight variations among Destination types, but red status generally means that ≥ 5% of health checks, aggregated over the Time window, reported one of the following conditions:

• An error inhibiting the Destination’s normal operation, such as a connection error.
• For multiple-output Destinations like Splunk Load Balanced or Output Router, > 50% of the Destination senders are in an error state.

Destination-State Notifications for Webhook Targets

​

If you are sending a Destination-state Notification to a Webhook Notification target, you can include a variety of expression fields in the target’s Source expression. For more information, see:

• Fields Common to All Notification Types
• Unhealthy Destination
• Destination Backpressure Activated

Configuring Destination Notifications

​

To configure a Destination-state Notification:

1. Configure and save the Destination.
2. Access this Destination’s Notifications tab by one of the following methods:
   • Click the Notifications button on the Manage…Destinations page’s appropriate row, or
   • Reopen the Destination’s config modal and click its Notifications tab.
3. Click Add Notification to access the New Notification modal shown below.

General

​

ID: Enter a unique ID for this Notification. Notifications are enabled by default, but you can disable the Notification by setting Enabled to No.

Configuration

​

When: Select one of the following Notification tiles:

• Destination Backpressure Activated
• Persistent Queue Usage
• Unhealthy Destination

You can set up multiple Notifications for the same Destination, but you must configure them separately.

Send Notification to: Click Add Target to send this Notification to additional targets. You can add multiple targets.

• Use the resulting Notification targets drop-down to select any target you’ve already configured.
• Click Create Target to configure a new target.

See Notification Targets for details.

Default target is always locked to System Messages.

Destination name: This field is locked to the Destination on which you’re setting this Notification.

Time window: This field’s value sets the threshold period before the Notification will trigger. The default 60s will generate a Notification when a Destination or Source has reported the trigger condition over the past 60 seconds. To enter alternative numeric values, append units of s for seconds, m for minutes, h for hours, and so forth.

Only notify on start and resolution: When this option is set to Yes, Cribl Stream will send a Notification at the onset of the triggering condition and a second Notification to report its resolution.

If you don’t enable this option and a Destination-state Notification’s trigger condition persists beyond your configured Time window, Cribl Stream will send a new Notification, once per Time window interval.

Metadata

​

You can enter user-defined fields called metadata, which Cribl Stream includes in the notification payload. See Metadata for more information.