About Notifications

With Notifications, you can set up alerts to notify Cribl Edge system administrators about events that require their immediate attention.

For example, you could set up a Notification to alert you when a Source stops sending data to a given Fleet, or when the volume of data sent to a Destination from the Fleet exceeds a threshold.

Conceptual Walkthrough of Notifications

​

This one-minute video provides a conceptual walkthrough of the Cribl Edge Notification capabilities.

Types of Notifications

​

Source and Destination Notifications operate based on information for the whole Fleet. For example, if you configure a Source Low Data Volume Notification in a Fleet of 10 Edge Nodes, Cribl Edge will send a Notification when the sum of data for all 10 Edge Nodes is below the configured threshold.

You can also configure Cribl Stream to relay Notifications generated by other Cribl products. For an example, see the Cribl Search topic Notifications via Cribl Stream.

Notifications require certain plan or license tiers. Without an appropriate license, the configuration options described here will be hidden or disabled in Cribl Edge. For more information on license types, and a feature comparison, see Pricing.

How to Get Notifications

​

Notifications appear as events in the user interface and internal logs, providing both application-wide views and filtered insights for affected Sources and Destinations. Cribl Edge stores these application-wide logs in the notifications.log file on the Leader Node, which also handles sending all Notifications.

Beyond internal logs and user interface Notifications, you can also configure Notification targets to send alerts through email, pagers, Slack messages, and more. For information about available targets and setup instructions, see Notification Targets.

Notification Permissions

​

Notifications work with Cribl Edge role-based access control (RBAC). For users with non-administrative permissions, their assigned Roles and Policies determine the Worker Groups on which they can view Notification messages, configure Notifications, and configure targets.

Add a Notification for a Source

​

To create alerts for a Source:

1. Navigate to your Fleet.

2. Select More > Sources.

3. Select the Source type to display the table of its configured Sources.

4. Select the Notifications button on the right side of the Source row you want to configure Notifications for.

5. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

6. In the Configuration section in the When… field, select the condition that will trigger the Notification and then configure the settings for that trigger condition. See Source Settings for more information about the trigger condition settings. The available Source Notification trigger conditions:

   • High Data Volume: Incoming data is above the data volume threshold for your configured time window.
   • Low Data Volume: Incoming data is below the data volume threshold for your configured time window.
   • No Data Received: The Source or Collector ingests zero data over your configured time window.
   • Persistent Queue Usage: The Source persistent queue accumulates files past the threshold percentage of storage capacity. See Optimize Source Persistent Queues for more information about working with Source persistent queues to manage backpressure.

7. In the Configuration section, set the Only notify on start and resolution toggle. When toggled on, Cribl Edge will send a Notification when the triggering condition begins and then a second Notification to report when the condition has ended.

8. Configure the Send Notification to setting to set the Notification target(s). If you don’t have a Notification target, the Messages icon will indicate you have a new Notification. See Notification Targets for more information about available targets and adding new targets.

   • Select Add Target to add an existing target.
   • Select Create Target to add a new target.

9. Optional: Under Metadata, add custom metadata fields if needed. Cribl Edge includes custom metadata in the Notification payload. See Metadata for more information.

10. Save the Notification.

Consider testing the condition to determine if you configured the Notification properly by deliberately triggering the Notification trigger.

Cloning a Source does not copy its associated Notifications. You need to recreate any Notifications manually on the cloned Source.

Add a Notification for a Destination

​

To create alerts for a Destination:

1. Navigate to your Fleet.

2. Select More > Destinations.

3. Select the Destination type to display the table of its configured Destinations.

4. Select the Notifications button on the right side of the Destination row you want to configure Notifications for.

5. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

6. In the Configuration section in the When… field, select the condition that will trigger the Notification and then configure the settings for that trigger condition. See Destination Settings for more information about the trigger condition settings. The available Destination Notification trigger conditions:

   • Destination Backpressure Activated: The Destination has engaged its backpressure behavior, such as when a downstream Destination receiver is unreachable or slow to respond. The triggering conditions:
     • The Destination’s Backpressure behavior is set to Block or Drop, and backpressure causes outgoing events to block or drop.
     • The Destination’s Backpressure behavior is set to Persistent Queue and its Queue‑full behavior is set to either Block or Drop new data. When the queue is full, it causes the Destination to block or drop outgoing events.
   • Persistent Queue Usage: The Destination persistent queue accumulates files past the threshold percentage of storage capacity. See Optimize Destination Persistent Queues for more information about working with Destination persistent queues to manage backpressure.
   • Unhealthy Destination: The health of the Destination has been in red status (as indicated on the Monitoring page) over the configured time window.

7. In the Configuration section, set the Only notify on start and resolution toggle. When toggled on, Cribl Edge will send a Notification when the triggering condition begins and then a second Notification to report when the condition has ended.

8. Configure the Send Notification to setting to set the Notification target(s). If you don’t have a Notification target, the Messages icon will indicate you have a new Notification. See Notification Targets for more information about available targets and adding new targets.

   • Select Add Target to add an existing target.
   • Select Create Target to add a new target.

9. Optional: Under Metadata, add custom metadata fields if needed. Cribl Edge includes custom metadata in the Notification payload. See Metadata for more information.

10. Save the Notification.

Consider testing the condition to determine if you configured the Notification properly by deliberately triggering the Notification trigger.

Cloning a Destination does not copy its associated Notifications. You need to recreate any Notifications manually on the cloned Destination.

Add a Notification about License Expiration

​

To create a license-expiration alert:

1. Navigate to Settings > Global > Licensing.

2. Select Add Expiration Notification.

3. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

4. Optional: Add or create a Notification target. See Notification Targets for more information. If you don’t have a Notification target, the Messages icon will indicate you have a new Notification.

5. Save the Notification.

Notification Settings

​

This section contains the reference for the fields in the Notification settings.

Source Settings

​

The function of the Data volume and Time Window settings change depending on which Notification trigger you selected:

Source name: This field locks to the Source for this Notification.

Time window: This field’s value sets the threshold period before the Notification will trigger. The default 60s will generate a Notification when the Source has reported the trigger condition over the past 60 seconds. To enter alternative numeric values, append units of s for seconds, m for minutes, h for hours, and so forth.

Data volume: Enter the threshold above which a Notification will trigger. Accepts numerals with units like KB, MB, and so forth. For example: 4GB. If you want the unit to be bytes, enter the numeral only, without a unit designator.

Usage threshold: The percentage of full or used storage disk capacity. Cribl Edge generates a Persistent Queue usage has surpassed <threshold>% Notification when the Source persistent queue storage disk capacity is past the specified Usage threshold percentage. This field appears only when you configure a Notification for Persistent Queue Usage.

Destination Settings

​

The function of the Time Window settings change depending on which Notification trigger you selected:

Destination name: This field locks to the Destination for this Notification.

Time window: This field’s value sets the threshold period before the Notification will trigger. The default 60s will generate a Notification when the Destination has reported the trigger condition over the past 60 seconds. To enter alternative numeric values, append units of s for seconds, m for minutes, h for hours, and so forth.

Usage threshold: The percentage of full or used storage disk capacity. Cribl Edge generates a Persistent Queue usage has surpassed <threshold>% Notification when the Destination persistent queue storage disk capacity is past the specified Usage threshold percentage. This field appears only when you configure a Notification for Persistent Queue Usage.

Metadata

​

Metadata fields are user-defined fields included in the Notification payload. All Notification types can contain metadata.

Select Add field here to add custom metadata fields to your Notifications in the form of key-value pairs:

• Name: Enter a name for this custom field.

• Value: Enter a JavaScript expression that defines this field’s value, enclosed in quotes or backticks. (Can evaluate to a constant.)

Once you’ve saved your Notifications, you can see Notification events specific to this Destination on the Events tab within the Destination configuration window. When you set Source-state Notifications, a corresponding Events tab is available on Source and Collector config modals. For a comprehensive view of all Notification events, see the system-wide Notifications Tab.

Manage Notifications

​

You can manage existing Notifications and targets by selecting Notifications in the sidebar. You can also reach this page by selecting an existing Notification in a Source or Destination modal or in Settings > Global > Licensing.

Notifications Tab

​

This tab lists all your configured Source and Destination Notifications, across all integrations, along with any configured license-expiration Notifications. You can’t create new Notifications here, but you can disable or delete existing Notifications. For any individual Notification, select View Events to see events that have triggered the Notification.

To modify a Notification, select anywhere on its row.

Targets Tab

​

This tab is where you centrally configure and manage targets that are available across Cribl Edge for all Sources, Destinations, and license-based Notifications. See Notification Targets for details.

To create a new target, select Add Target. To delete a target, select Delete in the appropriate row.