Notifications

With Notifications, you can set up alerts to notify Cribl Edge system administrators about events that require their immediate attention.

For example, you could set up a Notification to alert you when a Source stops sending data to a given Fleet, or when the volume of data sent to a Destination from the Fleet exceeds a threshold.

Notifications require certain plan or license tiers. Without an appropriate license, the configuration options described here will be hidden or disabled in Cribl Edge. For more information on license types, and a feature comparison, see Pricing.

How to Get Notifications

​

You can get notifications through:

• Internal logs: Cribl Edge stores these application-wide logs in the notifications.log file on the Leader Node, which also handles sending all Notifications. See Internal Logs for more information about logs.
• Events in the UI: Displays events in the Monitoring page. See Monitoring for more information about UI notifications.
• Custom targets: You can configure Notification targets to send alerts through email, pagers, Slack messages, and more. See Notification Targets for information about available targets and setup instructions.

Types of Notifications

​

You can also configure Cribl Stream to relay Notifications generated by other Cribl products. For an example, see the Cribl Search topic Notifications via Cribl Stream.

Notification Permissions

​

Notifications work with Cribl Edge role-based access control (RBAC). For users with non-administrative permissions, their assigned Roles and Policies determine the Fleets on which they can view Notification messages, configure Notifications, and configure targets.

Add a Notification for a Source

​

To create alerts for a Source or Collector:

1. Navigate to your Fleet. Then select More > Sources.

2. Select the Source type to display the table of its configured Sources. For example, select Syslog to set up notifications for a Syslog Source.

3. Select the Notifications button in the row for the Source you want to configure.

4. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

5. In the Configuration section in the When… field, select the condition that will trigger the Notification and then configure the settings for that trigger condition. See Source Settings for more information about the trigger condition settings. The available Source Notification trigger conditions:

   • High Data Volume: Incoming data is above the data volume threshold for your configured time window.
   • Low Data Volume: Incoming data is below the data volume threshold for your configured time window.
   • No Data Received: The Source or Collector ingests zero data over your configured time window.
   • Persistent Queue Usage: The Source persistent queue accumulates files past the threshold percentage of storage capacity. See Optimize Source Persistent Queues for more information about working with Source persistent queues to manage backpressure.

6. In the Configuration section, set the Only notify on start and resolution toggle. When toggled on, Cribl Edge will send a Notification when the triggering condition begins and then a second Notification to report when the condition has ended.

7. Configure the Send Notification to setting to set the Notification target(s). If you don’t have a Notification target, the Messages icon will indicate you have a new Notification. See Notification Targets for more information about available targets and adding new targets.

   • Select Add Target to add an existing target.
   • Select Create Target to add a new target.

8. Save the Notification.

Your Notification will now appear on the Events tab within the configuration window for this Source. To view all Notification events and targets, select Notifications in the sidebar.

Consider testing the condition to determine if you configured the Notification properly by deliberately triggering the Notification trigger.

Source Notification Settings

​

The function of the Data volume and Time window settings change depending on which Notification trigger you selected:

Source name: This field locks to the Source for this Notification and cannot be edited.

Time window: This field’s value sets the threshold period before the Notification will trigger. The default 60s will generate a Notification when the Source has reported the trigger condition over the past 60 seconds. To enter alternative numeric values, append units of s for seconds, m for minutes, h for hours, and so forth.

See Source Notifications Best Practices for information about avoiding false notifications when configuring the Time window setting for Collectors that run on a schedule.

Data volume: Enter the threshold above which a Notification will trigger. Accepts numerals with units like KB, MB, and so forth. For example: 4GB. If you want the unit to be bytes, enter the numeral only, without a unit designator.

Usage threshold: The percentage of full or used storage disk capacity. Cribl Edge generates a Persistent Queue usage has surpassed <threshold>% Notification when the Source persistent queue storage disk capacity is past the specified Usage threshold percentage. This field appears only when you configure a Notification for Persistent Queue Usage.

Metadata: User-defined fields included in the Notification payload. All Notification types can contain metadata. To add custom metadata fields, select Add field and add the fields as key-value pairs:

• Name: Enter a name for the custom field.
• Value: Enter a JavaScript expression that defines this field’s value, enclosed in quotes or backticks. This can evaluate to a constant if needed.

Source Notification Best Practices

​

To ensure your Source Notifications are reliable and easy to manage, follow these best practices:

• For Collectors that run on a schedule, always set the No Data Received Notification time window to a duration that is longer than the time between scheduled runs. Because there is no data flow between runs, a shorter time window will trigger false alarms.
• When cloning a Source, remember to manually recreate any Notifications you need on the new Source. Cloning a Source does not copy its associated Notifications.

Add a Notification for a Destination

​

To create alerts for a Destination:

1. Navigate to your Fleet. Then select More > Destinations.

2. Select the Destination type to display the table of its configured Sources. For example, select Syslog to set up notifications for a Syslog Destination.

3. Select the Notifications button in the row for the Destination you want to configure.

4. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

5. In the Configuration section in the When… field, select the condition that will trigger the Notification and then configure the settings for that trigger condition. See Destination Settings for more information about the trigger condition settings. The available Destination Notification trigger conditions:

   • Destination Backpressure Activated: The Destination has engaged its backpressure behavior, such as when a downstream Destination receiver is unreachable or slow to respond. The triggering conditions:
     • The Destination’s Backpressure behavior is set to Block or Drop, and backpressure causes outgoing events to block or drop.
     • The Destination’s Backpressure behavior is set to Persistent Queue and its Queue‑full behavior is set to either Block or Drop new data. When the queue is full, it causes the Destination to block or drop outgoing events.
   • Persistent Queue Usage: The Destination persistent queue accumulates files past the threshold percentage of storage capacity. See Optimize Destination Persistent Queues for more information about working with Destination persistent queues to manage backpressure.
   • Unhealthy Destination: The health of the Destination has been in red status (as indicated on the Monitoring page) over the configured time window.

6. In the Configuration section, set the Only notify on start and resolution toggle. When toggled on, Cribl Edge will send a Notification when the triggering condition begins and then a second Notification to report when the condition has ended.

7. Configure the Send Notification to setting to set the Notification target(s). If you don’t have a Notification target, the Messages icon will indicate you have a new Notification. See Notification Targets for more information about available targets and adding new targets.

   • Select Add Target to add an existing target.
   • Select Create Target to add a new target.

8. Save the Notification.

Your Notification will now appear on the Events tab within the configuration window for this Destination. To view all Notification events and targets, select Notifications in the sidebar.

Destination Notification Settings

​

The function of the Time Window settings change depending on which Notification trigger you selected:

Destination name: This field locks to the Destination for this Notification.

Time window: This field’s value sets the threshold period before the Notification will trigger. The default 60s will generate a Notification when the Destination has reported the trigger condition over the past 60 seconds. To enter alternative numeric values, append units of s for seconds, m for minutes, h for hours, and so forth.

Usage threshold: The percentage of full or used storage disk capacity. Cribl Edge generates a Persistent Queue usage has surpassed <threshold>% Notification when the Destination persistent queue storage disk capacity is past the specified Usage threshold percentage. This field appears only when you configure a Notification for Persistent Queue Usage.

Metadata: User-defined fields included in the Notification payload. All Notification types can contain metadata. To add custom metadata fields, select Add field and add the fields as key-value pairs:

• Name: Enter a name for the custom field.
• Value: Enter a JavaScript expression that defines this field’s value, enclosed in quotes or backticks. This can evaluate to a constant if needed.

Destination Notification Best Practices

​

When cloning a Destination, remember to manually recreate any Notifications you need on the new Destination. Cloning a Destination does not copy its associated Notifications.

Add a Notification About License Expiration

​

To create a license-expiration alert:

1. Navigate to Settings > Global > Licensing. Then select Add Expiration Notification.

2. In the Notification modal, configure the following settings under General:

   • ID: Provide a unique ID for the Notification in this section.
   • Enabled: Toggle on (default) if you want Cribl Edge to enable the Notification. Toggle off to turn off the Notification.

3. Optional: Add or create a Notification target. See Notification Targets for more information. If you don’t have a Notification target, the Messages icon will indicate you have a new Notification.

4. Save the Notification.

To view all Notification events and targets, select Notifications in the sidebar. You can also reach this page by selecting Settings > Global > Licensing.

Consider testing the condition to determine if you configured the Notification properly by deliberately triggering the Notification trigger.

Edit Notifications

​

To edit a Notification:

1. Select Notifications in the sidebar. You can also reach this page by selecting an existing Notification in a Source or Destination modal. This page lists all Notifications.

2. Select anywhere on the row the Notification you want to edit.

3. To edit a Notification target (the delivery method for the Notification), use the Targets tab on this screen. See Notification Targets for additional details.