The Monitors tab lists default Monitors you can enable, tune, and mute to get notified about meaningful conditions across your environment. You can adjust firing conditions, rule thresholds, add metadata, and route Notifications to targets such as email or webhooks.

The provided Monitors focus on:

• System health: Resource and component health signals such as Worker availability, CPU, memory, and other metrics that indicate whether Cribl products are up, responsive, and within safe operating thresholds.
• Data flow and reliability: Key data-path behaviors such as dropped events, queue buildup, ingestion or delivery gaps, and other symptoms that your Pipelines are backing up, losing data, or behaving unexpectedly.
• Monitoring and alerting behavior: Conditions that validate Insights itself is working as intended, such as failures in Monitor evaluation or notification delivery, so you can trust that if something is wrong you’ll actually get alerted.

Configure a Monitor

​

A Monitor configuration consists of the signal to evaluate defined by a query, the conditions for firing and clearing alerts, how severities map to thresholds, and how notifications are routed.

The default Monitors limit the ability to modify their Name, Product, Description, and Query. To adjust a default Monitor:

1. Go to Insights > Alerts > Monitors. Select an existing Monitor to edit its configuration.

2. Firing Conditions control the sensitivity of the monitor to prevent alerts from rapidly toggling on and off.

   • Trigger delay: The amount of time a condition must remain true before an alert opens. Increase this for spiky metrics to discard transient issues.
   • Clear delay: The amount of time a condition must remain healthy before the alert resolves. Set Clear delay greater than or equal to Trigger delay to ensure the metric has truly stabilized before closing the alert.

3. Rules map your data values to severity levels. You can define a default rule and add overrides for specific scenarios.

   • Rule Configuration:
     • Name: Give the rule a name.
     • Show on chart: Toggle this to visualize the threshold line or band directly on the metric chart.
     • Included/Excluded tags: Apply the rule only to specific subsets of data. For example, Include env=prod, Exclude wg=dev
   • Severity Table: For each rule, configure the rows for Critical, Warning, and Info:
     • Severity: Check the box to enable the severity level.
     • Operator: Select the comparison logic. For example, greater_than or less_than.
     • Threshold: Enter the numeric value to trigger the alert.
     • Times Triggered: (Optional) Require the condition to match for a number of consecutive evaluations before triggering. This reduces noise without lengthening the Trigger delay.

       Start with Warning less than Critical to avoid overlapping ranges. If grouping by route or pipeline, use Add Override to create stricter thresholds for critical paths and looser thresholds for non-production traffic.

4. In the Metadata section, use Add Fields to attach arbitrary key/value pairs to the Monitor. For example, env=prod, team=platform. These labels are included on every alert that the Monitor generates and become part of the labels map, so they can be used by both notification policies and templates for routing, filtering, and formatting.

5. Notifications determine how to deliver alerts.

   • Notification Mode:
     • Route via Notification Policies: (Recommended) Route alerts based on labels. Each policy’s label conditions test labels on the alert, including labels (fields) you added in the Monitor Metadata.
     • Select Specific Notification Target: Select specific Destinations like Email, Slack, or Webhook. Use this for ad-hoc or temporary routing.

6. Save and Enable the Monitor configuration.

7. To validate the configuration:

   • Observe the preview chart to ensure the rule lines appear where expected relative to your data.
   • To test safely, temporarily lower the threshold (or target a non-production tag) to force an alert, then revert the changes.
   • If the Monitor is too noisy, consider increasing the Trigger delay, increasing Times Triggered, or widening the Evaluation Window.