On This Page

Home / Cribl Insights/ Alerts/Monitors

Monitors

The Monitors tab lists default Monitors you can enable, tune, and mute to get notified about meaningful conditions across your environment. You can adjust firing conditions, rule thresholds, add metadata, and route Notifications to targets such as email or webhooks.

The provided Monitors focus on:

  • System health: Resource and component health signals such as Worker availability, CPU, memory, and other metrics that indicate whether Cribl products are up, responsive, and within safe operating thresholds.
  • Data flow and reliability: Key data-path behaviors such as dropped events, queue buildup, ingestion or delivery gaps, and other symptoms that your Pipelines are backing up, losing data, or behaving unexpectedly.
  • Monitoring and alerting behavior: Conditions that validate Insights itself is working as intended, such as failures in Monitor evaluation or notification delivery, so you can trust that if something is wrong you’ll actually get alerted.

Configure a Monitor
​

A Monitor configuration consists of the signal to evaluate defined by a query, the conditions for firing and clearing alerts, how severities map to thresholds, and how notifications are routed.

The default Monitors limit the ability to modify their Name, Product, Description, and Query. To adjust a default Monitor:

  1. Go to Insights > Alerts > Monitors. Select an existing Monitor to edit its configuration.

  2. Firing Conditions control the sensitivity of the monitor to prevent alerts from rapidly toggling on and off.

    • Trigger delay: The amount of time a condition must remain true before an alert opens. Increase this for spiky metrics to discard transient issues.
    • Clear delay: The amount of time a condition must remain healthy before the alert resolves. Set Clear delay greater than or equal to Trigger delay to ensure the metric has truly stabilized before closing the alert.

  3. Rules map your data values to severity levels. You can define a default rule and add overrides for specific scenarios.

    • Rule Configuration:
      • Name: Give the rule a name.
      • Show on chart: Toggle this to visualize the threshold line or band directly on the metric chart.
      • Included/Excluded tags: Scope the rule to specific subsets of metric series. For example, include env=prod or exclude wg=dev. For where tag keys and values come from and how the UI populates them, see Included and Excluded Tags.
    • Severity Table: For each rule, configure the rows for Critical, Warning, and Info:
      • Severity: Check the box to enable the severity level.
      • Operator: Select the comparison logic. For example, greater_than or less_than.
      • Threshold: Enter the numeric value to trigger the alert.
      • Times Triggered: (Optional) Require the condition to match for a number of consecutive evaluations before triggering. This reduces noise without lengthening the Trigger delay.

        Start with Warning less than Critical to avoid overlapping ranges. If grouping by route or pipeline, use Add Override to create stricter thresholds for critical paths and looser thresholds for non-production traffic.

  4. In the Metadata section, use Add Fields to attach arbitrary key/value pairs to the Monitor. For example, env=prod, team=platform. These labels are included on every alert that the Monitor generates and become part of the labels map, so they can be used by both notification policies and templates for emails, routing, filtering, and formatting.

  5. Notifications determine how to deliver alerts.

    • Notification Mode:
      • Route via Notification Policies: (Recommended) Route alerts based on labels. Each policy’s label conditions test labels on the alert, including labels (fields) you added in the Monitor Metadata.
      • Select Specific Notification Target: Select specific Destinations like Email, Slack, or Webhook. Use this for ad-hoc or temporary routing.

  6. Save and Enable the Monitor configuration.

  7. To validate the configuration:

    • Observe the preview chart to ensure the rule lines appear where expected relative to your data.
    • To test safely, temporarily lower the threshold (or target a non-production tag) to force an alert, then revert the changes.
    • If the Monitor is too noisy, consider increasing the Trigger delay, increasing Times Triggered, or widening the Evaluation Window.

Included and Excluded Tags
​

Included/Excluded tags scope a rule to specific subsets of metric series.

In Cribl Insights, tags for this control come from the label keys and values in the Monitor query, not from arbitrary tags on Cribl Stream or Cribl Edge configuration. For example, if your query groups by output, output_type, and __worker_group, those become the available tag keys, and their values populate from metrics that have actually been ingested with those labels.

  • Drop-down values only appear after metrics with those label values have been observed in the selected time range.
  • You can manually type a value to anticipate a label that will appear later.
  • Tags that you add on Sources and Destinations in Cribl Stream or Cribl Edge do not automatically become alert tags unless they are also emitted as metric labels and referenced in the Monitor query.

To alert only on a single Destination, add an Included tag with output=<destination_name>, preferably selected from the drop-down after that Destination has generated total_dropped_events metrics.

Email Metadata
​

For SMTP (email) notifications, use metadata to drive recipients without cloning the default template. For example:

  • Add to : alerts-oncall@example.com
  • Add cc : sre@example.com (optional)

These are read from the email template fields {{metadata.to}} and {{metadata.cc}} to populate the email’s headers.