On This Page

Home / Cribl Insights/ Alerts/Monitors

Monitors ​

Use Monitors to watch key signals across your environment and raise alerts when conditions cross defined thresholds. The page shows a set of default Monitors for common signals and lets you create custom Monitors when you need metric-based checks the defaults do not cover. You can enable, tune, and mute Monitors, adjust firing conditions and rule thresholds, add metadata, and route notifications to targets such as email or webhooks.

Default Monitors focus on:

  • System health: Resource and component health signals such as Worker availability, CPU, memory, and other metrics that indicate whether Cribl products are up, responsive, and within safe operating thresholds.
  • Data flow and reliability: Key data-path behaviors such as dropped events, queue buildup, ingestion or delivery gaps, and other symptoms that your Pipelines are backing up, losing data, or behaving unexpectedly.
  • Monitoring and alerting behavior: Conditions that validate that Cribl Insights is working as intended, such as failures in Monitor evaluation or Notification delivery, so you can trust that if something is wrong you’ll actually get alerted.

Configure a Monitor ​

  1. Go to Insights > Alerts > Monitors.
  2. Do one of the following:
    • Select Add Monitor to create a new custom Monitor.
    • Select an existing Monitor (custom or default) to edit it.

On the Monitor modal, you configure:

General Settings ​

For default Monitors, these fields are read-only. For custom Monitors, they are fully editable.

  • Name: A unique name for the Monitor. Names can include spaces and other allowed characters.
  • Product: The product context, such as Stream, Edge, Search, Lake, Insights, or Outpost.
  • Description: (Optional) A short note that explains the purpose of the Monitor.

The Monitor Query ​

The query defines what data the Monitor evaluates. You configure it with the visual builder. There is no raw code editor.

For default Monitors, the query settings are fixed and read-only. You tune these Monitors using Firing Conditions, Rules, Metadata, and Notifications. Calculation/binary operations are not available.

Custom Monitors:

  • Metric: Choose the base metric to evaluate from the options available to your org.
  • Label filters: Limit the data series using key/value matchers. The UI supports equals = and does not equal != for each row. Regex match operators (such as =~ or !~) are not supported. Select each label’s value from the dropdowns, or type to anticipate a label value that will appear later.
  • Operator: Select the aggregation to apply. The list includes sum, avg, max, min, and count. These determine how samples are rolled up over the evaluation window.
  • Group By: Select one or more dimensions from the labels that exist for the Metric you selected. Each group is evaluated independently and can create its own alert entities, so use grouping sparingly if you want a small number of alerts.
  • Evaluation window: Set the rolling lookback window, such as 5m or 1h. Longer windows smooth out noise, shorter windows react faster to spikes.
  • Unit: Select the display unit that matches your metric (for example, Percent, Bytes, or Milliseconds) so charts and thresholds are accurate.

Firing Conditions ​

Firing Conditions control how long a condition must hold before an alert opens or clears.

  • Trigger delay: The amount of time a condition must remain true before an alert opens. Increase this for spiky metrics to discard transient issues.
  • Clear delay: The amount of time a condition must remain healthy before the alert resolves. Set Clear delay greater than or equal to Trigger delay so the metric has stabilized before closing the alert.

Rules ​

Rules map evaluated values to severity levels. You can define a default rule and add overrides for specific scenarios.

  • Rule Configuration:

    • Name: Give the rule a name.
    • Show on chart: Toggle this to visualize the threshold line or band on the metric chart.
    • Included/Excluded tags: Scope the rule to specific subsets of metric series. For example, include env=prod or exclude wg=dev. For where tag keys and values come from and how the UI populates them, see Included and Excluded Tags.

  • Severity Table:

    For each rule, configure the rows for Critical, Warning, and Info:

    • Severity: Check the box to enable the severity level.
    • Operator: Select the comparison logic. For example, greater_than or less_than.
    • Threshold: Enter the numeric value to trigger the alert.
    • Times Triggered: (Optional) Require the condition to match for a number of consecutive evaluations before triggering. This reduces noise without lengthening the Trigger delay.

Metadata ​

In the Metadata section, use Add Fields to attach arbitrary key/value pairs to the Monitor. For example, env=prod, team=platform. These labels are included on every alert that the Monitor generates and become part of the labels map, so they can be used by both Notification policies and templates for emails, routing, filtering, and formatting.

Notifications ​

Notifications determine how to deliver alerts.

  • Notification Mode:
    • Route via Notification Policies: (Recommended) Route alerts based on labels. Each policy’s label conditions test labels on the alert, including labels (fields) you added in the Monitor Metadata.
    • Select Specific Notification Target: Select specific Destinations like Email, Slack, or Webhook. Use this for ad-hoc or temporary routing.

Validate a Monitor ​

After you save and enable a Monitor:

  • Use the preview chart to confirm the rule lines align with your data. If the chart is slow or shows a timeout, choose a shorter time range and try again.
  • To test safely, temporarily lower a threshold (or target a non-production tag) to force an alert, then revert the change.
  • If the Monitor is too noisy, consider increasing the Trigger delay, increasing Times Triggered, or (for custom Monitors) widening the Evaluation window.

Included and Excluded Tags ​

Included/Excluded tags scope a rule to specific subsets of metric series.

In Cribl Insights, tags for this control come from the label keys and values in the Monitor query, not from arbitrary tags on Cribl Stream or Cribl Edge configuration. For example, if your query groups by output, output_type, and __worker_group, those become the available tag keys, and their values populate from metrics that have actually been ingested with those labels.

  • Drop-down values only appear after metrics with those label values have been observed in the selected time range.
  • You can manually type a value to anticipate a label that will appear later.
  • Tags that you add on Sources and Destinations in Cribl Stream or Cribl Edge do not automatically become alert tags unless they are also emitted as metric labels and referenced in the Monitor query.

To alert only on a single Destination, add an Included tag with output=<destination_name>, preferably selected from the drop-down after that Destination has generated total_dropped_events metrics.

Email Metadata ​

For SMTP (email) notifications, use metadata to drive recipients without cloning the default template. For example:

  • Add to : alerts-oncall@example.com
  • Add cc : sre@example.com (optional)

These are read from the email template fields {{metadata.to}} and {{metadata.cc}} to populate the email’s headers.

Disable a Monitor ​

You can disable a Monitor by toggling Enabled off, but you cannot delete it.

Disabling a Monitor stops new evaluations and clears any active alert instances for that Monitor. The Monitor configuration remains in place so you keep history, auditability, and references from other settings, such as Notification policies or muting rules.