Create Monitors in Cribl Search
Preview Feature
Cribl is still developing this Preview feature. We don’t recommend using it in a production environment, because the feature might not be fully tested or optimized for performance, and related documentation could be incomplete.
Please continue to submit feedback through normal Cribl support channels, but assistance might be limited while the feature remains in Preview.
Get notified when a metric moves outside its expected range, using Monitors.
About Monitors
A Monitor watches a metric query and raises an alert when the metric meets a condition you define, such as crossing a threshold, behaving anomalously, or deviating from its peers.
- A Monitor is what keeps watch - a metric query, its detection method, thresholds, and firing conditions.
- An alert is the recorded moment a Monitor’s condition was met.
- A Notification delivers that alert to where your team works, such as email, Slack, or a webhook.
Monitors are part of the Metrics experience. To explore and visualize metrics before you alert on them, see Explore Metrics in Cribl Search.
To monitor events in your logs, convert them to metrics first. See Convert Logs to Metrics.
During Preview, some routing and suppression controls are not yet wired up - the sections below call out what is coming soon.
Monitor Permissions
Viewing a Monitor needs read access to every metrics Dataset that the Monitor evaluates. Creating, editing, and
deleting Monitors needs that same read access, plus the Editor or Admin Permission on Cribl Search.
Members with the Admin Permission on Cribl Search can manage Monitors on any metrics Dataset. Members with the
Editor Permission can manage Monitors only on the metrics Datasets that a Maintainer has
shared with them. Members with the User Permission can view those
Monitors and their alerts, but can’t change them. For details, see Metrics Permissions.
Open Monitors
On the Cribl.Cloud top bar, select Products > Search > Monitors.
The list shows only the Monitors whose queries all read metrics Datasets you can access. A Monitor on a Dataset you can’t access is hidden rather than shown as locked, so two Members can see different Monitors in the same Organization.
From the Monitors list, you can:
- Search and filter your Monitors.
- Enable or disable a Monitor with its row toggle.
- Select Monitors for bulk actions.
- Select Add Monitor to create one, or select an existing Monitor to edit it.
Build a Monitor
The Monitor page has a live preview chart at the top. Expand the sections below the chart to configure your Monitor settings.
- Turn Simulate on past data on to preview how the Monitor would have behaved against recent data.
- Use the Enabled toggle in the header to turn the Monitor on or off, then select Save.
Configure the Monitor in these sections:
- Detection Method: choose how the Monitor fires.
- General: name the Monitor and set its priority and team.
- Query: choose the metric and shape the query.
- Thresholds: map values to severities.
- Metadata: attach key-value labels to alerts.
- Firing Conditions: control how long a condition must hold.
- Notifications: decide how alerts are delivered.
When you edit an existing Monitor, the page splits into a Configure tab and an Activity tab. See View Alert Activity.
Detection Method
The Detection Method determines how the Monitor decides to fire an alert. Select one:
| Method | Fires when |
|---|---|
| Threshold | Metrics cross defined thresholds. |
| Change | The difference between a value N minutes ago and now. |
| Anomaly | A metric behaves anomalously based on historical data. |
| Outlier | A series deviates from its peers. |
| Forecast | A metric is projected to cross a threshold. |
To compare the series that one query returns, use Outlier. See Configure an Outlier Monitor.
General
- Name: a unique name for the Monitor. Required.
- Description: a short note on what the Monitor checks and why it matters.
- Priority: the urgency of the Monitor: P1 (Critical), P2 (High), P3 (Medium), P4 (Low), P5 (Info), or Not defined.
- Team: the team that owns the Monitor.
Query
The Query section defines what the Monitor evaluates. You can build the query visually or write Prometheus Query Language (PromQL) directly.
Select the Dataset that the query reads.
A new Monitor starts with a query on the first available metric, preferring the metrics Dataset.
Visual builder:
- Metric: the base metric to evaluate.
- Label Filters: narrow the series with key-value matchers. Operators are
=,!=,=~(regex), and!~(regex not). - Function: how samples roll up over the window:
count,avg,sum,min, ormax. Use the dropdown next to Function, which shows No group by until you choose labels, to evaluate one or more labels as separate series. - Evaluation window: the rolling lookback, such as
5mor1h.
PromQL:
- Enter PromQL directly in the query editor.
- Use
$variablein the query to add template parameters. Cribl Search shows a Resolved Query preview with the parameter values filled in.
You can add more than one query, labeled A through D.
Set Units to the unit of measure for the metric, such as seconds, bytes, or count. The unit determines the chart format and the scale of alert thresholds.
When a Monitor has more than one query, or when a query reads a different metrics Dataset than the Monitor’s default,
you need read access to every metrics Dataset the Monitor evaluates. Cribl Search rejects a Monitor that reads a
metrics Dataset you can’t access, and doesn’t save it. See Monitor Permissions.
Outlier Monitors score a single query. See Write the Outlier Query.
Thresholds
Thresholds map evaluated values to severities in a Default Rule. For each row, configure:
- Severity: Critical, Warning, or Info.
- Operator: greater than, greater than or equal, less than, less than or equal, equal to, or not equal to.
- Threshold: the numeric value to compare against.
- Times Triggered: the number of consecutive evaluations the condition must hold before the alert fires. The minimum is 1.
Select Add severity to add a row (up to six). Turn on Show on Chart to draw the threshold on the preview chart.
Use Included Tags and Excluded Tags to scope the rule to specific series.
Outlier Monitors set severities differently. See Set Outlier Thresholds.
Metadata
Add arbitrary key-value pairs, such as env=prod or team=platform. These become labels on every alert the Monitor
generates, so Notification policies and templates can use them for routing and formatting.
Firing Conditions
Firing Conditions control how often Cribl Search evaluates the Monitor and how long a condition must hold before an alert opens or clears. They also set what the Monitor does when data is missing. Delays help Monitors avoid alert flapping when a metric hovers around a threshold. All of these fields are required.
- Evaluate every: how often Cribl Search evaluates the Monitor. The default is 1 minute. Fire and clear delays shorter than this interval take effect at the next evaluation.
- Fire delay (minutes): how long the condition must stay true before the alert fires. The default is 5 minutes.
- Clear delay (minutes): how long the condition must stay healthy before the alert clears and recovers. The default is 10 minutes. A longer delay reduces flapping.
- When no data: what the Monitor does when the whole query returns no data. The default is Show No data.
- Missing series: evaluations before No data: how many evaluations in a row a single series can be missing before its state changes to No data. The default is 1.
The delay fields accept Immediate, 1 minute, 5 minutes, 10 minutes, 15 minutes, 30 minutes, or 1 hour. Set Clear delay (minutes) together with Times Triggered to smooth out noisy metrics.
Notifications
Choose how alerts are delivered:
- Route via Notification Policies: route alerts by matching their labels against shared policies. Policies are evaluated top-down, and each matching policy processes the alert until a final policy ends routing.
- Select Specific Notification Target: send directly to a chosen target, bypassing policies.
To create and manage the Targets, Templates, and Policies that these options use, see Manage Monitor Notifications.
To suppress the Notifications of a Monitor on a recurring calendar, attach a mute schedule to the Monitor. See Attach a Mute Schedule to a Monitor.
Configure an Outlier Monitor
An Outlier Monitor compares the series that its query returns, and raises an alert for each series that behaves differently from its peers. A series is one line on the chart, identified by its labels. Each offending series gets its own alert with its own severity.
Use an Outlier Monitor to find the one host, pod, or region that drifts from comparable ones when no single threshold fits all of them. To create one, select Outlier as the Detection Method.
Cribl Search measures how far each series sits from the median of its peer group, relative to the typical spread among peers. That spread is based on the median absolute deviation (MAD). A series reaches a severity when it stays outside that severity’s band for long enough.
Write the Outlier Query
An Outlier Monitor scores a single query, and that query must return a separate series for each peer. For example, in
the visual builder, group the Function by host to return one series per host.
A query that returns a single line, such as sum(...) with no grouping, gives Cribl Search nothing to compare.
Set Outlier Options
In the Outlier Detection section, set the options that define the peer groups and how Cribl Search compares each series with its group:
- Evaluation mode: how Cribl Search builds the peer baseline. Instant (default) compares each series with its peers at each moment, so the expected range moves with the data. Windowed builds one baseline for each grading window, so the expected range changes in steps from window to window.
- Group peers by (optional): the labels that define peer groups. Cribl Search compares a series only with series
that have the same values for these labels, or with all series if you leave the field empty. For example, grouping by
regioncompares each host only with hosts in the same region. - Direction: which side of the peer median counts as outlying. Select above the peer median, below the peer median, or Either side of the peer median (default). Deviations on the other side of the median don’t count.
Expand Advanced Options to change how Cribl Search grades each series:
- Grading window: how much recent data Cribl Search evaluates for each series, from 5 minutes to 7 days. The default is 30 minutes. In Windowed mode, each peer baseline covers one grading window.
- Breach threshold %: the percentage of the grading window that a series must spend outside a severity’s band to reach that severity. Enter 1 to 100. The default is 20.
- Minimum coverage (%): the percentage of the grading window that must contain data before Cribl Search grades a series. Cribl Search expects one data point per minute. Enter 1 to 100. The default is 80.
Set Outlier Thresholds
For each severity, enter how far a series must deviate from its peer median to reach that severity. Enter the distance as a multiple of the typical peer spread, using the number box or the slider. The defaults are Critical at 3, Warning at 2, and Info at 1.
A series reaches a severity when it deviates by more than that multiple for at least the Breach threshold % of the grading window.
Preview an Outlier Monitor
The preview chart draws each series with dashed Critical Band, Warning Band, and Info Band lines around the Cohort median. Hover over the chart to see where each severity begins.
The preview shows the Critical, Warning, and Info thresholds for all series at once. Select a single series to see how that series performs against its peers.
If the query returns fewer than three series, the preview shows a warning, draws no bands, and marks no series as breaching.
The preview uses the same scoring as scheduled evaluation. It can’t draw thresholds that use overrides or Included Tags and Excluded Tags, and it shows a message that explains why.
Understand Outlier Limits
Cribl Search doesn’t raise an alert for a series in these cases:
- Its peer group has fewer than three series. Instant mode counts the series that have data at each moment, and Windowed mode counts the series that have data anywhere in the grading window. You can’t change this minimum.
- Its peer group has mostly identical values, so the typical spread is zero. Cribl Search then can’t measure a deviation, even for a series that differs.
- Its data covers less than the Minimum coverage (%) of the grading window.
If Cribl Search finds no ingested data for the metric, any alert that was firing changes to a no-data state. Cribl Search evaluates the series again after data arrives.
Mute a Monitor
Muting suppresses the Notifications of a Monitor during a planned window, such as scheduled maintenance, without disabling the Monitor or losing alert history. For Insights Alerts, see Muting Rules.
Cribl Search offers two kinds of muting:
- A silence mutes one Monitor for one window of time, such as the next two hours.
- A mute schedule mutes any Monitors that you attach it to on a recurring calendar, such as every weekend.
A Monitor is muted while it has an active silence, or while any mute schedule attached to it is active. While a Monitor is muted, it still evaluates and still records alert history. Cribl Search suppresses only the Notifications.
To manage both kinds, on the Cribl.Cloud top bar, select Products > Search > Monitors, then select the Muting Rules tab. Use the Silences and Mute schedules buttons at the top of the tab to switch between them.
Members with read-only access to Monitors can open silences and mute schedules, but the controls to add, edit, and delete them are hidden. See Monitor Permissions.
Create a Silence
The Silences view lists every silence with these columns:
| Column | Description |
|---|---|
| Status | Whether the silence is Active, Scheduled, or Expired. |
| Name | The name of the silence. Select it to open the silence. |
| Monitor | The Monitor that the silence mutes. Select it to open the Monitor. |
| Duration | How long the silence lasts, such as 2 hours, or its start and end times. |
| Created By | The user who created the silence. |
| Actions | Edit and Delete. |
Use All, Active, Scheduled, or Expired to filter the list, and use Filter to search by name or Monitor.
To create a silence:
- In the Silences view, select Add Muting Rule.
- In Name (optional), enter a name that describes why the silence exists.
- In Monitor, select the Monitor to mute. A silence applies to exactly one Monitor.
- In Description (optional), enter more detail.
- Under Duration, select one of the following:
- Relative to enter a number and a unit of Minutes, Hours, Days, or Weeks. The silence starts when you save it.
- Date Range to enter a Start and an End date and time.
- Select Create.
A silence with a Date Range that starts in the future has the status Scheduled.
Create a Mute Schedule
A mute schedule is a named, reusable set of recurring windows. You define it once, then attach it to as many Monitors as you need. The Mute schedules view lists each schedule with its Name, Windows, Description, and Created By. The Windows column summarizes each window in plain words, always with its time zone. When a schedule has more than one window, the column joins them with or.
To create a mute schedule:
- In the Mute schedules view, select Add Mute Schedule.
- In Name, enter a name for the schedule. This field is required.
- In Description (optional), enter what the schedule is for.
- Under Recurring windows, define one or more windows. See Define a Window.
- Select Create.
To change a schedule, select its name or its Edit action, then select Save. Changes apply to every Monitor that uses the schedule.
You can’t delete a schedule while a Monitor still uses it. Cribl Search rejects the deletion and names those Monitors. Detach the schedule from each of them first.
Define a Window
A schedule mutes while any of its windows matches. Within one window, every field that you set must match. A field that you leave empty matches anything. New windows start with Weekdays set to Saturday and Sunday, and with your browser’s time zone.
| Field | What it does |
|---|---|
| Weekdays | The days of the week that match. Select one or more. |
| Time ranges | One or more daily ranges, entered as Start and End times on a 24-hour clock, such as 09:00 and 17:00. Select Add Time Range for more. The start time is included and the end time is not. 24:00 means the end of the day. |
| Days of month | A comma-separated list of days, such as 1:5, -1. Use first:last for a range of days. A negative number counts back from the end of the month, so -1 is the last day. |
| Months | The months that match. Select one or more. |
| Years | A comma-separated list of years, such as 2026, 2027. |
| Time zone | The time zone that the other fields use. The default for a new window is your browser’s time zone. |
Below each window, a line that starts with Mutes summarizes what you entered. Select Add Window to add another window, or Remove to delete one. A schedule always keeps at least one window.
Cribl Search compares each window against the local clock time in its time zone, so a window follows daylight saving changes. An hour that daylight saving skips never matches, and an hour that it repeats matches both times.
If Cribl Search can’t resolve the time zone of a window or can’t read a time range, that window mutes nothing. A configuration mistake never hides alerts, but the Monitor keeps sending Notifications during that window.
For example, to mute a Monitor outside business hours, create two windows. In the first window, set Weekdays to
Monday through Friday, and add the time ranges 00:00 to 09:00 and 17:00 to 24:00. In the second window,
set Weekdays to Saturday and Sunday, and add no time range. The schedule mutes whenever either window
matches.
Attach a Mute Schedule to a Monitor
- On the Monitor page, find the Mute schedules field after the Notifications section.
- Select one or more schedules. Start typing to search by name.
- Select Save.
Select Manage mute schedules to go to the Mute schedules view. If Cribl Search can’t load the schedules, the field shows the error in place of its usual hint.
View Alert Activity
When you edit a Monitor, the Activity tab shows its history, so you can review when it fired and how it changed over time. The Alerts area also has an Activity tab across all Monitors.
Alert activity follows the same access rules as the Monitors list, so you see activity only for the Monitors you can access. History isn’t available for a deleted Monitor, because Cribl Search no longer has a configuration to check your access against.