Once you’ve created your Search Datasets, set Dataset rules to control which data lands in which Dataset.

Highlights ​

• Each Log Dataset rule routes matching log events to one Log Dataset that you created.
• Each Metric Dataset rule routes matching metrics to a metrics Dataset, from a catch-all rule down.
• Verify Dataset assignment to make sure your data lands where you expect.

Dataset Rules Overview ​

Each Dataset rule captures data that matches a KQL expression, and then sends that data to a Search Dataset.

Dataset rules come in two types:

Within each tab, rules run top-down and the first match wins. Most of this page covers Log Dataset rules. For what differs about metrics, see Add Metric Dataset Rules.

To manage your Dataset rules: from the Cribl.Cloud top bar, select Products > Search > Data > Get Data In > 3. Dataset Rules.

The Log Rules and Metric Rules tabs appear only if your Organization has the Metric Rules Preview enabled. Otherwise, the Dataset Rules page lists your Log Dataset rules without tabs.

Add Log Dataset Rules ​

To add a new Log Dataset rule:

1. On the Cribl.Cloud top bar, select Products > Search > Data > Get Data In > 3. Dataset Rules.

   If the page shows tabs, select Log Rules. Labels and buttons then read Log Dataset Rule instead of Dataset Rule.

2. Select Add Dataset Rule. Name and describe your rule.
3. In Kusto expression to match, enter a KQL expression that matches the log events you want to route.

   See Log Dataset Rule Expressions for syntax and examples.

4. In Send data to, choose your target Log Dataset. This is where events matching the KQL expression will land.

   You can also select Drop to instead discard the events.

5. In Modify fields, you can perform Dataset-specific enrichment or normalization. See Modify Fields in Dataset Rules.
6. In Map fields to schema, you can rename vendor fields to a standard schema such as OCSF. See Map Fields to a Schema.
7. Make sure that Enabled in the top right corner is checked, and confirm with Add.

If you add more rules, drag them to change the order. Rules run top-down, and the first match wins. Put more specific rules above broader ones.

Events that don’t match any rule, or match a rule pointing to a deleted Dataset, fall back to the main Dataset.

Log Dataset Rule Expressions ​

Point your Log Dataset rule KQL expressions at these fields:

You can copy __inputId or other fields from the arriving events. To see them, select Live Data to sample incoming events.

You can also filter by any other field in your parsed data.

Same as with Datatype rule expressions, you can:

• Create KQL expressions that evaluate to true/false for matching events.
• Set case-insensitive conditions using = and wildcards (*).
• Pipe into | where ..., | find ..., or | search ... for richer logic.

But:

• You can’t use expressions that aggregate or reshape data (such as stats or project).
• You can’t use let or set statements.

Examples of Log Dataset Rule Expressions ​

See the examples below. For full reference on the Cribl Search implementation of KQL, see the KQL Reference.

datatype = "apache_httpd_accesslog_common"

datatype = "syslog_rfc3164" and __inputId = "syslog:my_source_id"

datatype = "aws_vpc_v2" | where host = "vpc-flow-logs"

__inputId = "open_telemetry:otel"

datatype in ("healthcheck", "heartbeat")

Verify Dataset Assignment ​

Check on your Log Datasets to make sure your log events are routed and retained as expected.

1. Go to Logs: On the Cribl.Cloud top bar, select Products > Search.
2. Under Available Datasets, select a Log Dataset you want to inspect.

   Log Datasets are marked with the lakehouse icon .

3. In the resulting details panel, look at the Fields section.

   If the Fields section is empty, select Retry to load the metadata.

   If there’s no Fields section at all, you’re looking at a federated Dataset. Select a Log Dataset instead.

4. Verify that the Dataset contains the fields you’d expect from your Datatyping configuration and Log Dataset rules.
   For more information, see Explore Fields in Log Datasets.

For more ways to explore your Datasets, see Inspect Your Log Datasets. To check where your metrics landed, select a metrics Dataset in the Metrics Explorer.

Modify Fields in Dataset Rules ​

You can enrich or normalize events on their way into a specific Search Dataset, without affecting your upstream Datatyping rules. Use this to normalize timestamps, convert units, or reshape values, while preserving the Auto-Datatyping flow.

1. On the Cribl.Cloud top bar, select Products > Search > Data > Get Data In > 3. Dataset Rules.
2. Select Add Dataset Rule (or edit an existing one). For details, see Add Log Dataset Rules.
3. Select Modify fields.
4. In the text box, write a KQL extend expression to add or overwrite fields on matched events.

See the examples below. For full reference on the Cribl Search implementation of KQL, see the KQL Reference.

_time = case(host startswith "eastcoast", _time + 3h, host startswith "westcoast", _time + 6h, _time)

size_mb = bytes / 1048576.0

priority = case(level == "CRITICAL" or level == "FATAL", "high", level == "ERROR" or level == "WARN", "medium", "low")

Map Fields to a Schema ​

You can map fields in your log events to a standard schema (such as OCSF v1.8) as they enter a Search Dataset. This lets you keep common field names across data sources.

For example, you can make a source IP address always land in src_endpoint.ip, whatever the original vendor called it.

Why Map to a Schema ​

Vendors name the same values differently. For example, the source IP address is source_ip in Palo Alto traffic logs, srcaddr in AWS VPC Flow Logs, and srcip in Fortinet FortiGate logs. With a standard schema, you can:

• Write one search, Dashboard, or alert that works across all your vendors, instead of one per Datatype.
• Switch or add a vendor without rewriting your queries.
• Use the same field names as other tools that support OCSF, such as your SIEM.

Use schema maps to normalize data to a standard schema. Use Modify fields for one-off changes that apply to a single Dataset rule.

Datatype Level and Dataset Level ​

Schema mapping has two parts, configured in two places:

A schema map does nothing until a Dataset rule applies it. To change a stock map or add one to a custom Datatype, see Set Schema Maps.

How Schema Mapping Works ​

Keep these things in mind when you map fields to a schema:

• Cribl Search maps fields after it applies Modify fields, so your schema map can use fields that the Dataset rule adds.
• Schema maps never change or remove _time, _raw, dataset, datatype, source, or internal fields that start with __.
• If an event’s Datatype has no map for the selected schema, Cribl Search stores the event without mapping it.
• Mapping applies to data as it arrives and isn’t retroactive.
• Map fields to schema also appears in Metric Dataset rules, but has no effect there, because metrics don’t have Datatypes.
• Unlike Modify fields, a schema map is reusable across rules and starts from the required OCSF fields.

Apply a Schema Map ​

To map fields in a Dataset rule:

1. On the Cribl.Cloud top bar, select Products > Search > Data > Get Data In > 3. Dataset Rules.

2. Select Add Dataset Rule (or edit an existing one). For details, see Add Log Dataset Rules.

3. In Map fields to schema, select a mode:

   Mode	What Cribl Search Does	Use When
   None	Doesn’t map fields. This is the default.	You don’t need a standard schema for this data.
   Add	Adds the schema fields and keeps the original fields. If names clash, the original value moves to a data_ field (for example, data_time).	Your existing searches, Dashboards, or alerts still use vendor field names, or you’re moving them to OCSF.
   Replace	Adds the schema fields and removes the original fields that the map copies as is.	You query by OCSF field names only, and don’t want your events to store the same value twice.

4. In Schema, select OCSF v1.8.

5. Confirm with Add, or with OK if you’re editing an existing rule.

To preview the mapped fields before you confirm your changes to an existing rule, select Live Data in the rule editor. To check mapped fields later, use Live Data with the After Dataset detection capture level.

Override Dataset Rules ​

To bypass Dataset rules, add a dataset field to your log events before they reach Cribl Search. You can add this and other override fields in Cribl Stream, or in any upstream sender.

Add Metric Dataset Rules ​

Preview Feature ​

Cribl is still developing this Preview feature. We don’t recommend using it in a production environment, because the feature might not be fully tested or optimized for performance, and related documentation could be incomplete.

Please continue to submit feedback through normal Cribl support channels, but assistance might be limited while the feature remains in Preview.

Metric Dataset rules are the metrics counterpart to Log Dataset rules: they match incoming metrics and route them to a metrics Dataset. To manage them, select Products > Search > Data > Get Data In > 3. Dataset Rules on the Cribl.Cloud top bar, then select the Metric Rules tab.

By default, a single Metrics catch-all rule (*) sends all metrics to the primary metrics Dataset. To send a Source to a different engine’s Dataset, add a rule that matches that Source and routes it to the target Dataset:

• Metric Dataset rules match on the internal __inputId field, which has the form <sourceType>:<sourceId> - for example, prometheus_rw:in_prometheus_rw for a Prometheus Remote Write Source, open_telemetry:otel_prod for an OpenTelemetry Source that stores metrics, datadog_agent:datadog_agent_prod for a Datadog Agent Source that stores metrics, or cribl_http:in_cribl_http for a Cribl HTTP Source that receives native metrics from Cribl Stream or Cribl Edge.
• Rules run top-down and the first match wins, so place more specific rules above the catch-all.

Otherwise, Metric Dataset rules work like Log Dataset rules, with two differences:

• You can’t match on the datatype field, because Datatyping doesn’t apply to metrics.
• Send data to offers only metrics Datasets, one per lakehouse engine.

Metric Dataset rules apply to data as it arrives and aren’t retroactive. Set up your engine, Source, and rule before you start sending data. Metrics that arrive before a matching rule exists fall through to the catch-all and stay in the primary metrics Dataset.

To learn which Sources can store metrics, see Ingest Prometheus Metrics into Cribl Search, Ingest OpenTelemetry Data into Cribl Search, Ingest Datadog Agent Data into Cribl Search, and Ingest Cribl Stream/Edge Data into Cribl Search.

Next Steps ​

Now that your data is organized into Datasets, capture a sample of incoming events to verify they arrive and route as expected. See View Live Data in Cribl Search.