Ingest Prometheus Metrics into Cribl Search
Preview Feature
Cribl is still developing this Preview feature. We don’t recommend using it in a production environment, because the feature might not be fully tested or optimized for performance, and related documentation could be incomplete.
Please continue to submit feedback through normal Cribl support channels, but assistance might be limited while the feature remains in Preview.
Collect metrics via the Prometheus Remote Write API to store them in Cribl Search for fast analysis.
Before You Begin
You’ll need:
- Cribl.Cloud Enterprise. For details, see Pricing.
- Search Admin Permission, or higher. Learn who can do what at Cribl Search Permissions.
- A Prometheus client that can reach Cribl Search over HTTP(S).
Prometheus Remote Write is currently the only supported way to ingest metrics into Cribl Search.
You don’t need Cribl Stream, Edge, or Lake. (Looking for the Prometheus Remote Write Source in Cribl Stream instead?)
To query your Prometheus instance without moving data into Cribl Search, see Connect Cribl Search to Prometheus.
1. Add a Lakehouse Engine
Metrics are stored on a Lakehouse Engine. See Lakehouse Engines in Cribl Search.
Metrics is a Preview feature. Adding metrics to a production Lakehouse Engine can affect its performance. Until the impact is better understood, use one of these approaches:
- On a production engine, keep the metrics volume small.
- Give metrics a dedicated engine, so their load doesn’t compete with production search. Route your metrics Source to that engine with Metric Rules.
2. Review the Metrics Dataset
When you create an engine, Cribl Search auto-provisions a metrics Dataset for it:
- The first engine’s Dataset is named
metrics. Each additional engine gets a Dataset namedmetrics_<engine_id>. - The Dataset is built-in, so you can’t delete or rename it.
- Its provider and type are
cribl_searchandmetrics. - The default retention is 365 days. You can change the retention in the Dataset configuration.
To review or configure the Dataset, on the Cribl.Cloud top bar select Products > Search > Data > Datasets, then select the metrics Dataset.
3. Add a Prometheus Remote Write Source in Cribl Search
Your metrics engine ships with a built-in Prometheus Remote Write Source (in_prometheus_rw) that listens on port
10092. You can point your Prometheus client at this Source directly, or add another Prometheus Remote Write Source -
for example, to send a second data stream to a different engine.
To add a Source, on the Cribl.Cloud top bar select Products > Search > Data > Add Source > Prometheus Remote Write.
Describe Your Source and Set the Endpoint
Under General, configure:
| Setting | Description | Example |
|---|---|---|
ID | Source ID, unique across your Cribl.Cloud Workspace. Use letters, numbers, underscores, hyphens. | prometheus_rw_prod |
| Description | Describe your Source so others know what it’s for. | Ingests Prometheus Remote Write metrics |
| Address | Hostname (FQDN) that your upstream sender connects to. You’ll need this to set up your upstream sender. | search.main.foo-bar-abc123.cribl.cloud |
| Port | Network port to listen on. The built-in metrics Source uses 10092. Change the default only if it conflicts with another service. | 10092 |
| Remote Write API endpoint | Base path on which to listen for Prometheus Remote Write API requests. | /api/v1/write (default) |
Set up Authentication
Use authentication to make sure only authorized senders can push data to your Cribl Search Source.
Under Authentication, select the Authentication type you want to use:
No authentication. Use only for testing or trusted internal networks.
Create a username and password. This is what your upstream sender will need to provide when sending data to your Source endpoint.
| Setting | Example |
|---|---|
| Username | prometheus_rw_user |
| Password | ******** |
Authenticate using a stored credentials secret instead of entering a username and password directly. This keeps credentials out of your Source configuration and makes them easier to rotate.
| Setting | Description | Example |
|---|---|---|
Credentials secret | Reference to a stored text secret that holds the credentials (username and password). Select a secret or Create a new one. (See Create and Manage Secrets in Cribl Stream). | sec_prometheus_rw_creds |
Create bearer tokens. This is what your upstream sender will need to provide in the authorization header.
Select
Add Token, then enter a token text or Generate a random one.
Authenticate using a stored token secret instead of entering a token text directly. This keeps tokens out of your Source configuration and makes them easier to rotate.
| Setting | Description | Example |
|---|---|---|
Token secret | Reference to a stored text secret that holds the token. Select a secret or Create a new one. (See Create and Manage Secrets in Cribl Stream). | sec_prometheus_rw_token |
Set Up Encryption
TLS encryption protects your data in transit between your upstream Prometheus client and the Cribl Search Source.
For the built-in metrics Source, TLS is enabled by default, with a minimum version of TLS 1.2. Cribl.Cloud provides and
manages the certificate and key, so you don’t need to supply your own. Because TLS is on by default, your Prometheus
client must connect over https.
Under Encrypt, you can review or adjust the Minimum TLS version you want to accept:
| TLS Version | When to Use |
|---|---|
| 1.3 | Provides the strongest security. Use when your clients support it. |
| 1.2 | The default. Use for broad client compatibility. |
| Older than 1.2 | Avoid if possible. These versions are no longer considered secure. |
Select Save to create the Source.
4. Set Up Datatyping
Datatyping applies to log data. If you’re only ingesting metrics, you can skip this step - metrics route to a metrics Dataset through Metric Rules instead. Configure Datatype rules to parse, filter, and normalize your data into structured fields. We call this process Datatyping.
On the Cribl.Cloud top bar, select Products > Search > Data > Datatyping (auto). Here, you can:
- Use Auto-Datatyping to parse your data automatically.
- Check for uncategorized data that didn’t match any Datatype rules.
- Handle the uncategorized data by adding custom Datatype rules.
See also:
- Datatypes in Cribl Search
- v2 Datatypes in Cribl Search
- List of Stock v2 Datatypes
- Add a Custom v2 Datatype
5. Route Metrics with Metric Rules
Metric Rules route incoming metrics to a metrics Dataset. On the Cribl.Cloud top bar, select Products > Search > Data > Dataset Rules, then select the Metric Rules tab.
By default, a single Metrics catch-all rule (*) sends all metrics to the primary metrics Dataset. To send a
Source to a different engine’s Dataset, add a rule that matches that Source and routes it to the target Dataset:
- Metric Rules match on the internal
__inputIdfield, which has the form<sourceType>:<sourceId>- for example,prometheus_rw:in_prometheus_rw. - Rules match top-down and the first match wins, so place more specific rules above the catch-all.
Metric Rules apply to data as it arrives and aren’t retroactive. Set up your engine, Source, and rule before you start sending data. Metrics that arrive before a matching rule exists fall through to the catch-all and stay in the primary
metricsDataset.
6. Set Up Your Prometheus Client
Configure your upstream Prometheus client to send data to your Cribl Search Source.
You’ll need these details from your Source configuration:
| Setting | Example |
|---|---|
| Address | search.main.foo-bar-abc123.cribl.cloud |
| Port | 10092 |
| Remote Write API endpoint | /api/v1/write (default) |
Example: Prometheus Remote Write > Cribl Search
Add a remote_write block to your prometheus.yml file, using the following example.
Replace the example address (search.main.foo-bar-abc123.cribl.cloud), username, password, endpoint, and port (if you
chose a different port) with your Source values.
remote_write:
- url: "https://search.main.foo-bar-abc123.cribl.cloud:10092/api/v1/write"
basic_auth:
username: "your_username"
password: "********"The built-in metrics Source uses TLS, so keep https. Use http only if you disabled TLS on your Source.
Replace the example address (search.main.foo-bar-abc123.cribl.cloud), token, endpoint, and port (if you changed the
default 10092) with your Source values.
remote_write:
- url: "https://search.main.foo-bar-abc123.cribl.cloud:10092/api/v1/write"
bearer_token: "420"The built-in metrics Source uses TLS, so keep https. Use http only if you disabled TLS on your Source.
7. Start Sending Data and Verify
Start sending events from your upstream Prometheus client, and verify that they’re successfully flowing into Cribl Search.
On the Cribl.Cloud top bar, select Products > Search > Data > Live Data.
Here, check for your Prometheus Remote Write Source. For details, see Live Data.
Read Metrics from External Tools
Each metrics Dataset exposes a Prometheus-compatible query endpoint that Prometheus-compatible tools such as Grafana can read from. The endpoint has this form:
https://<cribl-host>/api/v1/products/lakehouse_engine_metrics/engines/<engine>/datasets/<dataset>/prom/Find this endpoint, labeled Prometheus compatible endpoint, in the metrics Dataset configuration. Control access to it with the Dataset’s Query API Access authentication, which can be None or Basic (username and password).
Cribl strongly recommends setting Query API Access to Basic and configuring a username and password to lock down this endpoint. If you leave it set to None, the endpoint is completely open - anyone who can reach it can read your metrics, even if you never connect Grafana or another tool. Always set up Basic authentication to prevent unauthorized access.
Next Steps
Now that your metrics are in Cribl Search, you can start using them. For example: