Ingest Cribl Stream/Edge Data into Cribl Search
Collect data from your on-prem or Cribl.Cloud tenants to store it in Cribl Search for fast analysis.
Before You Begin
You’ll need:
- A lakehouse engine. See how to get one at lakehouse engines.
- Search Admin Permission, or higher. Learn who can do what at Cribl Search Permissions.
- Cribl Stream or Cribl Edge (on-prem or Cribl.Cloud).
- Cribl.Cloud Enterprise on both sides (Cribl Stream/Edge and Cribl Search). For details, see Pricing.
Looking for the Cribl HTTP Source in Cribl Stream instead?
1. Add a Cribl HTTP Source in Cribl Search
On the Cribl.Cloud top bar, select Products > Search > Data > Add Source > Cribl HTTP.
Describe Your Source
Under General, configure:
| Setting | Description | Example |
|---|---|---|
| ID | Source ID, unique across your Cribl.Cloud Workspace. Use letters, numbers, underscores, hyphens. | cribl_stream_prod |
| Description | Describe your Source so others know what it’s for. | Ingests data from Cribl Stream |
| Address | Hostname (FQDN) that Cribl Stream/Edge connects to (Cribl endpoint). | search.main.foo-bar-abc123.cribl.cloud |
| Port | Network port to listen on. Keep the default unless it conflicts with another service. | 10200 (default) |
Set up Authentication (Optional)
For cross-environment or cross-Workspace paths (for example, on-prem Cribl Stream to Cribl Search), you’ll need to set up authentication tokens.
Under Authentication, select Add Token. Add as many tokens as you need.
| Setting | Description | Example |
|---|---|---|
Token secret (text secret) | Reference to a stored secret containing the token. Select a secret or Create a new one. (See Create and Manage Secrets in Cribl Stream). | sec_cribl_stream_token |
| Description | Describe which clients or environments use the token. | Prod Cribl Stream |
Set Up Encryption
Use TLS encryption to protect your data in transit.
TLS must be enabled on both sides (Source and Destination), or disabled on both sides.
Under Encrypt, select Enabled, and set the Minimum TLS version you want to accept.
| TLS Version | When to Use |
|---|---|
| 1.3 | Recommended. Provides the best security. |
| 1.2 | Use only when connecting to older systems that don’t support TLS 1.3. |
| Older than 1.2 | Avoid if possible. These versions are no longer considered secure. |
Select Save to create the Source.
2. Assign Datatypes in Cribl Stream
When data comes from Cribl Stream or Cribl Edge, you can assign Datatypes upstream and skip Datatyping in Cribl Search.
Set the datatype override field on events in Cribl Stream or Cribl Edge to one of the
stock v2 Datatypes, or to a custom v2 Datatype you defined in Search.
The easiest way is to add an Eval function to the Pipeline that feeds your
Cribl Search Destination, and set datatype there.
If you’d rather configure Datatyping in Cribl Search instead, see Shape Your Data with Datatype Rules.
3. Route Events to Search Datasets
Create the Search Datasets you want to route into, then assign each event to a Dataset upstream by setting the dataset
override field in Cribl Stream or Cribl Edge. Search uses that value directly and skips Dataset
rules.
- In Cribl Search, add your Search Datasets and set their retention periods.
- In Cribl Stream or Cribl Edge, set the
datasetfield on each event to the ID of the target Search Dataset. Use an Eval function in the Pipeline that feeds your Cribl Search Destination.
Events with no dataset field, or with a dataset field pointing to a Dataset that doesn’t exist, fall back to the
main Dataset.
If you’d rather route events with Dataset rules in Cribl Search, see Organize Your Data with Dataset Rules.
4. Add a Cribl Search Destination in Cribl Stream or Cribl Edge
For Cribl Stream, see Cribl Search Destination in the Stream docs.
For Cribl Edge, see Cribl Search Destination in the Edge docs.
5. See Live Data Flow
Verify that events are successfully flowing from your HTTP client into Cribl Search.
On the Cribl.Cloud top bar, select Products > Search > Data > Live Data.
Here, check for your Cribl HTTP Source. For details, see See Live Data Flow.
Next Steps
Now that your data is in Cribl Search, you can start using it. For example: