Ingest Raw HTTP Data into Cribl Search
Collect HTTP data in any format to store it in Cribl Search for fast analysis.
Before You Begin
You’ll need:
- Cribl.Cloud Enterprise. For details, see Pricing.
- Search Admin Permission, or higher. Learn who can do what at Cribl Search Permissions.
You don’t need Cribl Stream, Edge, or Lake. (Looking for the Raw HTTP/S Source in Cribl Stream instead?)
1. Add a Lakehouse Engine
See Lakehouse Engines in Cribl Search.
2. Set Up Your Search Datasets
Create the Search Datasets you’ll route events into, and set their retention. See Create Search Datasets.
3. Add a Raw HTTP Source in Cribl Search
On the Cribl.Cloud top bar, select Products > Search > Data > Add Source > Raw HTTP.
Describe Your Source
Under General, configure:
| Setting | Description | Example |
|---|---|---|
| ID | Source ID, unique across your Cribl.Cloud Workspace. Use letters, numbers, underscores, hyphens. | raw_http_prod |
| Description | Describe your Source so others know what it’s for. | Ingests raw HTTP requests |
| Address | Hostname (FQDN) that your HTTP client connects to. You’ll need this to set up your HTTP client. | search.main.foo-bar-abc123.cribl.cloud |
| Port | Network port to listen on. Keep the default unless it conflicts with another service. | 20000 |
Set up Authentication
Use authentication to make sure only authorized HTTP clients can send data to your Cribl Search Source.
Under Authentication, select Add Token. Add as many tokens as you need.
| Setting | Description | Example |
|---|---|---|
| Token | Authentication string you’ll need to set up your HTTP client. Enter a token text, or select Generate for a random one. | 420 |
| Description | Describe which clients or environments use the token. | Prod HTTP |
| Fields | Extra fields to add to each event referencing this token. Use this to stamp events with metadata that you can later search by. In Value, enclose JavaScript expressions in backticks ` `. | Field name: fullSourceValue: `${host}:${source}` |
Set Up Encryption
Use TLS encryption to protect your data in transit between upstream HTTP clients and your Cribl Search Source.
Under Encrypt, select Enabled, and set the Minimum TLS version you want to accept.
| TLS Version | When to Use |
|---|---|
| 1.3 | Recommended. Provides the best security. |
| 1.2 | Use only when connecting to older systems that don’t support TLS 1.3. |
| Older than 1.2 | Avoid if possible. These versions are no longer considered secure. |
Select Save to create the Source.
4. Set Up Datatyping
Configure Datatype rules to parse, filter, and normalize your data into structured fields. We call this process Datatyping.
On the Cribl.Cloud top bar, select Products > Search > Data > Datatyping (auto). Here, you can:
- Use Auto-Datatyping to parse your data automatically.
- Check for uncategorized data that didn’t match any Datatype rules.
- Handle the uncategorized data by adding custom Datatype rules.
See also:
- Datatypes in Cribl Search
- v2 Datatypes in Cribl Search
- List of Stock v2 Datatypes
- Add a Custom v2 Datatype
5. Set Up Dataset Rules
Configure Dataset rules to route the parsed events into your Search Datasets.
On the Cribl.Cloud top bar, select Products > Search > Data > Datasets: Organize Your Data, and see Organize Data with Dataset Rules for details.
6. Set Up Your HTTP Client
Configure your upstream HTTP client to send data to your Cribl Search Source.
You’ll need these details from your Source configuration:
| Name | Example |
|---|---|
| Address | search.main.foo-bar-abc123.cribl.cloud |
| Port | 20000 |
| Auth Token | 420 |
To find these for an existing Source: On the Cribl.Cloud top bar, select Products > Search > Data > Sources, and select your Source.
Example Request (Raw HTTP > Cribl Search)
Replace the example address (search.main.foo-bar-abc123.cribl.cloud) and port (if you changed the default 20000) with
your Source values.
curl -k "https://search.main.foo-bar-abc123.cribl.cloud:20000" \
-H "Authorization: your_token" \
-d '{"_raw":"this is a sample event"}'Without TLS, use http instead of https.
curl -k "https://search.main.foo-bar-abc123.cribl.cloud:20000" \
-H "Authorization: your_token" \
-d $'{"_raw":"this is a sample event", "host":"myHost", "source":"mySource", "fieldA":"valueA", "fieldB":"valueB"}\n\
{"_raw":"this is another sample event", "host":"myOtherHost", "source":"myOtherSource", "fieldA":"valueA", "fieldB":"valueB"}'Without TLS, use http instead of https.
7. Start Sending Data
Start sending events from your HTTP client, and verify that they’re successfully flowing into Cribl Search.
On the Cribl.Cloud top bar, select Products > Search > Data > Live Data.
Here, check for your Raw HTTP Source. For details, see See Live Data Flow.
Next Steps
Now that your data is in Cribl Search, you can start using it. For example: