Configure Resources with the Cribl API
These code examples demonstrate how to use the Cribl API to create the following resources in Cribl Stream:
- A Syslog Source to receive data on port 9021.
- An S3 Destination to store processed data.
- A Pipeline that filters events and keeps only data in the
eventSourceandeventIDfields. - A Route that connects the Source, Pipeline, and Destination.
The examples also include deploying the resource configurations to a Worker Group to make them active. The examples use the Worker Group created in Configure Worker Groups to manage the resource configuration.
About Code Examples
Code examples use Bearer token authentication. Read the API authentication documentation to learn how to configure authentication. The API Credential (Cribl.Cloud and hybrid) or login credentials (on-prem) that you use to obtain the Bearer token must have the necessary Permissions for the operations in code examples.
Replace the variables in code examples with the corresponding information for your Cribl deployment.
For on-prem deployments, to use
httpsin request URLs, you must configure Transport Layer Security (TLS).Code examples do not include all available body parameters. For a complete list of body parameters for specific endpoints, refer to the documentation in the API Reference.
Create a Source
This example creates a Syslog Source to receive data on port 9021.
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/my-worker-group/system/inputs" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "in_syslog_9021",
"type": "syslog",
"disabled": true,
"tcpPort": 9021,
"host": "192.168.1.100"
}'curl --request POST \
--url "https://${hostname}:${port}/api/v1/m/my-worker-group/system/inputs" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "in_syslog_9021",
"type": "syslog",
"disabled": true,
"tcpPort": 9021,
"host": "192.168.1.100"
}'Create a Destination
This example creates an S3 Destination to store processed data. Replace placeholder values like your-aws-api-key before you run the example.
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/my-worker-group/system/outputs" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "out_s3",
"type": "s3",
"awsAuthenticationMethod": "manual",
"awsApiKey": "your-aws-api-key",
"awsSecretKey": "your-aws-secret-key",
"region": "us-east-2",
"bucket": "your-aws-bucket-name",
"compress": "gzip",
"compressionLevel": "best_speed",
"stagePath": "$CRIBL_HOME/state/outputs/staging",
"emptyDirCleanupSec": 300
}'curl --request POST \
--url "https://${hostname}:${port}/api/v1/m/my-worker-group/system/outputs" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "out_s3",
"type": "s3",
"awsAuthenticationMethod": "manual",
"awsApiKey": "your-aws-api-key",
"awsSecretKey": "your-aws-secret-key",
"region": "us-east-2",
"bucket": "your-aws-bucket-name",
"compress": "gzip",
"compressionLevel": "best_speed",
"stagePath": "$CRIBL_HOME/state/outputs/staging",
"emptyDirCleanupSec": 300
}'Create a Pipeline
This example creates a Pipeline that filters events and keeps only data in the eventSource and eventID fields.
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/my-worker-group/pipelines" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "my_pipeline",
"conf": {
"asyncFuncTimeout": 1000,
"functions": [
{
"filter": "true",
"conf": {
"remove": [
"*"
],
"keep": [
"eventSource",
"eventID"
]
},
"id": "eval",
"final": true
}
]
}
}'curl --request POST \
--url "https://${hostname}:${port}/api/v1/m/my-worker-group/pipelines" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "my_pipeline",
"conf": {
"asyncFuncTimeout": 1000,
"functions": [
{
"filter": "true",
"conf": {
"remove": [
"*"
],
"keep": [
"eventSource",
"eventID"
]
},
"id": "eval",
"final": true
}
]
}
}'Create a Route
This example creates a Route that connects the Syslog Source, Pipeline, and S3 Destination and adds it to the end of the default Routing table. The Routing table already includes a default Route, so this request changes the default Route’s setting for final to false so that it won’t block the new Route.
The
PATCH /routes/defaultendpoint requires a complete representation of the Routing table and its existing Routes in the request body. This endpoint does not support partial updates. Cribl removes any omitted fields when updating the Routing table. UseGET /routes/defaultto retrieve the existing Routing table to use in the body of the PATCH request.
curl --request PATCH \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/my-worker-group/routes/default" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "default",
"routes": [
{
"id": "0abcd9",
"name": "default",
"final": false,
"disabled": false,
"pipeline": "main",
"enableOutputExpression": false,
"filter": "true",
"output": "default"
},
{
"name": "my_route",
"final": true,
"disabled": false,
"pipeline": "my_pipeline",
"enableOutputExpression": false,
"filter": "__inputId == 'syslog:in_syslog_9021:tcp'",
"output": "out_s3",
"description": "This is my new Route"
}
]
}'curl --request PATCH \
--url "https://${hostname}:${port}/api/v1/m/my-worker-group/routes/default" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"id": "default",
"routes": [
{
"id": "0abcd9",
"name": "default",
"final": false,
"disabled": false,
"pipeline": "main",
"enableOutputExpression": false,
"filter": "true",
"output": "default"
},
{
"name": "my_route",
"final": true,
"disabled": false,
"pipeline": "my_pipeline",
"enableOutputExpression": false,
"filter": "__inputId == 'syslog:in_syslog_9021:tcp'",
"output": "out_s3",
"description": "This is my new Route"
}
]
}'Commit and Deploy the Resource Configurations
This example demonstrates how to commit and deploy the resource configurations to your Worker Group, then commit to the Leader to keep it in sync with the Worker Group.
Committing and deploying the Worker Group configuration requires three requests: commit to the Worker Group, deploy to the Worker Group, and commit the the Leader to keep it in sync.
First, commit the pending resource configurations to the Worker Group:
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/version/commit?groupId=my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"message": "Commit resource configurations to my-worker-group"
}'curl --request POST \
--url "https://${hostname}:${port}/api/v1/version/commit?groupId=my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"message": "Commit resource configurations to my-worker-group"
}'Next, deploy the committed changes to the Worker Group. This request includes the version body parameter, which uses the value of commit from the response body for the commit request:
curl --request PATCH \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups/my-worker-group/deploy" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"version": 1234abcd5678efgh9012ijkl3456mnop7EXAMPLE
}'curl --request PATCH \
--url "https://${hostname}:${port}/api/v1/products/stream/groups/my-worker-group/deploy" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"version": 1234abcd5678efgh9012ijkl3456mnop7EXAMPLE
}'Finally, commit the changes to the Leader to keep the Leader in sync with the Worker Group:
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/version/commit" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"message": "Sync my-worker-group resource configurations with Leader"
}curl --request POST \
--url "https://${hostname}:${port}/api/v1/version/commit" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
"message": "Sync my-worker-group resource configurations with Leader"
}