On This Page

Home / Cribl as Code/ Cribl API/ API Code Examples/Configure Worker Groups with the Cribl API

Configure Worker Groups with the Cribl API
​

These code examples demonstrate how to use the Cribl API to configure, scale, replicate, and deploy Worker Groups in Cribl Stream.

About the Code Examples
​

The code examples use Bearer token authentication. Read the API authentication documentation to learn how to configure authentication. The Permissions granted to your Bearer token must include creating and managing Worker Groups.

Replace the variables in the examples with the corresponding information for your Cribl deployment.

For on-prem deployments, to use https in the URLs, you must configure Transport Layer Security (TLS).

The configurations in the examples do not include all available body parameters. For a complete list of body parameters for each endpoint, refer to the documentation in the API Reference.

The examples for creating and scaling a Worker Group on Cribl.Cloud include the estimatedIngestRate property, which allows you to configure Worker Groups for optimal performance. For each supported estimatedIngestRate value, the following table maps the corresponding throughput and number of Worker Processes:

estimatedIngestRateThroughputWorker Processes
102412 MB/s6
204824 MB/s9
307236 MB/s14
409648 MB/s21
512060 MB/s30
716884 MB/s45
10240120 MB/s62
13312156 MB/s93
15360180 MB/s186

1. Create a Worker Group
​

This example creates a new Worker Group in Cribl Stream.

In the Cribl.Cloud example, the estimatedIngestRate is set to 2048, which is equivalent to a maximum of 24 MB/s with nine Worker Processes. In the on-prem deployment example, the new Worker Group is created with eight Workers.

API (Cribl.Cloud)API (On-Prem)
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "id": "my-worker-group",
  "name": "my-worker-group",
  "description": "Cribl.Cloud Worker Group",
  "cloud": {
    "provider": "aws",
    "region": "us-west-2"
  },
  "workerRemoteAccess": true,
  "isFleet": false,
  "isSearch": false,
  "onPrem": false,
  "estimatedIngestRate": 2048,
  "provisioned": false
}'
curl --request POST \
--url "https://${hostname}:${port}/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "id": "my-worker-group",
  "name": "my-worker-group",
  "description": "My Worker Group description",
  "workerRemoteAccess": true,
  "isFleet": false,
  "isSearch": false,
  "onPrem": true,
  "workerCount": 8
}'

2. Scale the Worker Group
​

The Cribl.Cloud example scales the Worker Group to an estimatedIngestRate of 4096, which is equivalent to a maximum of 48 MB/s with 21 Worker Processes. The example also sets provisioned to true to activate Cribl.Cloud resources.

The on-prem example assumes the Syslog Source load balancer (LB) is enabled on a Cribl Stream system with 6 physical cores hyperthreaded (12 vCPUs). Because the Syslog Source LB reserves an additional core, the example scales the process count from the default to -3: two cores for system/API overhead plus one for the LB process, so the system spawns nine Worker Processes. For more information, see Optimize a Distributed Deployment or Hybrid Group and Choose a Process Count. The on-prem example also updates the Worker Group system settings, commits and deploys the changes, and restarts the Worker Group to apply the changes.

The request bodies for the PATCH /products/{product}/groups/{id} (Cribl.Cloud) and PATCH /system/settings/conf (on-prem) endpoints must include a complete representation of the Worker Group or settings, respectively, that you want to update. These endpoints do not support partial updates. Cribl removes any omitted fields when updating the Worker Group or settings.

API (Cribl.Cloud)API (On-Prem)
curl --request PATCH \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups/my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "id": "my-worker-group",
  "name": "my-worker-group",
  "description": "Cribl.Cloud Worker Group",
  "cloud": {
    "provider": "aws",
    "region": "us-west-2"
  },
  "workerRemoteAccess": true,
  "isFleet": false,
  "isSearch": false,
  "onPrem": false,
  "estimatedIngestRate": 4096,
  "provisioned": true
}'
curl --request PATCH \
--url "https://${hostname}:${port}/api/v1/products/stream/groups/my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "api": {
    "host": "0.0.0.0",
    "port": 1234,
    "ssl": {
      "disabled": false,
      "certificateName": "myCertificate",
      "certPath": "$CRIBL_HOME/local/cribl/auth/certs/myCertificate.crt",
      "privKeyPath": "$CRIBL_HOME/local/cribl/auth/certs/myCertificate.key"
    }
  },
  "system": {
    "upgrade": "api",
    "intercom": true
  },
  "workers": {
    "count": -3,
    "memory": 2048,
    "minimum": 2    
  },
  "proxy": {
    "useEnvVars": true
  },
  "upgradeSettings": {
    "disableAutomaticUpgrade": true,
    "enableLegacyEdgeUpgrade": false
  },
  "shutdown": {
    "drainTimeout": 10
  }
}'

3. Replicate the Worker Group
​

This example creates a replica Worker Group in Cribl Stream by cloning an existing Worker Group configuration. The new Worker Group uses the sourceGroupId body parameter to specify the Worker Group to clone. The replica Worker Group inherits the configuration from the source Worker Group, including settings and resources like Sources, Destinations, and Pipelines.

To run this example, you must have at least one existing Worker Group named my-worker-group in Cribl Stream to use as the source Worker Group.

API (Cribl.Cloud)API (On-Prem)
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "id": "my-replica-worker-group",
  "name": "my-replica-worker-group",
  "description": "Worker Group cloned from my-worker-group with identical configuration",
  "cloud": {
    "provider": "aws",
    "region": "us-west-2"
  },
  "workerRemoteAccess": true,
  "isFleet": false,
  "isSearch": false,
  "onPrem": false,
  "sourceGroupId": "my-worker-group"
}'
curl --request POST \
--url "https://${hostname}:${port}/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "id": "my-replica-worker-group",
  "name": "my-replica-worker-group",
  "description": "Worker Group cloned from my-worker-group with identical configuration",
  "workerRemoteAccess": true,
  "isFleet": false,
  "isSearch": false,
  "onPrem": true,
  "sourceGroupId": "my-worker-group"
}'

4. Confirm the Worker Group Configuration
​

Use this example to retrieve a list of all Worker Groups in Cribl Stream so that you can review and confirm their configurations.

API (Cribl.Cloud)API (On-Prem)
curl --request GET \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json"
curl --request GET \
--url "https://${hostname}:${port}/api/v1/products/stream/groups" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json"

5. Commit and Deploy the Worker Group
​

This example demonstrates how to commit and deploy the Worker Group configurations, then commit to the Leader to keep it in sync with the Worker Groups. You can commit and deploy immediately after a single create or update request or after multiple requests.

On-prem deployments also require restarting the Worker Groups if you want their new configurations to take effect.

First, commit pending changes to the Worker Group. This request commits only the configuration changes for the Worker Group by specifying the file local/cribl/groups.yml. Repeat this request for each Worker Group whose changes you want to commit.

API (Cribl.Cloud)API (On-Prem)
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/version/commit?groupId=my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "files": [
    "local/cribl/groups.yml"
  ],
  "message": "Commit Worker Group configuration"
}'
curl --request POST \
--url "https://${hostname}:${port}/api/v1/version/commit?groupId=my-worker-group" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "files": [
    "local/cribl/groups.yml"
  ],
  "message": "Commit Worker Group configuration"
}'

Second, deploy the committed changes to the Worker Group. This request includes the version body parameter, which uses the value of commit from the response body for the commit request. Repeat this request for each Worker Group whose committed changes you want to deploy.

API (Cribl.Cloud)API (On-Prem)
curl --request PATCH \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/products/stream/groups/my-worker-group/deploy" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "version": 1234abcd5678efgh9012ijkl3456mnop7EXAMPLE
}'
curl --request PATCH \
--url "https://${hostname}:${port}/api/v1/products/stream/groups/my-worker-group/deploy" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "version": 1234abcd5678efgh9012ijkl3456mnop7EXAMPLE
}'

Commit the changes to the Leader to keep the Leader in sync with the Worker Group.

API (Cribl.Cloud)API (On-Prem)
curl --request POST \
--url "https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/version/commit" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "message": "Sync my-worker-group with Leader"
}'
curl --request POST \
--url "https://${hostname}:${port}/api/v1/version/commit" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json" \
--data '{
  "message": "Sync my-worker-group with Leader"
}'

Finally, for on-prem deployments only, restart the Worker Group. Repeat this request for each Worker Group whose configuration you want to take effect.

API (On-Prem)
curl --request POST \
--url "https://${hostname}:${port}/api/v1/m/my-worker-group/system/settings/restart" \
--header "Authorization: Bearer ${token}" \
--header "Content-Type: application/json"