On This Page

Home / API/ Workflows/Create and Update Lookups

Create and Update Lookups

Lookups provide a flexible way to enrich events en route to their Destinations. This topic demonstrates how to use the Cribl API to upload Lookup files, create Lookups, and deploy updated Lookup files to Worker Nodes.

About the Example Requests
​

Replace the variables in the example requests with the corresponding information for your Cribl deployment. In the cURL command options, replace ${token} with a valid API Bearer token. You can also set the $token environment variable to match the value of a Bearer token.

For customer-managed deployments, to use https in the URL for your requests as shown in these examples, you must configure Transport Layer Security (TLS).

In Cribl.Cloud and other distributed deployments, you must commit and deploy the changes you make. You can use the Cribl API to automate commit and deploy commands.

Create a Lookup
​

Follow these steps to use the Cribl API to create a Lookup.

1. Upload the Lookup File
​

Include the filename query parameter in the request URL to specify the name of the Lookup file you want to use (in this example, lookupExample.csv). The value for the custom header Content-Type is text/csv.

Cribl.CloudSingle-Instance (Customer-Managed)Distributed (Customer-Managed)
curl --request PUT \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@</path/to/file/lookupExample.csv>'
curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@</path/to/file/lookupExample.csv>'
curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@</path/to/file/lookupExample.csv>'

The response is a JSON object that includes the Lookup file name with a random ID and information about the file’s number of rows and size, similar to this example:

{"filename":"lookupExample.csv.ABCDeFg.tmp","rows":100,"size":200}

2. Create the Lookup
​

Send a POST request to the /system/lookups endpoint to create the Lookup from the uploaded file. Provide the following values in the request body:

  • id: File name to use for the Lookup
  • fileInfo.filename: Value of the filename attribute from the response to the upload request in the previous step
Cribl.CloudSingle-Instance (Customer-Managed)Distributed (Customer-Managed)
curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'
curl --request POST \
--url 'https://${hostname}:${port}/api/v1/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'
curl --request POST \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'

The response is a JSON object that includes the Lookup file name, a random version for the Lookup, and the Lookup size, similar to this example:

{"items":[{"id":"newLookup.csv","version":"12AbC3D4","size":200}],"count":1}

Replace an Existing Lookup
​

To replace an existing Lookup, first upload the new Lookup file as described in step 1 of the Create a Lookup instructions. Then, send a PATCH request to the /system/lookups/{id} endpoint.

In the request URL, replace {id} with the file name of the Lookup you want to replace. Provide the following values in the request body:

  • id: File name of the existing Lookup
  • fileInfo.filename: Value of the filename attribute from the response to the upload request

In this example, the existing Lookup is newLookup.csv and the filename value is lookupUpdate.csv.1234aBcD.tmp.

Cribl.CloudSingle-Instance (Customer-Managed)Distributed (Customer-Managed)
curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'
curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'
curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'

The response is a JSON object that includes the Lookup file name, a random version for the Lookup, and the Lookup size, similar to this example:

{"items":[{"id":"newLookup.csv","version":"Xyz56789","size":200}],"count":1}

Deploy an Updated Lookup File to Worker Nodes
​

In Distributed environments, we recommend uploading the Lookup file only to the Leader Node and making a selective commit that includes only the Lookup file changes. The Leader then notifies Worker Nodes that a new configuration is available, and Worker Nodes pull the new configuration from the Leader Node.

For example, suppose an external team modifies a large Lookup file each night. It’s impractical to manually update Lookup files that are frequently modified. Instead, you can set up a workflow with the Cribl API to upload and deploy modified Lookup files to all Worker Nodes in the Worker Group.

The example in this section demonstrates how to configure such a workflow using the following sequence of requests to Cribl API endpoints:

  1. PUT /system/lookups to upload the initial Lookup file
  2. POST /system/lookups to create the Lookup on the Leader Node
  3. PUT /system/lookups to upload the modified Lookup file
  4. PATCH /system/lookups/{id} to update the Lookup on the Leader node
  5. POST /version/commit to commit the modified Lookup
  6. PATCH /master/groups/${groupName}/deploy to deploy the modified Lookup

The modified Lookup file will propagate to the Worker Nodes that belong to the Worker Group.

1. Upload the Lookup File to the Leader Node
​

In this example, the Lookup file host_destinations.csv is stored on the Leader Node default at /opt/host_destinations.csv.

Cribl.CloudDistributed (Customer-Managed)
curl --request PUT \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups?filename=host_destinations.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destinations.csv'
curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups?filename=host_destinations.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destinations.csv'

The response is a JSON object that includes the Lookup file name with a random ID and information about the file’s number of rows and size, similar to this example:

{"filename":"host_destinations.csv.ABCDeFg.tmp","rows":15,"size":200}

2. Create the Lookup
​

Send a POST request to the /system/lookups endpoint. Provide the following values in the request body:

  • id: File name to use for the Lookup
  • fileInfo.filename: Value of the filename attribute from the response to the upload request in the previous step
Cribl.CloudDistributed (Customer-Managed)
curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destinations",
  "fileInfo": {
    "filename": "host_destinations.csv.ABCDeFg.tmp"
  }
}'
curl --request POST \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destinations",
  "fileInfo": {
    "filename": "host_destinations.csv.ABCDeFg.tmp"
  }
}'

The response is a JSON object that includes the Lookup file name, a random version for the Lookup, and the Lookup size, similar to this example:

{"items":[{"id":"host_destinations.csv","version":"12AbC3D4","size":200}],"count":1}

3. Upload the Modified Lookup File
​

Cribl.CloudDistributed (Customer-Managed)
curl --request PUT \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups?filename=host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destination.csv'
curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups?filename=host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destination.csv'

The response is a JSON object that includes the Lookup file name with a random ID and information about the file’s number of rows and size, similar to this example:

{"filename":"host_destination.csv.ABCDeFg.tmp","rows":11,"size":245}

4. Update the Lookup File on the Worker Group
​

Update the modified Lookup file that you uploaded on the default Worker Group. In the request URL, replace {id} with the file name of the Lookup you’re modifying. Provide the following values in the request body:

  • id: File name of the existing Lookup
  • fileInfo.filename: Value of the filename attribute from the response to the upload request
Cribl.CloudDistributed (Customer-Managed)
curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups/host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destination.csv",
  "fileInfo": {
    "filename": "host_destination.csv.ABCDeFg.tmp"
  }
}'
curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups/host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destination.csv",
  "fileInfo": {
    "filename": "host_destination.csv.ABCDeFg.tmp"
  }
}'

The response is a JSON object that includes the Lookup file name, a random version for the Lookup, and the Lookup size, similar to this example:

{"items":[{"id":"host_destination.csv","version":"Xyz56789","size":245}],"count":1}

5. Commit the Modified Lookup File in the Worker Group
​

Commit only the modified lookup file in the default Worker Group. For example:

Cribl.CloudDistributed (Customer-Managed)
curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/version/commit' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "message": "automation@cribl:commit",
  "group": "default",
  "files": [
    "groups/default/data/lookups/host_destination.csv",
    "groups/default/data/lookups/host_destination.yml"
  ]
}'
curl --request POST \
--url 'https://${hostname}:${port}/api/v1/version/commit' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "message": "automation@cribl:commit",
  "group": "default",
  "files": [
    "groups/default/data/lookups/host_destination.csv",
    "groups/default/data/lookups/host_destination.yml"
  ]
}'

The response is a JSON object, similar to the following example:

{"items":[{"branch":"master","commit":"1234abcdbd21f460d9554f671dc79d215678wxyz","root":false,"summary":{"changes":2,"insertions":2,"deletions":0},"files":{"modified":["groups/default/data/lookups/host_destination.csv","groups/default/data/lookups/host_destination.yml"]}}],"count":1}

6. Deploy the Modified Lookup File
​

Send a PATCH request to the /master/groups/${groupName}/deploy endpoint. The request body must include the version parameter, whose value is the commit value from the response in the previous step.

Cribl.CloudDistributed (Customer-Managed)
curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/master/groups/default/deploy' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
"version": "1234abcdbd21f460d9554f671dc79d215678wxyz"
}'
curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/master/groups/default/deploy' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
"version": "1234abcdbd21f460d9554f671dc79d215678wxyz"
}'

The response is a JSON object, similar to the following example:

{"items":[{"description":"Default Worker Group","tags":"default","configVersion":"1234abcdbd21f460d9554f671dc79d215678wxyz","id":"default"}],"count":1}