Create and Update Lookups

Lookups provide a flexible way to enrich events en route to their Destinations. This topic demonstrates how to use the Cribl API to get a list of lookups, upload lookup files, create lookups, and deploy frequenty modified lookup files.

About the Example Requests

​

Replace the variables in the example requests with the corresponding information for your Cribl deployment. In the cURL command options, replace ${token} with a valid API Bearer token. You can also set the $token environment variable to match the value of a Bearer token.

For customer-managed deployments, to use https in the URL for your requests as shown in these examples, you must configure Transport Layer Security (TLS).

In Cribl.Cloud and other distributed deployments, you must commit and deploy the changes you make. You can use the Cribl API to automate commit and deploy commands.

Get Existing Lookups

​

Before creating or updating lookups, send a GET request to the /system/lookups endpoint to retrieve a list of existing lookups. Getting a list of existing lookups is helpful for validating data and streamlining subsequent requests. For example, checking whether a lookup already exists helps you avoid errors due to sending a request to create a duplicate lookup.

curl --request GET \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

curl --request GET \
--url 'https://${hostname}:${port}/api/v1/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

curl --request GET \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

The response is a JSON object that includes lookup file names and sizes, along with available information like modes, number of rows, descriptions, tags, and random version numbers for each lookup. The response also includes the total number of lookups, similar to this example:

{"items":[{"id":"host_destinations.csv","size":77824,"mode":"disk","description":"Host destination lookups","tags": "destinations","rows":935,"version":"56eFG5H6"},{"id":"service_names_port_numbers.csv","size":25146},{"id":"name_vendor_products.csv","size":6002}],"count":3}

Create a Lookup

​

Follow these steps to use the Cribl API to create a lookup.

1. Upload the Lookup File

​

Include the filename query parameter in the request URL to specify the name of the lookup file you want to use (in this example, lookupExample.csv). The value for the custom header Content-Type is text/csv.

curl --request PUT \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@${/path/to/file/lookupExample.csv}'

curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@${/path/to/file/lookupExample.csv}'

curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups?filename=lookupExample.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@${/path/to/file/lookupExample.csv}'

The response is a JSON object that includes the lookup file name with a random ID and information about the file’s number of rows and size, similar to this example:

{"filename":"lookupExample.csv.ABCDeFg.tmp","rows":1433,"size":23014}

2. Create the Lookup

​

Send a POST request to the /system/lookups endpoint to create the lookup from the uploaded file. Provide the following values in the request body:

• id: File name to use for the lookup.
• fileInfo.filename: Value of the filename attribute from the response to the upload request in the previous step.

curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'

curl --request POST \
--url 'https://${hostname}:${port}/api/v1/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'

curl --request POST \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }
}'

This example creates an in-memory lookup, which is the default. To create a disk-based lookup, include "mode": "disk" in the request body as follows:

{
  "id": "newLookup",
  "fileInfo": {
    "filename": "lookupExample.csv.ABCDeFg.tmp"
  }.
  "mode": "disk"
}'

The response is a JSON object that includes the lookup file name, a random version for the lookup, and the lookup size, similar to this example:

{"items":[{"id":"newLookup.csv","version":"12AbC3D4","size":23014}],"count":1}

For disk-based lookups, the response includes mode and pendingTask, similar to this example:

{"items":[{"id":"newLookup.csv","mode":"disk","version":"12AbC3D4","pendingTask":{"id":"GuPdiPM8","type":"IMPORT"},"size":23014}],"count":1}

Read Choose a Lookup File Storage Mode for more information about in-memory and disk-based lookups.

Replace an Existing Lookup

​

To replace an existing lookup, first upload the new lookup file as described in step 1 of the Create a Lookup instructions. Then, send a PATCH request to the /system/lookups/{id} endpoint.

This example replaces an in-memory lookup. For an example that replaces a disk-based lookup, see Deploy Frequently Modified Lookup Files. For more information about the lookup modes, read Choose a Lookup File Storage Mode.

In the request URL, replace {id} with the file name of the lookup you want to replace. Provide the following values in the request body:

• id: File name of the existing lookup.
• fileInfo.filename: Value of the filename attribute from the response to the upload request.

In this example, the existing lookup is newLookup.csv and the filename value is lookupUpdate.csv.1234aBcD.tmp.

curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/${groupName}/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'

curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'

curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/m/${groupName}/system/lookups/newLookup.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "newLookup.csv",
  "fileInfo": {
    "filename": "lookupUpdate.csv.1234aBcD.tmp"
  }
}'

The response is a JSON object that includes the lookup file name, a random version for the lookup, and the lookup size, similar to this example:

{"items":[{"id":"newLookup.csv","version":"Xyz56789","size":200}],"count":1}

Deploy Frequently Modified Lookup Files

​

Manual updates are impractical for frequently modified lookup files. Instead, configure a workflow with the Cribl API to deploy modified lookup files to a Worker Group. You can deploy only the modified lookup file so that the updated data is accessible without restarting any Workers.

This example deploys a modified disk-based lookup. For an example that replaces an in-memory lookup, see Replace an Existing Lookup. For more information about the lookup modes, read Choose a Lookup File Storage Mode.

When deploying lookup updates, remember that each deploy request must specify all lookups intended for the Workers. For example, if you initially deployed lookups A (version a1), B (version b1), and C (version c1), all three reside on the Workers. To update to a modified lookup C (version c2), the deploy request must include A (a1), B (b1), and C (c2). Omitting A and B in the update request for C (c2) will result in the Workers having only lookup C.

In Distributed environments, we recommend updating the lookup file only on the Leader Node. The Leader then notifies Worker Nodes that a new configuration is available, and Worker Nodes pull the new configuration from the Leader Node.

This workflow uses the following sequence of requests to Cribl API endpoints:

1. GET /system/lookups to get a list of lookups to confirm that the lookup that you want to update exists.
2. GET /system/lookups/{id}/content?raw=1 to download the lookup file so that you can modify it.
3. PUT /system/lookups?filename={id} to upload the modified lookup file.
4. PATCH /system/lookups/{id} to replace the existing lookup file with the modified version.
5. GET /master/groups/{groupName} to get information about the modified lookup file.
6. PATCH /master/groups/{groupName}/deploy to selectively deploy the modified lookup file.

The modified lookup file will propagate to the Worker Nodes that belong to the Worker Group.

1. Confirm that the Lookup Exists

​

To prevent errors, get a list of the lookups that exist on the Worker Group so that you can confirm that the lookup that you want to modify is included.

curl --request GET \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

curl --request GET \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

The response is a JSON object that lists the lookups, similar to this example:

{"items":[{"id":"newLookup.csv","size":23014,"version":"12AbC3D4"},{"id":"host_destinations.csv","size":77824,"mode":"disk","version":"56eFG5H6","rows":935,"indices":[{"fields":["service_name"],"caseSensitive":true}]},{"id":"service_names_port_numbers.csv","size":25146},{"id":"name_vendor_products.csv","size":6002}],"count":4}

This example uses the host_destinations.csv lookup, which is included in the list of lookups for the default Worker Group.

2. Download the Lookup File

​

Download the lookup file so that you can modify it locally.

curl --request GET \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups/host_destinations.csv/content?raw=1' \
--header 'Authorization: Bearer ${token}' \
--output 'host_destinations.csv'

curl --request GET \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups/host_destinations.csv/content?raw=1' \
--header 'Authorization: Bearer ${token}' \
--output 'host_destinations.csv'

The client initiates a download of the lookup file. You may see a prompt to save the file, or the file may automatically download to a specified location, depending on your local browser settings.

3. Upload the Modified Lookup File

​

After you modify the lookup file, upload it from its current location on the local machine. In this example, the lookup file host_destinations.csv is stored in the /opt directory.

curl --request PUT \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups?filename=host_destinations.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destinations.csv'

curl --request PUT \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups?filename=host_destinations.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: text/csv' \
--data-binary '@/opt/host_destinations.csv'

The response is a JSON object that includes the lookup file name with a random ID and information about the file’s number of rows and size, similar to this example:

{"filename":"host_destinations.csv.ABCDeFg.tmp","rows":935,"size":77824}

4. Replace the Existing Lookup File

​

Replace the existing lookup file on the Worker Group with the modified lookup file that you uploaded. In the request URL, replace {id} with the file name of the lookup you’re modifying. Provide the following values in the request body:

• id: File name of the existing lookup.
• fileInfo.filename: Value of the filename attribute from the response to the upload request in step 3.
• mode: The type of lookup (in this example, disk).

curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default/system/lookups/host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destination.csv",
  "fileInfo": {
    "filename": "host_destination.csv.ABCDeFg.tmp"
  },
  "mode": "disk"
}'

curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/m/default/system/lookups/host_destination.csv' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "host_destination.csv",
  "fileInfo": {
    "filename": "host_destination.csv.ABCDeFg.tmp"
  },
  "mode": "disk"
}'

The response is a JSON object that includes information about the lookup file, similar to this example:

{"items":[{"id":"host_destination.csv","mode":"disk","version":"Xyz56789","size":78960,"pendingTask":{"id":"mNPQR456","type":"IMPORT"}}],"count":1}

5. Get Information About the Modified Lookup File

​

Get the commit number, context, file name, and version number for the modified lookup file. The request URL includes the fields query parameter to ensure that the response includes this information.

curl --request GET \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/master/groups/fields=git.log%2Cgit.commit%2Cgit.localChanges%2Clookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

curl --request GET \
--url 'https://${hostname}:${port}/api/v1/master/groups/fields=git.log%2Cgit.commit%2Cgit.localChanges%2Clookups' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json'

The response is a JSON object, similar to the following example. In the response, find the commit, lookupDeployments.context, lookupDeployments.lookups.file, and lookupDeployments.lookups.version attributes. You will need the values for these attributes to deploy the modified lookup file in step 6.

{"items":[{"onPrem":false,"estimatedIngestRate":1024,"provisioned":true,"streamtags":["default1"],"isFleet":false,"workerRemoteAccess":true,"isSearch":false,"description":"Default Worker Group","tags":"default","name":"default","configVersion":"27bbd48","cloud":{"provider":"aws","region":"us-west-2"},"lookupDeployments":[{"context":"cribl","lookups":[{"file":"host_destination.csv","version":"Xyz56789","deployedVersion":"56eFG5H6"}]}],"id":"default","git":{"localChanges":4,"log":[{"hash":"abcdefg1234567hijklmnop123456789qrstuv11","date":"2025-05-07 15:12:07 +0000","message":"default commit message","refs":"HEAD -> master","body":"","author_name":"Cribl Admin","author_email":"admin@cribl.cloud"},{"hash":"zyxwvut1234567srqponmlk123456789jihgfe11","date":"2025-04-30 22:34:58 +0000","message":"default commit message","refs":"","body":"","author_name":"Cribl Admin","author_email":"admin@cribl.cloud"}],"commit":"62aab48"}}],"count":1}

6. Selectively Deploy the Modified Lookup File

​

Send a PATCH request to the /master/groups/${groupName}/deploy endpoint to selectively deploy the modified lookup file without restarting any Workers. In the request body, provide the following values from the response to the request in step 5:

• version: Value of the commit attribute.
• lookups.context: Value of the lookupDeployments.context attribute.
• lookups.lookups.file: Value of the lookupDeployments.lookups.file.
• lookups.lookups.version: Value of the lookupDeployments.lookups.version.

curl --request PATCH \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/master/groups/default/deploy' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "version": "62aab48",
  "lookups": [
    {
      "context": "cribl",
      "lookups": [
        {
          "file": "host_destination.csv",
          "version": "Xyz56789"
        }
      ]
    }
  ]
}'

curl --request PATCH \
--url 'https://${hostname}:${port}/api/v1/master/groups/default/deploy' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "version": "257cd56",
  "lookups": [
    {
      "context": "cribl",
      "lookups": [
        {
          "file": "host_destination.csv",
          "version": "Xyz56789"
        }
      ]
    }
  ]
}'

The response is a JSON object that includes the lookup file name and its deployed version, similar to the following example:

{"items":[{"onPrem":false,"estimatedIngestRate":1024,"provisioned":true,"streamtags":["default1"],"isFleet":false,"workerRemoteAccess":true,"isSearch":false,"description":"Default Worker Group","tags":"default","name":"default","configVersion":"27bbd48","cloud":{"provider":"aws","region":"us-west-2"},"lookupDeployments":[{"context":"cribl","lookups":[{"file":"host_destination.csv","deployedVersion":"Xyz56789"}]}],"id":"default"}],"count":1}

You can deploy more than one modified lookup file in a single command. To do this, in your request body, add an object for each file. For example:

{
  "version": "257cd56",
  "lookups": [
    {
      "context": "cribl",
      "lookups": [
        {
          "file": "host_destination.csv",
          "version": "Xyz56789"
        },
        {
          "file": "top_domains.csv",
          "version": "RsT45TUv"
        }
      ]
    },
    {
      "context": "dbpack",
      "lookups": [
        {
          "file": "service_names_port_numbers.csv",
          "version": "aBC123gH"
        }
      ]
    }
  ]
}