Add a Cribl Search Pack and Create a Lake Dataset

Preview Feature

​

The Cribl SDKs are Preview features that are still being developed. We do not recommend using them in a production environment, because the features might not be fully tested or optimized for performance, and related documentation could be incomplete.

Please continue to submit feedback through normal Cribl support channels, but assistance might be limited while the features remain in Preview.

These code examples demonstrate how to use the Cribl Python SDK for the control plane or the Cribl API to add a Cribl Search Pack from the Cribl Packs Dispensary and create a Lake Dataset.

About the Code Examples

​

The code examples use Bearer token authentication. Read the authentication documentation to learn how to get a Bearer token. The Permissions granted to your Bearer token must include creating and managing Packs and Lake Datasets.

Replace the variables in the examples with the corresponding information for your Cribl deployment.

The configurations in the examples do not include all available body parameters. For a complete list of body parameters for each endpoint, refer to the documentation in the API Reference.

Cribl Search and Cribl Lake are available only on Cribl.Cloud, so this page does not include examples for customer-managed deployments.

Add a Search Pack and Create a Lake Dataset with the Python SDK

​

This example demonstrates how to use the Python SDK for the control plane to:

• Add the Cribl Search AWS VPC Flow Logs Search Pack from the Cribl Packs Dispensary.

• Create a Lake Dataset with a basic configuration: : Dataset ID, Cribl Lake storage location, retention period of 30 days, and Direct Access (HTTP) disabled.

This example includes the Pack URL for Cribl Search AWS VPC Flow Logs, which is used as the value of the source parameter. To get the URL for a different Pack, see Get the URL for a Search Pack.

"""
Replace the placeholder values for ORG_ID, CLIENT_ID, CLIENT_SECRET, and
WORKSPACE_NAME with your Organization ID, Client ID and Secret, and
Workspace name. To get your CLIENT_ID and CLIENT_SECRET values, follow
the steps at https://docs.cribl.io/cribl-as-code/authentication/#cloud-auth.
Your Client ID and Secret are sensitive information and should be kept private.

NOTE: This example is for Cribl.Cloud deployments only.
"""

import asyncio
from cribl_control_plane import CriblControlPlane
from cribl_control_plane.models import Security, SchemeClientOauth

# Cribl.Cloud configuration: Replace the placeholder values
ORG_ID = "your-org-id"  # Replace with your Organization ID
CLIENT_ID = "your-client-id"  # Replace with your OAuth2 Client ID
CLIENT_SECRET = "your-client-secret"  # Replace with your OAuth2 Client Secret
WORKSPACE_NAME = "your-workspace-name"  # Replace with your Workspace name

# AWS VPC Flow Logs Search Pack from Cribl Packs Dispensary
PACK_URL = "https://packs.cribl.io/dl/cribl-search-aws-vpc-flow-logs/0.1.1/cribl-search-aws-vpc-flow-logs-0.1.1.crbl"
PACK_ID = "cribl-search-aws-vpc-flow-logs"

LAKE_ID = "default"
DATASET_ID = "aws-vpc-flow-logs-dataset"
base_url = f"https://{WORKSPACE_NAME}-{ORG_ID}.cribl.cloud/api/v1"

# Create authenticated SDK client
async def main():
    client_oauth = SchemeClientOauth(
        client_id=CLIENT_ID,
        client_secret=CLIENT_SECRET,
        token_url="https://login.cribl.cloud/oauth/token",
        audience="https://api.cribl.cloud",
    )

    security = Security(client_oauth=client_oauth)
    cribl = CriblControlPlane(server_url=base_url, security=security)

    # Construct URLs for pack installation
    search_group_url = f"{base_url}/m/default_search"

    # Install AWS VPC Flow Logs Search Pack
    cribl.packs.install(
        request={
            "source": PACK_URL,
            "id": PACK_ID,
        },
        server_url=search_group_url,
    )
    print(f"✅ Installed Search Pack {PACK_ID} from Cribl Packs Dispensary")

    # Create lake dataset
    cribl.lake_datasets.create(
        lake_id=LAKE_ID,
        id=DATASET_ID,
        retention_period_in_days=30,
        http_da_used=False,
        storage_location_id="cribl_lake",
    )

    print(f"✅ Created Lake Dataset: {DATASET_ID}")


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except Exception as error:
        print(f"❌ Something went wrong: {error}")

Add a Search Pack and Create a Lake Dataset with the Cribl API

​

The example requests in this section demonstrate how to use the Cribl API to add a Cribl Search Pack from the Cribl Packs Dispensary and create a Lake Dataset.

Add a Search Pack

​

This example adds the Cribl Search AWS VPC Flow Logs Search Pack from the Cribl Packs Dispensary.

This example includes the Pack URL for Cribl Search AWS VPC Flow Logs, which is used as the value of the source parameter. To get the URL for a different Pack, see Get the URL for a Search Pack.

curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/default_search/packs' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "cribl-search-aws-vpc-flow-logs",
  "source": "https://packs.cribl.io/dl/cribl-search-aws-vpc-flow-logs/0.1.1/cribl-search-aws-vpc-flow-logs-0.1.1.crbl"
}'

Create a Lake Dataset

​

This example creates a Lake Dataset in the default lake with a basic configuration: Dataset ID, Cribl Lake storage location, retention period of 30 days, and Direct Access (HTTP) disabled.

curl --request POST \
--url 'https://${workspaceName}-${organizationId}.cribl.cloud/api/v1/m/products/lake/lakes/default/datasets' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data '{
  "id": "aws-vpc-flow-logs-dataset",
  "storageLocationId": "cribl_lake",
  "retentionPeriodInDays": 30,
  "httpDAUsed": false
}'

Get the URL for a Search Pack

​

To add a Search Pack from the Cribl Packs Dispensary or the Dispensary GitHub Repository, provide the Pack URL as the value for the source parameter in your request. The URL must be the direct URL location of the .crbl file for the Pack.

Read Pack Repositories to learn about the differences between the Cribl Packs Dispensary and the Dispensary GitHub Repository.

Here’s how to get a Pack’s URL from the Cribl Packs Dispensary:

1. Search the Cribl Packs Dispensary to find the Pack that you want to add. Select the Pack’s tile to open the Pack sidebar.

2. Open the developer tools for your browser and select the Network tab.

3. At the upper-right of the Pack sidebar, select the download icon.

4. In your browser’s developer tools, in the Name column, select the entry for the Pack’s .crbl file and the Headers tab. Copy the Request URL: the copied URL is the value to provide for the source parameter in your request.

To get a Pack’s URL from the Dispensary GitHub Repository:

1. Search the Dispensary GitHub Repository to find the repository for the Pack that you want to add. Select the Pack’s repository to open it.

2. In the right sidebar, select Releases.

3. On the Releases page, find the release that you want to use and expand the Assets section.

4. In the Assets section, find the .crbl file for the Pack. Right-click the .crbl file and select the option to copy the link. The copied link URL is the value to provide for the source parameter in your request.