Configure Resources with the Cribl Terraform Provider

Preview Feature
The Cribl Terraform provider is a Preview feature that is still being developed. We do not recommend using it in a production environment, because the feature might not be fully tested or optimized for performance, and related documentation could be incomplete.
Please continue to submit feedback through normal Cribl support channels, but assistance might be limited while the feature remains in Preview.

This code example demonstrates how to use the Cribl Terraform provider to create the following resources in Cribl Stream:

A Worker Group to manage the configuration.
A Syslog Source to receive data on port 9021.
An S3 Destination to store processed data.
A Pipeline that filters events and keeps only data in the eventSource and eventID fields.
A Route that connects the Source, Pipeline, and Destination.

This example also commits and deploys the resource configurations to the Worker Group to make them active.

About Code Examples
Code examples require you to authenticate with the Cribl Terraform provider. Read the Terraform provider authentication documentation to learn how to configure authentication. The API Credential (Cribl.Cloud and hybrid) or login credentials (on-prem) that you use must have the necessary Permissions for the operations in the code examples.
Replace the variables in code examples with the corresponding information for your Cribl deployment.
For on-prem deployments, if CRIBL_ONPREM_SERVER_URL uses https, you must configure Transport Layer Security (TLS).
Code examples do not include all available resource arguments. For a complete list of arguments for specific resources, refer to the documentation in the Terraform Registry.

The Terraform configuration in this example is split into four files that together form one Terraform module:

variables.tf
terraform.tfvars
provider.tf
main.tf

Copy the example files into an empty local working directory as you complete the following sections. When you run the example, Terraform reads the files in your local working directory, resolves dependencies from attribute references (such as group_id), and creates resources in a single terraform apply.

Declare Input Variables with `variables.tf`

Copy variables.tf from the example into your local working directory. The file declares the input variables for the module. Each variable block gives a name, data type, and description for a value that you supply at apply time (for example, the Worker Group ID or S3 bucket name). Sensitive values such as API credentials and AWS keys are marked sensitive = true so Terraform does not print them in logs.

In the Cribl.Cloud example, this file also declares variables for Organization ID, Workspace name, and API Credential values. The on-prem example declares only the Worker Group and S3 variables because on-prem login credentials are supplied through environment variables or a credentials file as described in Authenticate in On-Prem Deployments.

# User-supplied parameters
#
# Supply values for the declared variables in this file in terraform.tfvars.
#
# Required for Cribl.Cloud authentication:
# - organization_id: Organization ID
# - workspace_id: Workspace name
# - client_id: Client ID for your API Credential
# - client_secret: Client Secret for your API Credential
#
# Required for the Worker Group:
# - worker_group_id: ID for the new Worker Group (must not already exist)
#
# Required for the S3 Destination:
# - aws_api_key: AWS Access Key ID
# - aws_secret_key: AWS Secret Access Key
# - aws_bucket_name: S3 bucket name
# - aws_region: S3 bucket region, such as us-east-2
#
# To create an API Credential, see:
# https://docs.cribl.io/cribl-as-code/terraform-auth/#terraform-auth-cloud

variable "organization_id" {
  type        = string
  description = "Cribl.Cloud Organization ID"
}

variable "workspace_id" {
  type        = string
  description = "Cribl.Cloud Workspace name"
}

variable "worker_group_id" {
  type        = string
  description = "ID for the new Worker Group. Must not already exist in your deployment."
}

variable "aws_api_key" {
  type        = string
  description = "AWS Access Key ID for the S3 Destination"
  sensitive   = true
}

variable "aws_secret_key" {
  type        = string
  description = "AWS Secret Access Key for the S3 Destination"
  sensitive   = true
}

variable "aws_bucket_name" {
  type        = string
  description = "S3 bucket name for the S3 Destination"
}

variable "aws_region" {
  type        = string
  description = "AWS region for the S3 bucket, such as us-east-2"
}

variable "client_id" {
  type        = string
  description = "Client ID from your Cribl API Credential"
  sensitive   = true
}

variable "client_secret" {
  type        = string
  description = "Client Secret from your Cribl API Credential"
  sensitive   = true
}

# User-supplied parameters
#
# Supply values for the declared variables in this file in terraform.tfvars.
#
# Required for the Worker Group:
# - worker_group_id: ID for the new Worker Group (must not already exist)
#
# Required for the S3 Destination:
# - aws_api_key: AWS Access Key ID
# - aws_secret_key: AWS Secret Access Key
# - aws_bucket_name: S3 bucket name
# - aws_region: S3 bucket region, such as us-east-2

variable "worker_group_id" {
  type        = string
  description = "ID for the new Worker Group. Must not already exist in your deployment."
}

variable "aws_api_key" {
  type        = string
  description = "AWS Access Key ID for the S3 Destination"
  sensitive   = true
}

variable "aws_secret_key" {
  type        = string
  description = "AWS Secret Access Key for the S3 Destination"
  sensitive   = true
}

variable "aws_bucket_name" {
  type        = string
  description = "S3 bucket name for the S3 Destination"
}

variable "aws_region" {
  type        = string
  description = "AWS region for the S3 bucket, such as us-east-2"
}

Set Variable Values with `terraform.tfvars`

This example uses a terraform.tfvars file to supply the values for the input variables declared in variables.tf. Copy terraform.tfvars from the example into your local working directory and replace each placeholder with a value for your deployment. Terraform loads terraform.tfvars automatically when you run terraform plan or terraform apply.

If you don’t want to use a terraform.tfvars file to set variable values, you can use one of these methods instead:

Pass values on the command line with -var or -var-file.
Set TF_VAR_<variable_name> environment variables.

Do not commit terraform.tfvars or any other files that contain sensitive information to version control.

# Copy this file to terraform.tfvars and replace each placeholder with the value
# for your Cribl.Cloud deployment. Do not commit terraform.tfvars to version control.

organization_id = "your-organization-id"
workspace_id    = "your-workspace-name"
client_id       = "client-id-from-your-api-credential"
client_secret   = "client-secret-from-your-api-credential"
worker_group_id = "id-for-new-worker-group-must-not-already-exist"
aws_api_key     = "your-aws-access-key-id"
aws_secret_key  = "your-aws-secret-access-key"
aws_bucket_name = "your-s3-bucket-name"
aws_region      = "your-s3-bucket-region-such-as-us-east-2"

# Copy this file to terraform.tfvars and replace each placeholder with the value
# for your on-prem deployment. Do not commit terraform.tfvars to version control.

worker_group_id = "id-for-new-worker-group-must-not-already-exist"
aws_api_key     = "your-aws-access-key-id"
aws_secret_key  = "your-aws-secret-access-key"
aws_bucket_name = "your-s3-bucket-name"
aws_region      = "your-s3-bucket-region-such-as-us-east-2"

Configure Terraform Provider with `provider.tf`

Copy provider.tf from the example into your local working directory. The file configures the Cribl Terraform provider and declares the criblio provider source and minimum version in a terraform block.

In the Cribl.Cloud example, the provider "criblio" block sets Organization ID, Workspace name, and API Credential values from the variables you supply in terraform.tfvars.
In the on-prem example, the provider "criblio" block is empty. Configure authentication using environment variables or a credentials file as described in Authenticate in On-Prem Deployments.

# Configure provider
#
# Authenticates for Cribl.Cloud using organization_id, workspace_id, client_id,
# and client_secret declared in variables.tf. Supply their values in terraform.tfvars.
#
# For other authentication methods, see:
# https://docs.cribl.io/cribl-as-code/terraform-auth/

terraform {
  required_providers {
    criblio = {
      source  = "criblio/criblio"
      version = ">= 1.20.138"
    }
  }
}

provider "criblio" {
  organization_id = var.organization_id
  workspace_id    = var.workspace_id
  client_id       = var.client_id
  client_secret   = var.client_secret
}

# Configure provider
#
# The provider block is empty. Configure authentication using
# environment variables or a credentials file as described in
# https://docs.cribl.io/cribl-as-code/terraform-auth/#terraform-auth-on-prem.

terraform {
  required_providers {
    criblio = {
      source  = "criblio/criblio"
      version = ">= 1.20.138"
    }
  }
}

provider "criblio" {
}

Create Cribl Stream Resources with `main.tf`

Copy main.tf from the example into your local working directory. The file defines the Cribl Stream resources that the example creates, in dependency order:

criblio_group creates a new Worker Group.
criblio_source creates a Syslog Source that listens on port 9021.
criblio_destination creates an S3 Destination that uses aws_bucket_name, aws_region, aws_api_key, and aws_secret_key.
criblio_pipeline creates a Pipeline with an Eval function that keeps only the eventSource and eventID fields.
criblio_routes updates the Routing table with a Route from the Syslog Source through the Pipeline to the S3 Destination. The criblio_routes resource requires a complete representation of the Routing table, so the example includes a default catch-all Route to preserve existing behavior.
criblio_commit commits the Worker Group configuration.
criblio_config_version (data source) reads the version ID produced by the commit.
criblio_deploy deploys the committed configuration to the Worker Group.

In the Cribl.Cloud example, the Worker Group is provisioned in AWS (on_prem = false). In the on-prem example, the Worker Group sets on_prem = true and omits cloud provisioning settings.

# Create Worker Group
resource "criblio_group" "example" {
  id                    = var.worker_group_id
  name                  = var.worker_group_id
  product               = "stream"
  on_prem               = false
  is_fleet              = false
  worker_remote_access  = true
  estimated_ingest_rate = 2048 # Equivalent to 24 MB/s with 9 Worker Processes
  provisioned           = false
  cloud = {
    provider = "aws"
    region   = "us-east-1" # Replace with a different AWS region if desired
  }
}

# Create Syslog Source
resource "criblio_source" "syslog" {
  id       = "in-syslog-9021"
  group_id = criblio_group.example.id

  input_syslog = {
    id             = "in-syslog-9021"
    type           = "syslog"
    host           = "0.0.0.0"
    tcp_port       = 9021 # Replace with a different port number if desired
    disabled       = false
    send_to_routes = true
    tls = {
      disabled = true
    }
  }

  depends_on = [criblio_group.example]
}

# Create S3 Destination
resource "criblio_destination" "s3" {
  id       = "out_s3"
  group_id = criblio_group.example.id

  output_s3 = {
    id                    = "out_s3"
    type                  = "s3"
    bucket                = var.aws_bucket_name
    region                = var.aws_region
    aws_api_key           = var.aws_api_key
    aws_secret_key        = var.aws_secret_key
    stage_path            = "/tmp/cribl_stage"
    compress              = "gzip"
    compression_level     = "best_speed"
    empty_dir_cleanup_sec = 300
  }

  depends_on = [criblio_group.example]
}

# Create Pipeline
resource "criblio_pipeline" "filter" {
  id       = "my_pipeline"
  group_id = criblio_group.example.id

  conf = {
    async_func_timeout = 1000
    functions = [
      {
        id       = "eval"
        filter   = "true"
        disabled = false
        final    = true
        conf = jsonencode({
          remove = ["*"]
          keep   = ["eventSource", "eventID"]
        })
      }
    ]
  }

  depends_on = [criblio_group.example]
}

# Create Routes
resource "criblio_routes" "main" {
  group_id = criblio_group.example.id
  id       = "default" # Routing table ID (do not change; the supported value is default)

  routes = [
    {
      name        = "your_route"              # Replace with the name of the Route
      description = "This is my new Route"    # Replace with the desired Route description
      pipeline    = criblio_pipeline.filter.id
      output      = criblio_destination.s3.id
      filter      = "__inputId=='in-syslog-9021'"
      final       = true
      disabled    = false
    },
    {
      name     = "default"
      pipeline = "main"
      output   = "default"
      filter   = "true"
      final    = false
      disabled = false
    }
  ]

  depends_on = [
    criblio_source.syslog,
    criblio_destination.s3,
    criblio_pipeline.filter,
  ]
}

# Commit configuration
resource "criblio_commit" "example" {
  effective = true
  group     = criblio_group.example.id
  message   = "Commit for Cribl Stream example"

  depends_on = [criblio_routes.main]
}

# Read config version
data "criblio_config_version" "latest" {
  id         = criblio_group.example.id
  depends_on = [criblio_commit.example]
}

# Deploy configuration
resource "criblio_deploy" "example" {
  id      = criblio_group.example.id
  version = data.criblio_config_version.latest.items[0]
}

# Create Worker Group
resource "criblio_group" "example" {
  id                   = var.worker_group_id
  name                 = var.worker_group_id
  product              = "stream"
  on_prem              = true
  is_fleet             = false
  worker_remote_access = true
  provisioned          = false
}

# Create Syslog Source
resource "criblio_source" "syslog" {
  id       = "in-syslog-9021"
  group_id = criblio_group.example.id

  input_syslog = {
    id             = "in-syslog-9021"
    type           = "syslog"
    host           = "0.0.0.0"
    tcp_port       = 9021 # Replace with a different port number if desired
    disabled       = false
    send_to_routes = true
    tls = {
      disabled = true
    }
  }

  depends_on = [criblio_group.example]
}

# Create S3 Destination
resource "criblio_destination" "s3" {
  id       = "out_s3"
  group_id = criblio_group.example.id

  output_s3 = {
    id                    = "out_s3"
    type                  = "s3"
    bucket                = var.aws_bucket_name
    region                = var.aws_region
    aws_api_key           = var.aws_api_key
    aws_secret_key        = var.aws_secret_key
    stage_path            = "/tmp/cribl_stage"
    compress              = "gzip"
    compression_level     = "best_speed"
    empty_dir_cleanup_sec = 300
  }

  depends_on = [criblio_group.example]
}

# Create Pipeline
resource "criblio_pipeline" "filter" {
  id       = "my_pipeline"
  group_id = criblio_group.example.id

  conf = {
    async_func_timeout = 1000
    functions = [
      {
        id       = "eval"
        filter   = "true"
        disabled = false
        final    = true
        conf = jsonencode({
          remove = ["*"]
          keep   = ["eventSource", "eventID"]
        })
      }
    ]
  }

  depends_on = [criblio_group.example]
}

# Create Routes
resource "criblio_routes" "main" {
  group_id = criblio_group.example.id
  id       = "default" # Routing table ID (do not change; the supported value is default)

  routes = [
    {
      name        = "your_route"              # Replace with the name of the Route
      description = "This is my new Route"    # Replace with the desired Route description
      pipeline    = criblio_pipeline.filter.id
      output      = criblio_destination.s3.id
      filter      = "__inputId=='in-syslog-9021'"
      final       = true
      disabled    = false
    },
    {
      name     = "default"
      pipeline = "main"
      output   = "default"
      filter   = "true"
      final    = false
      disabled = false
    }
  ]

  depends_on = [
    criblio_source.syslog,
    criblio_destination.s3,
    criblio_pipeline.filter,
  ]
}

# Commit configuration
resource "criblio_commit" "example" {
  effective = true
  group     = criblio_group.example.id
  message   = "Commit for Cribl Stream example"

  depends_on = [criblio_routes.main]
}

# Read config version
data "criblio_config_version" "latest" {
  id         = criblio_group.example.id
  depends_on = [criblio_commit.example]
}

# Deploy configuration
resource "criblio_deploy" "example" {
  id      = criblio_group.example.id
  version = data.criblio_config_version.latest.items[0]
}

Run the Example

Complete the preceding sections to copy the example files into your local working directory and set variable values in terraform.tfvars. Before you run the example, confirm that Terraform 1.0 or later is installed.

Open a terminal in your local working directory and run terraform init to download and install the provider plugin:
```
terraform init
```
Review the planned changes:
```
terraform plan
```
Apply the configuration:
```
terraform apply
```
Review the execution plan that terraform apply displays and type yes when Terraform prompts you to apply the changes.

Configure authentication using environment variables or a credentials file as described in Authenticate in On-Prem Deployments.
Open a terminal in your local working directory and run terraform init to download and install the provider plugin:
```
terraform init
```
Review the planned changes:
```
terraform plan
```
Apply the configuration:
```
terraform apply
```
Review the execution plan that terraform apply displays and type yes when Terraform prompts you to apply the changes.

After a successful terraform apply, your Cribl Stream deployment has a new Worker Group with a Syslog Source on port 9021, a Pipeline, an S3 Destination, and a Route that connects them. The Worker Group configuration is committed and deployed.

Configure Resources with the Cribl Terraform Provider ​

Preview Feature ​

About Code Examples ​

Declare Input Variables with variables.tf ​

Set Variable Values with terraform.tfvars ​

Configure Terraform Provider with provider.tf ​

Create Cribl Stream Resources with main.tf ​

Run the Example ​

Common Resources