This topic explains how to configure Amazon S3 as a Cribl Lake Storage Location.

Use an Amazon S3 bucket as a Storage Location for Cribl Lake Datasets so you keep direct ownership of your data for regional or compliance purposes, while using Cribl Lake provisioning, retention, access control, and analytics.

For connecting Datasets, detaching Storage Locations, API overview, and general BYOS limits, see Storage Locations.

Using any S3 bucket with a Cribl Lake Storage Location will incur AWS charges in addition to Cribl Lake credits.

Configure an Amazon S3 Storage Location ​

When you follow the recommended path in this section, Cribl Lake will simplify creating a new bucket on Amazon S3. After you configure initial settings in the Lake UI, Cribl will provide a ready-made CloudFormation template that you can download, then launch in the AWS CloudFormation console to create your infrastructure-as-code.

If you prefer to configure the bucket yourself on S3, see Setting Up a Dedicated Bucket Outside Cribl Lake.

Each Storage Location maps to one bucket. So you’ll repeat the configuration steps for each external bucket you want to create.

Select a Storage Provider ​

1. Navigate to Lake > Storage Locations > New Storage Location.
2. Select AWS, then Next.

Configure Settings ​

When you finish the Settings form, select Next to open the Authentication page.

Configure Authentication and Infrastructure ​

From New Storage Location > AWS > Authentication in Cribl Lake, you create your Amazon S3 infrastructure, and you then authenticate Cribl Lake on it.

For details about required AWS permissions, see Authentication.

1. Select Download Template and save the resulting CloudFormation template locally.
2. Select Open CloudFormation Console.

   

   

3. On the resulting AWS CloudFormation Stacks > Create stack page, select Upload a template file, then Choose file.

   

   

4. Upload the template you saved, then select Next.
5. In the resulting CloudFormation Configure stack options sequence, accept defaults, or adjust them to match your AWS practices. Then select Next.
6. On the Specify stack details page, assign your stack an arbitrary name. Then complete the Configure stack options sequence, accepting defaults and acknowledgments as appropriate.
7. On the Review and create page, select Submit.

   On the CloudFormation Stacks page, you will now see your new bucket progress from CREATE_IN_PROGRESS status to CREATE_COMPLETE.

8. Select the Outputs tab, and copy your new bucket’s AccessRoleARN value to the clipboard.

   

   

9. Back on the Cribl Lake Authentication page, paste this ARN into the AssumeRole ARN field.
10. Select Save to link your Cribl Lake Storage Location with your S3 bucket.

    Your new Storage Location will appear on the left side of the Lake Storage Locations page.

The Retention period that you set in the Cribl Lake Dataset configuration will automatically be reflected in the Amazon S3 Console’s Management tab. After you’ve connected a Dataset to an external S3 bucket, do not change the retention period directly in the S3 Console.

When you attach a Dataset to this Storage Location, you can also select an S3 storage class (Standard, Intelligent-Tiering, Glacier, and others) to control how Cribl Lake stores objects in your bucket. For details, see S3 Storage Class for BYOS Datasets.

Supported AWS Regions ​

Each Amazon S3 Storage Location can map to a bucket in one of the following AWS Regions:

• US East (Virginia)
• US East (Ohio)
• US West (Oregon)
• Canada (Central)
• Europe (Frankfurt)
• Europe (London)
• Europe (Zurich)
• Asia Pacific (Singapore)
• Asia Pacific (Sydney)

FedRAMP-Supported Regions ​

In Cribl.Cloud Gov (FedRAMP Moderate), Cribl Lake storage locations that use Amazon S3 support the following AWS regions:

• us-east-1 - US East (N. Virginia)
• us-east-2 - US East (Ohio)
• us-west-2 - US West (Oregon)

Cribl.Cloud Government currently does not support AWS GovCloud regions (such as us-gov-west-1, us-gov-east-1) for BYOS storage locations.

Detach an AWS Storage Location ​

To detach this storage location:

1. Remove any Datasets with connected Collectors and Destinations.
2. Open the Storage Location and select Detach.
3. Select the checkbox and type DETACH in the text field.
4. Go to your AWS S3 account and revoke Cribl Lake permissions for the bucket.

API Support for Amazon S3 Storage Locations ​

For automation and scripting, Members with appropriate Permissions can use Cribl storageLocations REST API endpoints. However, in this case, Cribl does not generate a CloudFormation template. So you will need to explicitly create corresponding S3 buckets and IAM roles with appropriate trust policy and permissions. (See Authentication for required permissions.)

Create a Storage Location via API ​

Here is an example request to create a Storage Location:

POST /products/lake/lakes/{lakeId}/storage-locations

Your request body should include the Storage Location details. Here’s an example:

{
  "id": "byosLocation",
  "provider": "aws-s3",
  "credentials": {
    "method": "auto",
    "roleToAssume": "arn:aws:iam::1111111111:role/test"
  },
  "config": {
    "bucketName": "byosbucket",
    "region": "us-west-2"
  }
}

Attach a Dataset via API ​

When adding a Dataset via API, you reference the target Storage Location instead of a bucket name. Here is an example YAML configuration:

datasets:
  - id: my_dataset
    storageLocation: my-byos-location
    retention: 30d
    format: json
    storageClass: STANDARD

Set Up a Dedicated Bucket Outside Cribl Lake ​

If you follow the procedure in Configure an Amazon S3 Storage Location, the provided CloudFormation template will create your new Amazon S3 bucket’s structure for you.

For advanced users who prefer to manage AWS resources directly, you can create a new, dedicated bucket and IAM role on S3, then link it as a Cribl Lake Storage Location. This is a more complex path, which requires you to explicitly configure access control on AWS.

Using the AWS Management Console or AWS API, you would:

1. Create a simple bucket configuration in an AWS Region that Cribl Lake supports.
2. Create a corresponding IAM role with an appropriate trust policy and permissions. (See Authentication for required permissions.)

Do not connect an existing S3 bucket that contains other data. Cribl Lake requires full control over lifecycle rules and S3 Inventory configuration, so each Cribl Lake Storage Location must be backed by a bucket dedicated exclusively to Cribl Lake.

Authentication ​

Cribl Lake accesses Amazon S3 buckets using STS Assume Role. If you follow the procedure in Configure an Amazon S3 Storage Location, the provided CloudFormation template will create the required resources and permissions for you.

IAM Role Permissions ​

Read the remainder of this section only if you choose to configure access control on an S3 bucket yourself (see Setting Up a Dedicated Bucket Outside Cribl Lake). These instructions assume that you are permitted to view and manage access details on both AWS and your Cribl Organization.

Your IAM role must provide the permissions listed in the following example. Replace the <bucketName> placeholder with your own bucket’s name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketAdminPermissions",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/Actor": "admin"
        }
      },
      "Action": [
        "s3:PutLifecycleConfiguration",
        "s3:GetLifecycleConfiguration",
        "s3:PutInventoryConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:PutBucketNotification",
        "s3:GetBucketNotification",
        "s3:GetEncryptionConfiguration",
        "s3:GetBucketTagging"
      ],
      "Resource": "arn:aws:s3:::<bucketName>",
      "Effect": "Allow"
    },
    {
      "Sid": "BucketObjectLevelReadWritePermissions",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:GetObjectTagging",
        "s3:PutObjectTagging",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::<bucketName>/*",
      "Effect": "Allow"
    },
    {
      "Sid": "BucketLevelReadPermissions",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads"],
      "Resource": "arn:aws:s3:::<bucketName>",
      "Effect": "Allow"
    }
  ]
}

Also, for Cribl resources to be able to assume your IAM role, the IAM role must provide sts:AssumeRole and sts:TagSession permissions to these Cribl roles:

• role/saas/cribl-lake-admin-<workspaceId> (manages your bucket properties, like lifecycle and inventory).

• role/<workspaceId>-* (Workers from all Cribl Stream Worker Groups need read/write access to your bucket).

• role/search-exec-<workspaceId> (Cribl Search needs read/write access to be able to perform searches against your bucket).

IAM Role Trust Policy ​

If you choose to configure access control on an S3 bucket yourself, you must provide a trust policy comparable to the following example. (See also the Restrictive Trust Policy option.) Replace the <CRIBL_AWS_ACCOUNT>, <workspaceId>, and <externalId> placeholders as follows:

1. In your Cribl.Cloud Organization, select Products from the top bar, then select Workspace from the sidebar.
2. To insert the <CRIBL_AWS_ACCOUNT>, select Trust from the sidebar. Then, from the displayed Worker ARN field, copy the numeric value that follows the string arn:aws:iam::.
3. To insert the <workspaceId>, select the Workspaces button on the top bar. In the resulting modal, copy the ID of the appropriate Workspace.
4. To insert the <externalId>, switch to your Storage Location config, and copy the External ID value you defined there. (For details, see Configure Initial Settings.)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CriblLakeAdminAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/saas/cribl-lake-admin-<workspaceId>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<externalId>",
          "aws:RequestTag/Actor": "admin"
        }
      }
    },
    {
      "Sid": "CriblLakeAdminTagSession",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/saas/cribl-lake-admin-<workspaceId>"
      },
      "Action": "sts:TagSession",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/Actor": "admin"
        }
      }
    },
    {
      "Sid": "CriblWorkerAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<CRIBL_TRUST_ACCOUNT>:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<externalId>"
        },
        "StringLike": {
          "aws:PrincipalArn": "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/<workspaceId>-*"
        }
      }
    },
    {
      "Sid": "CriblSearchAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/search-exec-<workspaceId>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<externalId>"
        }
      }
    }
  ]
}

Restrictive Trust Policy ​

The preceding model trust policy enables all Cribl Stream Worker Groups in the current Workspace to write to the configured bucket. If you instead wish to limit write permissions to only certain Worker Groups, modify the model policy’s CriblWorkerAssumeRole element as shown in the following listing.

Replace the <workerGroupId1> and <workerGroupId2> placeholders with the names of each Worker Group you want to authorize. Expand or contract the number of defined Worker Groups to meet your needs:

{
  "Sid": "CriblWorkerAssumeRole",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/<workspaceId>-<workerGroupId1>",
      "arn:aws:iam::<CRIBL_AWS_ACCOUNT>:role/<workspaceId>-<workerGroupId2>"
    ]
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": {
      "sts:ExternalId": "<externalId>"
    }
  }
}

Amazon S3-Specific Limitations ​

Each bucket must reside on Amazon S3 (in an AWS Region supported by Cribl Lake), not on other S3-compatible stores.

For limits that apply to all Storage Locations (Direct Access, Lakehouses, visibility of out-of-band writes), see Limitations on Storage Locations.