On This Page

Home / Lake/Learn About Storage Locations (Bring Your Own Storage)

Learn About Storage Locations (Bring Your Own Storage) ​

Create Lake Datasets on storage that you directly own, achieving compliance while taking advantage of streamlined management in Cribl Lake.

You can host Cribl Lake Datasets on Amazon S3 or Azure. Storage Locations are available on an Enterprise billing plan. For details, see Cribl Pricing.

Prerequisite: Configure a Storage Location by Provider ​

First, you need to configure at least one Storage Location. Choose a provider and follow the steps to configure the Storage Location. Then, return to this topic to connect Lake Datasets.

Connect Lake Datasets to Storage Locations ​

You can link multiple Connected Datasets per Storage Location.

Follow the procedure in Manage Lake Datasets to create a new Dataset, or to edit an existing Dataset. On the Dataset configuration page, select a configured Storage Location from the drop-down. Then save or resave the Dataset.

If you don’t make a selection on this drop-down, the Dataset will use the internal Cribl Lake default location.

Screenshot of a Cribl Lake Dataset configuration’s Storage Location drop-down, showing selection of an external S3 bucket.
Connecting a Dataset to a Storage Location

The Dataset will now use the selected (or default) Storage Location to store its data. On the Storage Location page, this Dataset will appear in the right column of Connected Datasets.

As with other Datasets, you can write to this Dataset from Cribl Stream with the Cribl Lake Destination. And you can access the Dataset contents from Cribl Search and Stream (subject to some Limitations).

The Retention period that you set in the Cribl Lake Dataset configuration is reflected in your cloud provider’s object lifecycle or retention UI where applicable. After you’ve connected a Dataset to an external bucket, avoid changing retention outside Cribl Lake in ways that conflict with Lake Dataset lifecycle expectations.

Detach a Storage Location Connection ​

If you choose to detach a Storage Location, all associated Datasets will also be removed from Cribl Lake. However, your data will not be deleted from the underlying bucket or storage container.

For steps to detach, see your storage location provider:

API Support for Storage Locations ​

For automation and scripting, Members with appropriate Permissions can use Cribl storageLocations REST API endpoints. See Amazon S3 storage locations for an example request body.

Limitations on Storage Locations ​

  • Each Storage Location must map to one bucket or equivalent container in the target cloud.
  • You can’t populate an external storage container/bucket using Cribl Lake Direct Access.
  • You can’t assign storage containers/buckets for these Connected Datasets to Lakehouses.
  • Data sent to an external storage container/bucket through the Cribl Lake Destination will be accessible to Cribl Search, and to Cribl Stream for replay through its Cribl Lake Collector. However, any data that you send directly to the storage container/bucket outside of Cribl products will not be visible to Cribl Search or Stream.
  • Provider-specific limits (supported regions, authentication, and setup paths) are documented on Amazon S3 storage locations and Azure storage locations.