This topic explains how to configure Microsoft Azure as a Cribl Lake Storage Location.

You can set an Azure Blob Storage container as the Storage Location for a Cribl Lake Dataset. This enables you to keep your data stored in Azure, but be able to search it in Cribl as a Dataset.

To connect Datasets, detach Storage Locations, and general BYOS limits, see Learn About Storage Locations.

Why Use an Azure Storage Location ​

By adding an Azure Storage Location to your Cribl Lake, you can:

• Keep log and security data in your own Azure storage account for compliance, but use Cribl for search and analytics over that data.
• Use Cribl Lake and Cribl Search to turn low-cost Azure Blob storage into a searchable archive.

Before You Begin ​

You don’t need to create a storage account or assign roles in advance. The ARM template that Cribl generates during setup creates the storage account and assigns the roles that Cribl Lake needs.

Before you configure an Azure Storage Location, make sure you have:

• An Azure resource group to hold the storage account. The ARM template creates the storage account in the resource group’s region, so choose a resource group in the region where you want to store data. See Supported Azure Regions.
• One of the Azure roles required for admin consent.
• Permissions to deploy the ARM template in the resource group. See the ARM template prerequisites.

If you want Azure management operations for this Storage Location to run from inside your own network, you’ll need an existing Outpost Group with at least one connected Outpost. See Route Azure Management Operations Through an Outpost.

Configure an Azure Storage Location ​

By following the recommended setup in this section, Cribl Lake will simplify creating a new Azure Blob Storage location for BYOS Lake Datasets.

Select a Storage Provider ​

1. Navigate to Lake > Storage Locations > New Storage Location.
2. Select Azure, then Next.

Configure Settings ​

On the Settings step, fill in Cribl Configuration first, then Azure Configuration. When you finish, select Next to continue to Authentication.

Cribl Configuration ​

Azure Configuration ​

Authenticate the Storage Account Access ​

After you configure the initial settings, you will authorize Cribl.Cloud in your Microsoft account, then generate an ARM template that you can download. Finally, launch the template in the Azure Data Storage console to create your infrastructure-as-code.

1. Select Authorize Cribl.Cloud to launch a new window and sign into your Microsoft account. This creates an enterprise application for your Cribl.Cloud organization in your Microsoft Azure account. For details about what admin consent grants and does not grant, see About Azure Authentication for Cribl Lake.
2. Once consent is confirmed, select Generate ARM Template, which downloads a JSON file. When you deploy the template to your resource group, it creates the storage account and grants Cribl Lake access to it.

About the ARM Template ​

When you create an Azure Storage Location in Cribl Lake, the system generates a customized Azure Resource Manager (ARM) template for your environment. Deploying this template in Azure is a required step in the Cribl Lake Azure BYOS onboarding flow.

You deploy the template in your Azure environment. Cribl does not deploy or run the ARM template on your behalf.

To learn about the ARM template, the resources it creates, storage account configuration, role assignments, inventory policy, and detailed steps on deploying it, see BYOS Azure ARM Template Reference.

Deploy the Custom Template in Microsoft Azure ​

To deploy the template in Azure:

1. In Microsoft Azure, navigate to Deploy a Custom Template.
2. Select Build your own template in the editor.
3. Select Load file and load the .json ARM template file you downloaded from Cribl.
4. Select the target Subscription and Resource Group.
5. Confirm the template loaded in Azure properly, then select Save.

To deploy the template in the command line:

Azure CLI

az deployment group create \
  --resource-group <your-resource-group> \
  --template-file CriblLakeByosAzureArmTemplate.json

PowerShell

New-AzResourceGroupDeployment `
  -ResourceGroupName <your-resource-group> `
  -TemplateFile CriblLakeByosAzureArmTemplate.json

Validate and Save the Azure Storage Location ​

Return to Cribl and move to Step 3: Confirm Template Deployment.

1. Check the box to tell Cribl that you’ve deployed the ARM template in Microsoft Azure.
2. Select Save.

Cribl will validate the configuration and connection to your Azure Storage account. At this step, Cribl will display any missing permissions if they are present.

If you selected an Outpost Group, Cribl first shares the Storage Location configuration with the Outpost Group, then validates your Azure resources through an Outpost in that group. This can take several minutes. If no Outpost in the group is connected, the save fails with an error.

When the connection is successful, you’ll see the Tenant ID in the list of Storage Locations.

When you attach a Dataset to this Storage Location, you can also select an Azure access tier (Inferred, Hot, Cool, Cold, or Archive) to control how Cribl Lake stores objects in your container. For details, see Storage Class for BYOS Datasets.

About Networking and Security ​

Cribl reaches your storage account for two purposes:

• Management operations: Calls to Azure Storage management APIs, such as reading storage account properties and creating or deleting containers, lifecycle policies, and retention policies.
• Data access: Cribl Stream Worker Groups write Dataset data, and Cribl Search reads it.

By default, all of this traffic reaches your storage account over public Azure endpoints. Two optional features, each enabled per Storage Location, change where traffic comes from and how it reaches your storage account. You can use either feature, both, or neither.

Public Path (Default) ​

Without an Outpost Group or a private endpoint, Cribl.Cloud calls the Azure Storage management APIs directly, and Cribl Search reads Dataset data over public Azure endpoints. Cribl authenticates with the Cribl Lake Enterprise Application and the roles that the ARM template assigns. See About Azure Authentication for Cribl Lake.

Management Operations Through an Outpost ​

When you select an Outpost Group for the Storage Location, an Outpost in your network makes every management call for that location, and Cribl.Cloud doesn’t call Azure directly. See Route Azure Management Operations Through an Outpost.

Private Endpoint for Cribl Search ​

When you enable a private endpoint, Cribl Search reads and writes data, including with export to lake, over a private endpoint that you approve in Azure. This traffic travels over Microsoft’s private network backbone instead of the public internet.

A private endpoint doesn’t route Azure Storage management API calls from the Cribl.Cloud Leader. You can use the private endpoint without an Outpost Group if the storage account continues to accept those calls over its public endpoint.

Cribl recommends disabling public network access to your storage account. The ARM template doesn’t disable it. If you disable public network access separately, set up Cribl Outpost, then select an Outpost Group for the Storage Location so that an Outpost makes management API calls from inside your network. Also configure your own private endpoint for any hybrid Worker Groups that write to this Storage Location. See Private Endpoint and Public Network Access.

Use an Outpost and a Private Endpoint Together ​

Using an Outpost Group and a private endpoint together doesn’t cause a conflict. You can enable both on the same Storage Location: the Outpost handles management operations, and the private endpoint handles Cribl Search reads.

Route Azure Management Operations Through an Outpost ​

If your security policy doesn’t allow a third party to write to your network from outside it, you can select an Outpost Group for an Azure Storage Location. Outposts in that group then make management calls for the location from inside your network, such as your Azure virtual network.

Selecting an Outpost Group is optional and applies to one Storage Location at a time. Storage Locations without an Outpost Group work as before, and selecting a group for one location doesn’t affect your other locations.

How It Works ​

• The Leader in Cribl.Cloud determines which operations to run and sends each one to a connected Outpost in the selected group. The Outpost calls the Azure Storage management APIs from your network and returns the result.
• Every management call for a Storage Location with an Outpost Group comes from an Outpost. Cribl.Cloud doesn’t fall back to calling Azure directly. If no Outpost in the group is available, the operation fails.
• The Leader remains the system of record. It stores the Storage Location configuration and rolls back multistep operations that fail partway, so a failed operation doesn’t leave orphaned Azure resources.
• The Outpost doesn’t relay Dataset data. Worker Groups write data, and Cribl Search reads it, as described in Networking and Security.

Security Model ​

• Outbound-only connection: The Outpost opens and maintains its connection to the Leader, and receives operations over that existing connection. Cribl.Cloud doesn’t open connections into your network.
• No private endpoint required: Routing management operations through an Outpost works without a private endpoint.
• No new privilege grant: Storage Locations that use an Outpost Group have the same Cribl Lake Enterprise Application, admin consent, and ARM template role assignments as any other Azure Storage Location. You don’t create an additional Enterprise Application, grant additional admin consent, or change role assignments. See About Azure Authentication for Cribl Lake.
• No stored credentials: For each operation, the Leader passes the Outpost a short-lived token that the Outpost uses only for that operation and doesn’t store. The Storage Location configuration that Cribl shares with the Outpost Group contains only identifiers: the Storage Location ID, tenant ID, client ID, subscription ID, resource group name, and storage account name.

Outpost Requirements ​

To route management operations through an Outpost Group, you need:

• An Outpost Group with at least one connected Outpost. To set one up, see Set Up Cribl Outpost. Cribl Outpost requires an Enterprise plan or license.
• Outposts that can reach the Azure Storage management APIs and your storage account endpoints from where they run.
• Permission to deploy configuration to the Outpost Group. Cribl shares the Storage Location configuration with the group when you save.

For more about Outposts, see Connect Nodes to Leader Through Cribl Outpost and Outpost Group Settings.

Select an Outpost Group ​

To select an Outpost Group when you create a Storage Location, select it in Outpost Group on the Settings step.

To select an Outpost Group for an existing Storage Location:

1. Navigate to Lake > Storage Locations and open the Storage Location.
2. In Outpost Group, select the group.
3. Select Save.

When you save, Cribl adds the Storage Location configuration to the Outpost Group and deploys it to the group’s Outposts. From then on, Outposts in that group make all management calls for the Storage Location.

Change or Clear the Outpost Group ​

Open the Storage Location, change or clear Outpost Group, then select Save. If you don’t have permission to update the Storage Location, Outpost Group is read-only.

• Change to a different Outpost Group: Cribl shares the Storage Location configuration with the new Outpost Group, switches the Storage Location to the new group, then removes the configuration from the previous group.

  • If deployment to the new Outpost Group fails, the Storage Location stays on the previous group.
  • If removing the configuration from the previous group fails, the Storage Location still works through the new group.
  • Changing groups requires permission to deploy to both the previous and new groups.

• Clear the group: Cribl removes the Storage Location configuration from the Outpost Group, redeploys the group, and returns the Storage Location to the default path, where Cribl.Cloud calls Azure directly. You can clear the group even if none of its Outposts is connected. Clearing requires permission to deploy to the group.

After you clear the Outpost Group, management calls come from Cribl.Cloud over public Azure endpoints. Make sure your storage account accepts these calls before you clear the group.

If you detach or delete a Storage Location that uses an Outpost Group, Cribl removes its configuration from the group automatically.

Supported Azure Regions ​

You can select any publicly available Azure region when you create an Azure Storage Location. There is no fixed subset of supported regions.

The ARM template creates the storage account in the region of the resource group you deploy it to, which follows Microsoft’s recommendation to place resources in the same region as their resource group. In Region, select the region of that resource group.

A storage account’s region can’t be changed after the account is created, so choose a resource group in a region that meets your data residency, latency, and compliance requirements. To keep data in its region of origin, create a resource group and a Cribl Lake Storage Location in each region where you need to land data.

When you save the Storage Location, Cribl validates that the storage account is in the region you selected. If it isn’t, Cribl reports an error so you can correct the selection before finishing setup.

Existing Storage Locations in previously supported regions are unaffected.

FedRAMP Support ​

FedRAMP deployments of Cribl.Cloud can’t use Cribl Lake Azure storage locations.

Detach an Azure Storage Location ​

To detach this storage location:

1. Remove any Datasets with connected Collectors and Destinations.
2. Open the Storage Location and select Detach.
3. Select the checkbox and type DETACH in the text field.
4. Select Detach. Next, revoke Cribl access in Azure by following the steps in the Azure documentation: Delete an enterprise application.
5. In Cribl, confirm you revoked access in Azure.

For a Storage Location with an Outpost Group, Cribl doesn’t check whether you revoked access in Azure, so you can select Complete immediately. Cribl removes the Storage Location configuration from the Outpost Group in the background.

Attach a Dataset via API ​

When you add a Dataset via API, you’ll reference the target Storage Location instead of a container name. You can set the Azure access tier with the storageClass field. Valid values are Inferred, Hot, Cool, Cold, and Archive. If you omit the field, the Dataset defaults to Inferred.

If the Storage Location has a default retention period, omit retention on the Dataset. The Dataset inherits the location retention, and you cannot override it. Here is an example YAML configuration:

datasets:
  - id: my_dataset
    storageLocation: my-azure-byos-location
    retention: 30d
    format: json
    storageClass: Cool

Select an Outpost Group via API ​

In the Storage Location API, the Outpost Group setting is config.controlPlaneOutpostGroupId. Set it to an Outpost Group ID to select the group, or change it to another ID to select a different group. To clear the selection, set it to null in an update request. If you omit the field from an update request, Cribl keeps the current selection.

{
  "config": {
    "controlPlaneOutpostGroupId": "azure_outpost"
  }
}

The same requirements and lifecycle behavior apply as in the UI.

Next Steps ​

Now that you’ve successfully created an Azure Storage Location, you can attach a Dataset to it. See Connect Datasets to Storage Locations for details.