Skip to main content
Version: 3.2

Azure Blob Storage

Azure Blob Storage is a non-streaming Destination type. Cribl LogStream does not have to run on Azure in order to deliver data to it. Azure Data Lake Storage Gen2 (hierarchical namespace) is also supported.

Configuring LogStream to Output to Azure Blob Storage

In the QuickConnect UI: Click + Add beside Destinations. From the resulting drawer's tiles, select Azure > Blob Storage. Next, click either + Add New or (if displayed) Select Existing. The resulting drawer will provide the following options and fields.

Or, in the Data Routes UI: From the top nav of a LogStream instance or Group, select Data > Destinations. From the resulting page's tiles or the Destinations left nav, select Azure > Blob Storage. Next, click + Add New to open a New Destination modal that provides the following options and fields.

General Settings

Output ID: Enter a unique name to identify this Destination definition.

Authentication method: See Authentication Settings below.

Container name: Enter the container name. (A container organizes a set of blobs, similar to a directory in a file system.)

Create container: Toggle to Yes to create the configured container in Azure Blob Storage if one does not already exist.

Blob prefix: Root directory to prepend to path before uploading.

Staging location: Local filesystem location in which to buffer files before compressing and moving them to the final destination. Cribl recommends that this location be stable and high-performance.

Partitioning expression: JavaScript expression to define how files are partitioned and organized. Defaults to `${host}/${sourcetype}` If left blank, Cribl LogStream will fall back to event.__partition.

Data format: Format of the output data. Defaults to json.

File name prefix expression: The output filename prefix. Must be a JavaScript expression (which can evaluate to a constant), enclosed in quotes or backticks. Defaults to CriblOut.

Compress: Data compression format used before moving to final destination. Defaults to none. Cribl recommends setting to gzip.

Backpressure behavior: Whether to block or drop events when all receivers are exerting backpressure. (Causes might include an accumulation of too many files needing to be closed.) Defaults to Block.

Authentication Settings

Use the Authentication method buttons to select one of these options:

  • Manual: Use this default option to enter your Azure Storage connection string directly. Exposes a Connection string field for this purpose. (If left blank, LogStream will fall back to env.AZURE_STORAGE_CONNECTION_STRING.)

  • Secret: This option exposes a Connection string (text secret) drop-down, in which you can select a stored secret that references an Azure Storage connection string. A Create link is available to store a new, reusable secret.

Connection String Format

Either authentication method uses an Azure Storage connection string in this format: DefaultEndpointsProtocol=[http|https];AccountName=<your‑account‑name>;AccountKey=<your‑account‑key>

A fictitious example, using Microsoft's recommended HTTPS option, is: DefaultEndpointsProtocol=https;AccountName=storagesample;AccountKey=12345678...32

Processing Settings

Post‑Processing

Pipeline: Pipeline to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the LogStream Pipeline that processed the event). Supports wildcards. Other options include:

  • cribl_host – LogStream Node that processed the event.
  • cribl_wp – LogStream Worker Process that processed the event.
  • cribl_input – LogStream Source that processed the event.
  • cribl_output – LogStream Destination that processed the event.

Advanced Settings

Max file size (MB): Maximum uncompressed output file size. Files reaching this size will be closed and moved to the final output location. Defaults to 32.

Max file open time (sec): Maximum amount of time to write to a file. Files open for longer than this limit will be closed and moved to final output location. Defaults to 300.

Max file idle time (sec): Maximum amount of time to keep inactive files open. Files open for longer than this limit will be closed and moved to final output location. Default: 30.

Max open files: Maximum number of files to keep open concurrently. When exceeded, the oldest open files will be closed and moved to final output location. Default: 100.

LogStream will close files when either of the Max file size (MB) or the Max file open time (sec) conditions are met.

Add Output ID: When set to Yes (the default), adds the Output ID field's value to the staging location's file path. This ensures that each Destination's logs will write to its own bucket.

For a Destination originally configured in a LogStream version below 2.4.0, the Add Output ID behavior will be switched off on the backend, regardless of this slider's state. This is to avoid losing any files pending in the original staging directory, upon LogStream upgrade and restart. To enable this option for such Destinations, Cribl's recommended migration path is:

  • Clone the Destination.
  • Redirect the Routes referencing the original Destination to instead reference the new, cloned Destination.

This way, the original Destination will process pending files (after an idle timeout), and the new, cloned Destination will process newly arriving events with Add output ID enabled.

Remove staging dirs: Toggle to Yes to delete empty staging directories after moving files. This prevents the proliferation of orphaned empty directories.

Environment: If you're using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Internal Fields

Cribl LogStream uses a set of internal fields to assist in forwarding data to a Destination.

Field for this Destination:

  • __partition