On This Page

Home / Stream/ Integrations/ Sources/ Azure/Azure VNet Flow Logs Source

Azure VNet Flow Logs Source ​

The Azure VNet Flow Logs Source in Cribl Stream ingests Azure Virtual Network (VNet) flow logs from Azure Blob Storage. Use this Source to process new flow records as Azure adds them to hourly block blobs, without downloading the entire growing blob after every update. Cribl recommends a Storage queue that receives only VNet flow log blob notifications. The Source dequeues every message from the configured queue. Non-VNet messages go back on the queue, or are permanently deleted after they reach Max dequeue count.

Type: Pull | TLS Support: Yes | Event Breaker Support: Yes

This Source is designed for Azure VNet flow log .json block blobs updated through PutBlockList. For block blobs that are written once, use the Azure Blob Storage Source.

Prerequisites ​

Before you configure this Source, you need:

Configuring an Azure VNet Flow Logs Source ​

  1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
    • To configure via QuickConnect, navigate to Routing > QuickConnect. Select Add Source, select Azure VNet Flow Logs, and then select Add New.
    • To configure via the Routes, select Data > Sources. Select Azure VNet Flow Logs, and then select Add Source.
  2. Under General Settings, configure:
    • Input ID: Enter a unique name for this Source.
    • Description: Optionally, enter a description.
    • Queue: Enter the Azure Storage queue name as a JavaScript expression enclosed in single quotes, double quotes, or backticks. The expression is evaluated only when the Source initializes. For example: 'vnet-flow-logs'. Cribl recommends a queue that receives only VNet flow log blob notifications.
  3. In the Authentication section, select an authentication method.
  4. Under Optional Settings, configure:
    • Filename filter: Optionally, enter a regular expression that matches the VNet flow log blob paths to process. Defaults to /.*/.
    • Tags: Optionally, add tags that you can use to filter and group Sources in the Stream UI. Tags are not added to events.
  5. Optionally, configure Processing Settings, Advanced Settings, and Connected Destinations.
  6. Select Save, then Commit & Deploy.

Authentication ​

Select an Authentication method:

  • Text Secret (default): For Connection string (text secret), choose Select local secret or Select global secret, then select or create a stored text secret that contains your Azure Storage connection string.
  • Client Secret: Authenticate with a Microsoft Entra service principal and a stored client secret.
  • Client Certificate: Authenticate with a Microsoft Entra service principal and a stored certificate.

Worker Nodes running Cribl Stream version 4.8.1 and older do not support client secret/certificate authentication and will fail to start with this configuration.

Service Principal Settings ​

Client Secret and Client Certificate expose these settings:

  • Storage account name: Name of the Azure Storage account.
  • Tenant ID: Microsoft Entra tenant ID for the service principal.
  • Client ID: Application ID for the service principal.
  • Client secret (text secret) (with Client Secret selected): Choose Select local secret or Select global secret, then select or create a stored text secret that contains the service principal’s client secret.
  • Certificate (with Client Certificate selected): Select the certificate you registered as credentials for your app in the Azure portal, or select Create.
  • Azure Cloud: Select Azure Public Cloud, Azure China, or Azure Government. If you leave this field empty, Stream uses Azure Public Cloud. To override the endpoint suffix, use Endpoint suffix in Advanced Settings.
    • Selecting Azure China sets the authority host to login.chinacloudapi.cn and the endpoint suffix to core.chinacloudapi.cn.
    • Selecting Azure Government sets the authority host to login.microsoftonline.us and the endpoint suffix to core.usgovcloudapi.net.

Processing Settings ​

Event Breakers ​

The Source applies Azure Ruleset by default. This ruleset includes the Azure VNet Flow Event Breaker, which converts each comma-separated flow tuple into a structured event and preserves its parent VNet metadata.

Event Breaker rulesets: Rulesets to apply, in order, before data enters Routes and Pipelines. Select Add Ruleset to apply additional rulesets.

Event Breaker buffer timeout: How long the Event Breaker waits for more data on a channel before flushing it. Defaults to 10000 ms.

Fields ​

In this section, define new fields or modify existing fields using JavaScript expressions, similar to the Eval function.

  • Field Name: Name of a new field or an existing field to modify.
  • Value: JavaScript expression that computes the field value.

Pre-Processing ​

Select a Pipeline or Pack to process data before it is sent through Routes.

Advanced Settings ​

Message limit: Maximum messages to request from the Azure Storage queue in one poll. Defaults to 1. Range: 1 to 32. Cribl recommends leaving this at 1. Higher values pull more messages per poll and increase the chance that the same blob is processed in parallel when the queue is busy, which can duplicate data.

Visibility timeout (secs): How long a received message is hidden from other queue consumers. Defaults to 600. Range: 0 to 604800. Stream extends this timeout while a blob is still being processed.

Number of receivers: Number of receiver processes to run. Defaults to 1. Range: 1 to 100. Higher values can increase throughput at the expense of CPU. In burst scenarios, a high value also increases the chance of duplicated data.

Max dequeue count: Number of times a nonmatching message can be dequeued before it is permanently deleted. Defaults to 1, which deletes nonmatching messages immediately, matching the Azure Blob Storage Source’s behavior. Range: 1 to 100. Cribl recommends a dedicated VNet flow log queue. If you share a queue, increase this value so non-VNet messages can be returned for other consumers instead of being deleted immediately.

Service period (secs): How often Stream checks pollers and restarts them if they have exited. Defaults to 5. Range: 1 to 10.

Endpoint suffix: Available with the Client Secret and Client Certificate authentication methods. Optionally, override the Storage endpoint suffix that Azure Cloud selects. Defaults to core.windows.net. Use this setting only for a custom or nonstandard Azure Blob Storage endpoint.

Environment: For GitOps, optionally limit this configuration to a single Git branch.

Connected Destinations ​

Send to Routes: Use the Routing table for conditional routing, filtering, and cloning.

QuickConnect: Send this Source’s output directly to one or more Destinations.

Internal Fields ​

Stream adds these internal fields:

  • __accountName
  • __final
  • __inputId
  • __source
  • __topic
  • _time

You can use internal fields in Functions, but they are not part of the original Azure event.

Troubleshooting ​

No events: Confirm that the queue receives Event Grid Blob Created messages whose data.api value is PutBlockList. Also confirm that the event has a blob URL, a valid contentLength, and a .json path that matches Filename filter.

Source health is yellow: The Source could not retrieve offset state. Messages remain on the queue for retry until the state service becomes available.

Repeated data: Confirm that only one writer appends records to each blob. After offset state expires (two hours), the next event requires a full download and can replay records from the current blob.

Authentication errors: Confirm that the selected credentials can read blobs and receive and delete messages from the Storage queue.