Cribl Stream supports receiving data from Azure Event Hubs using the native AMQP protocol. This Source is separate from the Kafka-based Azure Event Hubs Source.

Type: Pull | TLS Support: Yes | Event Breaker Support: No

Azure Event Hubs AMQP uses a binary protocol over TCP. It does not support HTTP proxies, so Cribl Stream must receive events directly from senders. You might need to adjust your firewall rules to allow this traffic.

Prerequisites ​

Before configuring this Source, ensure you have the following Azure permissions:

• Azure Event Hubs Data Receiver on the Event Hub namespace.
• Storage Blob Data Contributor on the checkpoint container, if using Azure Blob checkpointing with a Microsoft Entra service principal (Client Secret or Client Certificate authentication).

For details, see Authorize access to Azure Event Hubs resources using Microsoft Entra ID in the Microsoft documentation.

Why AMQP over Kafka for Event Hubs? ​

AMQP is the core protocol for Azure Event Hubs, and Microsoft actively maintains the underlying SDK. The AMQP stores consumer group offsets externally in Azure Blob Storage rather than inside the Event Hub itself. This lets multiple independent applications (for example, production and development environments) share the same consumer group using separate checkpoint containers, without interfering with each other.

Configure Cribl Stream to Receive Data from Azure Event Hubs (AMQP) ​

1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
   • To configure via QuickConnect, navigate to Routing > QuickConnect. Select Add Source and select the Source you want from the list, choosing either Select Existing or Add New.
   • To configure via the Routes, select Data > Sources. Select the Source you want. Next, select Add Source.
2. In the New Source modal, configure the following under General Settings:
   • Input ID: Enter a unique name to identify this Source definition. If you clone this Source, Cribl Stream will add -CLONE to the original Input ID.
   • Description: Optionally, enter a description.
   • Consumer group: The name of the Event Hubs consumer group this Cribl Stream instance belongs to. Each consumer group maintains its own independent set of offsets. Defaults to $Default.
3. Next, you can configure the following Optional Settings:
   • Event Hub name: The name of the Azure Event Hub to consume from. You can omit this field if the Event Hub name is embedded in the connection string via an EntityPath= parameter.
   • From beginning: When toggled on (default), Cribl Stream starts reading from the earliest available event position on initial subscription. When toggled off, reading starts from the latest position. This setting applies only when no checkpoint data is available.
   • Tags: Optionally, add tags to filter and group Sources in Cribl Stream’s UI. These tags are not added to processed events.
4. Configure Authentication, Checkpointing, Processing Settings, Retries, and Advanced Settings as needed.
5. Select Save, then Commit & Deploy.

Authentication ​

Authentication mechanism: Select how to authenticate to Azure Event Hubs. You can choose between Connection String and OAuth Bearer.

Select Connection String to authenticate via a connection string. This exposes the following fields:

• Connection string (text secret): Select a stored text secret that contains the connection string for your Event Hubs namespace or Event Hub. A Create link is available to define a new secret. The connection string uses the format Endpoint=sb://<FQDN>/;SharedAccessKeyName=<key-name>;SharedAccessKey=<key-value>.

If the connection string includes an EntityPath=<event-hub-name> segment, you can omit the Event Hub name field in General Settings.

Select OAuth Bearer to authenticate using Microsoft Entra ID (formerly Azure Active Directory). This exposes the following fields:

• Microsoft Entra ID authentication endpoint: The Microsoft Entra authority host URL for acquiring tokens. Defaults to https://login.microsoftonline.com. You can instead select https://login.microsoftonline.us or https://login.partner.microsoftonline.cn.

• Client ID: The client ID (application ID) of your registered application.

• Tenant identifier: Your Microsoft Entra ID directory (tenant) ID.

• Fully qualified namespace: The fully qualified hostname of your Event Hubs namespace, for example mynamespace.servicebus.windows.net.

• Authentication method: Select how to provide the client credential. You can choose between Secret and Certificate.

Checkpointing ​

Checkpointing persists consumer group partition offsets so that Cribl Stream can resume from its last position after a restart. It also lets multiple independent applications share the same consumer group with different checkpoint containers.

Configure the following fields:

• Container name: The name of the Azure Blob Storage container to use for checkpoint data. Must be 3-63 lowercase alphanumeric characters or hyphens, with no consecutive hyphens.
• Authentication method: Select how to authenticate to the blob storage account. You can choose among Text Secret, Client Secret, and Client Certificate.

Select Text Secret to authenticate using a storage account connection string. This exposes the following field:

• Connection string (text secret): Select a stored text secret that contains the storage account connection string. A Create link is available to define a new secret.

Select Client Secret to authenticate using a Microsoft Entra service principal with a client secret. This exposes the following fields:

• Storage account name: The name of the Azure Storage account.
• Tenant ID: Your Microsoft Entra directory (tenant) ID.
• Client ID: The client ID of your registered application.
• Azure Cloud: Optionally, select the Azure cloud environment. If not set, defaults to the Azure public cloud.
• Endpoint suffix: Optionally, enter a custom storage endpoint suffix for sovereign or private cloud deployments.
• Client secret (text secret): Select a stored text secret that contains the client secret. A Create link is available to define a new secret.

Select Client Certificate to authenticate using a Microsoft Entra service principal with a certificate. This exposes the following fields:

• Storage account name: The name of the Azure Storage account.
• Tenant ID: Your Microsoft Entra directory (tenant) ID.
• Client ID: The client ID of your registered application.
• Azure Cloud: Optionally, select the Azure cloud environment. If not set, defaults to the Azure public cloud.
• Endpoint suffix: Optionally, enter a custom storage endpoint suffix for sovereign or private cloud deployments.
• Certificate: Select a stored certificate. A Create link is available to define a new certificate.

Retries ​

These settings control how the Azure Event Hubs SDK retries individual failed operations (such as receiving a batch).

Retry limit: Maximum number of times to retry a failed operation before treating it as a failure. Defaults to 5. Valid range is 0-100.

Initial retry interval (ms): Starting delay before the first retry, in milliseconds. The delay grows exponentially with each retry up to Backoff limit. Defaults to 300 ms. Valid range is 300-600,000 ms.

Backoff limit (ms): Maximum delay between retries, in milliseconds. Defaults to 30,000 ms. Valid range is 30,000-180,000 ms.

Request timeout (ms): Maximum time to wait for a single operation to complete before timing out, in milliseconds. Defaults to 60,000 ms. Valid range is 1,000-3,600,000 ms.

Advanced Settings ​

Use these settings to tune how Cribl Stream pulls batches from the Event Hub and how it reconnects after connection-level failures.

Max batch size: Maximum number of events to receive in a single batch. Defaults to 100.

Max wait time (secs): Maximum time in seconds to wait for a full batch before processing a partial batch. Defaults to 60.

Prefetch count: Number of events to prefetch and buffer locally from the Event Hub. Higher values can increase throughput at the cost of memory. Defaults to 300.

Connection initial retry interval (ms): Starting delay before the first reconnect attempt after a connection-level failure, in milliseconds. The delay grows exponentially with each attempt up to Connection backoff limit. Defaults to 1,000 ms. Valid range is 1-600,000 ms.

Connection backoff limit (ms): Maximum delay between reconnect attempts, in milliseconds. Defaults to 64,000 ms. Valid range is 1-180,000 ms.

Connection timeout (ms): Maximum time allowed when establishing the initial connection to the Event Hub, in milliseconds. Defaults to 10,000 ms. Valid range is 1,000-3,600,000 ms.

Environment: If you’re using GitOps, optionally specify a single Git branch on which to enable this configuration. If empty, the configuration is enabled everywhere.

Processing Settings ​

Fields ​

In this section, you can define new fields or modify existing ones using JavaScript expressions, similar to the Eval function.

• The Field Name can either be a new field (unique within the event) or an existing field name to modify its value.
• The Value is a JavaScript expression (enclosed in quotes or backticks) to compute the field’s value (can be a constant). Select this field’s advanced mode icon (far right) if you’d like to open a modal where you can work with sample data and iterate on results.

This flexibility means you can:

• Add new fields to enrich the event.
• Modify existing fields by overwriting their values.
• Compute logic or transformations using JavaScript expressions.

Pre-Processing ​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline or Pack to process data from this input before the data is sent through the Routes.

Connected Destinations ​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

How Cribl Stream Pulls Data ​

Cribl Stream uses the Azure Event Hubs AMQP protocol to subscribe to partitions in a consumer group. The @azure/event-hubs SDK automatically load-balances partition ownership across all Cribl Stream Worker Nodes that are members of the same consumer group.

If Cribl Stream logs No partitions owned -- this consumer may be idle due to more consumers than partitions, there are more Worker Nodes consuming from this Source than there are partitions in the Event Hub. This is expected and does not indicate an error.

When checkpointing is configured with Azure Blob Storage, each Worker Node commits its current offset for each owned partition into the Blob container. On restart, Cribl Stream resumes from the last committed checkpoint rather than replaying from the beginning.

How Cribl Stream Handles the _time Field ​

Events emitted by this Source always contain a _time field, and sometimes also an __origTime field. Cribl Stream determines what to set as follows:

• If the incoming event body contains a _time field whose value is a Unix timestamp, that value becomes the emitted event’s _time.
• If the incoming event body contains no _time field, the Event Hubs enqueuedTimeUtc timestamp becomes the emitted event’s _time.
• If the incoming event body contains a _time field whose value is not a Unix timestamp (for example, an ISO or UTC timestamp), that value is moved to __origTime, and enqueuedTimeUtc becomes _time.

Internal Fields ​

Cribl Stream uses a set of internal fields to assist in handling of data. These “meta” fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

Fields for this Source:

• __inputId
• __offset – byte offset of the event within its Event Hub partition log.
• __sequenceNumber – monotonically increasing sequence number assigned by Event Hubs when the event is stored in a partition.
• __partitionKey – optional key used by the producer to route the event to a specific partition. Only present when set by the producer.

Troubleshooting ​

The Source’s configuration modal has helpful tabs for troubleshooting:

Live Data: Try capturing live data to see real-time events as they are ingested. On the Live Data tab, click Start Capture to begin viewing real-time data.

Logs: Review and search the logs that provide detailed information about the ingestion process, including any errors or warnings that may have occurred.

You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify ingestion issues. Analyze the graphs showing events and bytes in/out over time.

Common Issues ​