Cribl LogStream supports receiving data records from Azure Event Hubs.
Type: Pull | TLS Support: YES (secure API) | Event Breaker Support: No
Configuring LogStream to Receive Data from Azure Event Hubs
In the QuickConnect UI: Click + New Source, or click + Add beside Sources. From the resulting drawer's tiles, select [Pull >] Azure > Event Hubs. Next, click either + Add New or (if displayed) Select Existing. The drawer will now provide the following options and fields.
Or, in the Data Routes UI: From the top nav of a LogStream instance or Group, select Data > Sources. From the resulting page's tiles or the Sources left nav, select [Pull >] Azure > Event Hubs. Next, click + Add New to open a New Source modal that provides the following options and fields.
Input ID: Enter a unique name to identify this source definition.
Brokers: List of Event Hubs Kafka brokers to connect to, e.g.,
yourdomain.servicebus.windows.net:9093. Get the hostname from the host portion of the primary or secondary connection string in Shared Access Policies.
Event Hub name: The name of the Event Hub (a.k.a. Kafka Topic) to subscribe to.
Group ID: The name of the consumer group that includes this LogStream instance. Defaults to
To prevent excessive Kafka rebalancing and reduced throughput, each Group ID that you specify here should be subscribed to only one Kafka Topic – i.e., only to the single Topic you specify in Event Hub name. This has two implications:
- The Group ID should be something other than
$Default, especially if Event Hubs are stored In shared accounts, where the
$Defaultgroup might be subscribed to other Topics.
- You should configure a separate Azure Event Hubs Source for each Group:Topic pair whose events you want to subscribe to.
From beginning: Whether to start reading from the earliest available data. Relevant only during initial subscription. Defaults to
TLS Settings (Client Side)
Enabled: Defaults to
Validate server certs: Whether to reject connections to servers without signed certificates. Defaults to
No – and for Event Hubs, must always be disabled.
Enabled: Defaults to
No. When toggled to
Yes, all the settings in this section are required.
SASL mechanism: SASL (Simple Authentication and Security Layer) authentication mechanism to use. Currently,
PLAINis the only mechanism supported for Event Hubs Kafka brokers.
Username: The username for authentication. For Event Hubs, this should always be
Use the Authentication method buttons to select one of these options:
Manual: Use this default option to enter your Event Hubs connection string. Exposes a Password field for this purpose.
Secret: This option exposes a Connection string (text secret) drop-down, in which you can select a stored secret that references an Event Hubs connection string. The secret can reside in LogStream's internal secrets manager or (if enabled) in an external KMS. A Create link is available if you need a new secret.
Connection String Format
Either authentication method uses an Azure Event Hubs connection string in this format:
A fictitious example, is:
In this section, you can add fields/metadata to each event using Eval-like functionality.
Name: Field name.
In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.
Use these settings to fine-tune LogStream's integration with Event Hubs Kafka brokers. For details, see Azure Event Hubs' recommended configuration documentation. If you are unfamiliar with these parameters, contact Cribl Support to understand the implications of changing the defaults.
Session timeout (ms): Timeout used to detect client failures when using Kafka's group management facilities. (Corresponds to
session.timeout.ms in the Kafka domain.) If the client sends the broker no heartbeats before this timeout expires, the broker will remove this client from the group, and will initiate a rebalance. Value must be lower than
rebalanceTimeout. Defaults to
30000 ms, i.e., 30 seconds.
Rebalance timeout (ms): Maximum allowed time for each worker to join the group after a rebalance has begun. (Corresponds to
rebalance.timeout.ms in the Kafka domain.) If this timeout is exceeded, the coordinator broker will remove the worker from the group. Defaults to
60000 ms, i.e., 1 minute.
Heartbeat interval (ms): Expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. (Corresponds to
heartbeat.interval.ms in the Kafka domain.) Value must be lower than
sessionTimeout, and typically should not exceed 1/3 of the
sessionTimeout value. Defaults to
3000 ms, i.e., 3 seconds.
If you observe an excessive number of group rebalances, and/or you observe consumers not regularly pulling messages, try increasing the values of all three of the above parameters.
Environment: If you're using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.
Select Send to Routes to enable conditional routing, filtering, and cloning of this Source's data via the Routing table.
Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.
How LogStream Pulls Data
Azure Event Hubs treat all the Worker Nodes as members of a Consumer Group, and each Worker gets its share of the load from Azure Event Hubs. This is the same process as normal Kafka. By default, Workers will poll every 5 seconds. In the case of Leader failure, Worker Nodes will continue to receive data as normal.