Cribl Edge supports sending data directly to Google BigQuery using the BigQuery Storage Write API. This eliminates the need to stage data in Google Cloud Storage first, reducing end-to-end latency significantly.

Type: Streaming | TLS Support: Yes | PQ Support: Yes

Configure Cribl Edge to Output to Google BigQuery ​

1. On the top bar, select Products, and then select Cribl Edge. Under Fleets, select a Fleet. Next, you have two options:
   • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Destination and select the Destination you want from the list, choosing either Select Existing or Add New.
   • To configure via the Routes, select Data > Destinations or More > Destinations (Edge). Select the Destination you want. Next, select Add Destination.
2. In the New Destination modal, configure the following under General Settings:
   • Output ID: Enter a unique name to identify this BigQuery destination. If you clone this Destination, Cribl Edge will add -CLONE to the original Output ID.
   • Description: Optionally, enter a description.
   • Project ID: The Google Cloud project that contains the target BigQuery dataset.
   • Dataset ID: The BigQuery dataset to write to.
   • Table ID: The BigQuery table to write to. The table must already exist; Cribl Edge does not create tables automatically.
3. Next, you can configure the following Optional Settings:
   • Backpressure behavior: Whether to block, drop, or queue events when all receivers are exerting backpressure. Defaults to Block.
   • Tags: Optionally, add tags that you can use to filter and group Destinations on the Destinations page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.
4. Optionally, you can adjust the Authentication, Persistent Queue, Processing, and Advanced settings outlined in the sections below.
5. Select Save, then Commit & Deploy.

Authentication ​

Use the Authentication method drop-down to select one of these options:

• Auto: Uses Google Application Default Credentials (ADC) with no manual configuration. Cribl Edge checks the following credential sources in order:

  1. The GOOGLE_APPLICATION_CREDENTIALS environment variable, which should point to a service account key file or contain the credentials JSON directly.
  2. Credentials attached to the Google Cloud instance (workload identity federation or the instance metadata server).

  Use this option when running Cribl Edge on Google Cloud infrastructure or when credentials are managed externally.

• Secret: Select a stored text secret containing the service account credentials JSON. Use the Create link to store a new, reusable secret.

The Manual option (entering credentials inline) is not available for this Destination. Store service account credentials as a secret and use the Secret option instead.

Persistent Queue Settings ​

The Persistent Queue Settings tab displays when the Backpressure behavior option in General settings is set to Persistent Queue. Persistent queue buffers and preserves incoming events when a downstream Destination has an outage or experiences backpressure.

Before enabling persistent queue, learn more about persistent queue behavior and how to optimize it with your system:

• About Persistent Queues
• Optimize Destination Persistent Queues
• About Destination Backpressure Triggers

Some of these settings might not appear if you set the persistent queue storage mode to Network filesystem or AWS S3 at the Worker Group level (Worker Group Settings > System > PQ Storage). When using these storage types, this behavior is managed at the Worker Group level. See Worker Group PQ Storage Fields for more information.

On Cribl-managed Cloud Workers (with an Enterprise plan), this tab exposes only the destructive Clear Persistent Queue button (described at the end of this section). A maximum queue size of 1 GB disk space is automatically allocated per PQ-enabled Destination, per Worker Process. The 1 GB limit is on outbound uncompressed data, and no compression is applied to the queue.

This limit is not configurable. If the queue fills up, Cribl Edge blocks outbound data. To configure the queue size, compression, queue-full fallback behavior, and other options below, use a hybrid Group.

Mode: Use this menu to select when Cribl Edge engages the persistent queue in response to backpressure events from this Destination. The options are:

If a Worker/Edge Node starts with an invalid Mode setting, it automatically switches to Error mode. This might happen if the Worker/Edge Node is running a version that does not support other modes (older than 4.9.0), or if it encounters a nonexistent value in YAML configuration files.

File size limit: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, and so on. Defaults to 10 MB. When you save the configuration, File size limit must be greater than or equal to Buffer size limit (bytes). Cribl Edge rejects invalid combinations.

Queue size limit: The maximum amount of disk space that the queue can consume on each Worker Process. When the queue reaches this limit, the Destination stops queueing data and applies the Queue-full behavior. Defaults to 5 GB. This field accepts positive numbers with units of KB, MB, GB, and so on. You can set it as high as 1 TB, unless you’ve configured a different Worker Process PQ size limit on the Worker Group/Fleet Settings page.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. Cribl Edge appends /<worker-id>/<output-id> to this value.

Compression: Set the codec to use when compressing the persisted data after closing a file. Defaults to None. Gzip is also available.

Queue-full behavior: Whether to block or drop events when the queue begins to exert backpressure. A queue begins to exert backpressure when the disk is low or at full capacity. This setting has two options:

• Block: The output will refuse to accept new data until the receiver is ready. The system will return block signals back to the sender.
• Drop new data: Discard all new events until the backpressure event has resolved and the receiver is ready.

Buffer size limit (bytes): The maximum memory to buffer events before flushing them to persistent queue on disk. Enter a value with a unit suffix. For example: 64KB, 1MB, or 10MB. Spaces between the number and unit are allowed (such as 64 KB). Units are case-insensitive. Plain byte values (like 65536) are accepted, but decimals (like 0.5MB) are not. The valid range is 64KB to 10MB. Values outside this range are rounded to the nearest limit. Defaults to 1MB.

Deprecation Notice

The Buffer size limit (bytes) setting replaces the deprecated Max buffer size setting to provide more predictable memory management in version 4.18.0. The Max buffer size setting will be removed in version 4.19.1. For upgraded Worker Groups and Fleets, the new byte-based limit defaults to 1MB. Update your configurations to the new byte-based limit to ensure optimal memory stability.

Strict ordering: Toggle on (default) to enable FIFO (first in, first out) event forwarding, ensuring Cribl Edge sends earlier queued events first when receivers recover. The persistent queue flushes every 10 seconds in this mode. Toggle off to prioritize new events over queued events, configure a custom drain rate for the queue, and display this option:

• Drain rate limit (EPS): Optionally, set a throttling rate (in events per second) on writing from the queue to receivers. (The default 0 value disables throttling.) Throttling the queue drain rate can boost the throughput of new and active connections by reserving more resources for them. You can further optimize Worker startup connections and CPU load in the Worker Processes settings.

Clear Persistent Queue: For Cloud Enterprise only, select this button if you want to delete the files that are currently queued for delivery to this Destination. If you select this button, a confirmation modal appears. Clearing the queue frees up disk space by permanently deleting the queued data, without delivering it to downstream receivers. This button only appears after you define the Output ID.

Use the Clear Persistent Queue button with caution to avoid data loss. See Steps to Safely Disable and Clear Persistent Queues for more information.

Processing Settings ​

Post-Processing ​

Pipeline: Pipeline or Pack to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the Cribl Edge Pipeline that processed the event). Supports wildcards. Other options include:

• cribl_host - Cribl Edge Node that processed the event.
• cribl_input - Cribl Edge Source that processed the event.
• cribl_output - Cribl Edge Destination that processed the event.
• cribl_route - Cribl Edge Route (or QuickConnect) that processed the event.
• cribl_wp - Cribl Edge Worker Process that processed the event.

Advanced Settings ​

Timestamp column: Name of the BigQuery column that receives event timestamps. When set, Cribl Edge converts each event’s _time field (Unix epoch, in seconds) to a BigQuery TIMESTAMP value (microsecond precision) and writes it to this column. The column must exist in the table schema and be of type TIMESTAMP. If the column name differs from _time, the _time field is omitted from the row. Use this setting when the table is partitioned by a timestamp column.

Queue size limit: Maximum number of batches that can queue before Cribl Edge applies backpressure. Defaults to 100. Minimum: 1.

Batch size limit (KB): Maximum size of a single append request sent to BigQuery. BigQuery enforces a 10 MB per-request limit. Defaults to 1024 KB. Minimum: 1, Maximum: 10240.

Flush period (sec): Maximum time to wait before sending a batch when the configured Batch size limit has not been reached. Defaults to 1 second. Minimum: 1.

Concurrent request limit: Maximum number of in-progress API requests before Cribl Edge applies backpressure. Defaults to 5. Minimum: 1, Maximum: 100.

Max send retries: Maximum number of retries per batch for retryable failures before the batch is dropped. Set to 0 to retry indefinitely. Defaults to 0. Minimum: 0.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Delivery Semantics ​

Cribl Edge writes to BigQuery using the BigQuery Storage Write API default write stream, which provides at-least-once delivery. Rows may be duplicated in the event of a retry. If your downstream queries are sensitive to duplicates, use a deduplication key in your table design or query logic.

Event-to-Row Mapping ​

When Cribl Edge writes an event to BigQuery, it maps each event field to a column in the target table:

• All event fields are included as columns, except fields with names that start with __ (internal Cribl Edge metadata fields).
• If Timestamp column is configured, the event’s _time field is written to that column as a BigQuery TIMESTAMP. If the column name differs from _time, the _time field itself is omitted from the row.
• The target table schema is fetched at initialization. Columns present in the event but absent from the table schema are silently dropped by BigQuery.

Google Cloud Roles and Permissions ​

Your Google Cloud service account must have at least the following IAM roles on the target dataset:

• roles/bigquery.dataEditor – grants bigquery.tables.updateData and bigquery.tables.get, which are the only permissions required by the Storage Write API.

For details on granting this role, see the Google Cloud documentation on BigQuery access control.

Troubleshooting ​

The Destination’s configuration modal has helpful tabs for troubleshooting:

Live Data: Try capturing live data to see real-time events as they flow through the Destination. On the Live Data tab, click Start Capture to begin viewing real-time data.

Logs: Review and search the logs that provide detailed information about the delivery process, including any errors or warnings that may have occurred.

Test: Ensures that the Destination is correctly set up and reachable. Verify that sample events are sent correctly by clicking Run Test.

You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify delivery issues. Analyze the graphs showing events and bytes in/out over time.