These docs are for Cribl Stream 4.11 and are no longer actively maintained.

See the latest version (4.13).

Google Cloud Pub/Sub Destination

Cribl Stream supports sending data to Google Cloud Pub/Sub, a managed real-time messaging service for sending and receiving messages between applications.

Type: Streaming | TLS Support: Yes | PQ Support: Yes

Prerequisites

​

When working with Google Cloud Pub/Sub, you need to know:

• The project where your credentials originated.
• The project that contains any topic you want to work with.

If the desired topic resides in the same project where your credentials originated, describe it using either its short name or its fully-qualified name.

If the desired topic resides in a different project than the one where your credentials originated, you must describe it using its fully-qualified name.

Here’s an example of a fully-qualified topic name: projects/my-project-id/topics/my-topic-id.

The short name for the same topic would be: my-topic-id.

Configure Cribl Stream to Output to Pub/Sub

​

1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
   • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Destination and select the Destination you want from the list, choosing either Select Existing or Add New.
   • To configure via the Routes, select Data > Destinations or More > Destinations (Edge). Select the Destination you want. Next, select Add Destination.
2. In the New Destination modal, configure the following under General Settings:
   • Output ID: Enter a unique name to identify this Pub/Sub output definition. If you clone this Destination, Cribl Stream will add -CLONE to the original Output ID.
   • Description: Optionally, enter a description.
   • Topic ID: ID of the Pub/Sub topic to send events to. This static initial configuration of the topic is required; but you can optionally also override it dynamically on a per-event basis.
3. Next, you can configure the following Optional Settings:

• Create topic: Toggle on if you want Cribl Stream to create the topic on Pub/Sub if it does not exist.
• Ordered delivery: Toggle on if you want Cribl Stream to send events in the order that they arrived in the queue. For ordered delivery to work correctly, ensure that both the process receiving events and the subscription from which messages are read have ordered delivery enabled.
• Region: Region to publish messages to. Select default to allow Google to auto-select the nearest region. (If you’ve enabled Ordered delivery, the selected region must be allowed by message storage policy.)
• Backpressure behavior: Whether to block, drop, or queue events when all receivers are exerting backpressure. Defaults to Block.
• Tags: Optionally, add tags that you can use to filter and group Destinations on the Destinations page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.

4. Optionally, you can adjust the Authentication, Persistent Queue, Processing, and Advanced settings outlined in the sections below.
5. Select Save, then Commit & Deploy.

Authentication

​

Use the Authentication method drop-down to select one of these options:

• Auto: This option uses the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the location of a credential JSON file. The environment variable can also be set directly in JSON format. When this option is set, Cribl Stream will first check whether the GOOGLE_APPLICATION_CREDENTIALS environment variable exists and properly set with valid credentials. If this environmental variable is missing or incorrect, Cribl Stream checks for credentials in two additional fallback locations, as described in the official Google Cloud documentation: How Application Default Credentials works. In addition to using the GOOGLE_APPLICATION_CREDENTIALS environment variable, you can use credentials attached to your Google Cloud instance.

• Manual: This default option displays a Service account credentials field for you to enter the contents of your service account credentials file (a set of JSON keys), as downloaded from Google Cloud.
  To insert the file itself, select the upload button at this field’s upper right. As an alternative, you can use environment variables, as outlined here.

• Secret: This option exposes a drop-down in which you can select a stored secret that references the service account credentials described above. A Create link is available to store a new, reusable secret.

Persistent Queue Settings

​

The Persistent Queue Settings tab displays when the Backpressure behavior option in General settings is set to Persistent Queue. Persistent queue buffers and preserves incoming events when a downstream Destination has an outage or experiences backpressure.

Before enabling persistent queue, learn more about persistent queue behavior and how to optimize it with your system:

• About Persistent Queues
• Optimize Destination Persistent Queues (dPQ)
• Destination Backpressure Triggers

On Cribl-managed Cloud Workers (with an Enterprise plan), this tab exposes only the destructive Clear Persistent Queue button (described at the end of this section). A maximum queue size of 1 GB disk space is automatically allocated per PQ‑enabled Destination, per Worker Process. The 1 GB limit is on outbound uncompressed data, and no compression is applied to the queue.

This limit is not configurable. If the queue fills up, Cribl Stream/Edge will block outbound data. To configure the queue size, compression, queue-full fallback behavior, and other options below, use a hybrid Group.

Mode: Use this menu to select when Cribl Stream/Edge engages the persistent queue in response to backpressure events from this Destination. The options are:

If a Worker/Edge Node starts with an invalid Mode setting, it automatically switches to Error mode. This might happen if the Worker/Edge Node is running a version that does not support other modes (older than 4.9.0), or if it encounters a nonexistent value in YAML configuration files.

File size limit: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, etc. Defaults to 1 MB.

Max queue size: The maximum amount of disk space that the queue can consume on each Worker Process. When the queue reaches this limit, the Destination stops queueing data and applies the Queue‑full behavior. Defaults to 5 GB. This field accepts positive numbers with units of KB, MB, GB, and so on. You can set it as high as 1 TB, unless you’ve configured a different Worker Process PQ size limit on the Group Settings/Fleet Settings page.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. Cribl Stream/Edge will append /<worker‑id>/<output‑id> to this value.

Compression: Set the codec to use when compressing the persisted data after closing a file. Defaults to None. Gzip is also available.

Queue-full behavior: Whether to block or drop events when the queue begins to exert backpressure. A queue begins to exert backpressure when the disk is low or at full capacity. This setting has two options:

• Block: The output will refuse to accept new data until the receiver is ready. The system will return block signals back to the sender.
• Drop new data: Discard all new events until the backpressure event has resolved and the receiver is ready.

Backpressure duration Limit: When Mode is set to Backpressure, this setting controls how long to wait during network slowdowns before activating queues. A shorter duration enhances critical data loss prevention, while a longer duration helps avoid unnecessary queue transitions in environments with frequent, brief network fluctuations. The default value is 30 seconds.

Strict ordering: Toggle on (default) to enable FIFO (first in, first out) event forwarding, ensuring Cribl Stream/Edge sends earlier queued events first when receivers recover. The persistent queue flushes every 10 seconds in this mode. Toggle off to prioritize new events over queued events, configure a custom drain rate for the queue, and display this option:

• Drain rate limit (EPS): Optionally, set a throttling rate (in events per second) on writing from the queue to receivers. (The default 0 value disables throttling.) Throttling the queue drain rate can boost the throughput of new and active connections by reserving more resources for them. You can further optimize Worker startup connections and CPU load in the Worker Processes settings.

Clear Persistent Queue: For Cloud Enterprise only, click this button if you want to delete the files that are currently queued for delivery to this Destination. If you click this button, a confirmation modal appears. Clearing the queue frees up disk space by permanently deleting the queued data, without delivering it to downstream receivers. This button only appears after you define the Output ID.

Use the Clear Persistent Queue button with caution to avoid data loss. See Steps to Safely Disable and Clear Persistent Queues for more information.

Processing Settings

​

Post‑Processing

​

Pipeline: Pipeline or Pack to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the Cribl Stream Pipeline that processed the event). Supports wildcards. Other options include:

• cribl_host – Cribl Stream Node that processed the event.
• cribl_input – Cribl Stream Source that processed the event.
• cribl_output – Cribl Stream Destination that processed the event.
• cribl_route – Cribl Stream Route (or QuickConnect) that processed the event.
• cribl_wp – Cribl Stream Worker Process that processed the event.

Advanced Settings

​

Batch size: The maximum number of items the Google API should batch before it sends them to the topic. Defaults to 10 items.

Batch timeout (ms): The maximum interval (in milliseconds) that the Google API should wait to send a batch (if the configured Batch size limit has not been reached).. Defaults to 100 ms.

Max queue size: Maximum number of queued batches before blocking. Defaults to 100.

Max batch size (KB): Maximum size for each sent batch. Defaults to 256 KB.

Max concurrent requests: The maximum number of in-progress API requests before Cribl Stream applies backpressure. Defaults to 10.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Google Cloud Roles and Permissions

​

Your Google Cloud service account should have at least the following roles on topics:

• roles/pubsub.publisher
• roles/pubsub.viewer or roles/viewer

To enable Cribl Stream’s Create topic option, your service account should have one of the following (or higher) roles:

• roles/pubsub.editor
• roles/editor

Either editor role confers multiple permissions, including those from the lower viewer, subscriber, and publisher roles. For additional details, see the Google Cloud Access Control topic.

Proxying Requests

​

If you need to proxy HTTP/S requests, see System Proxy Configuration.

Let’s Change the Topic

​

The Pub/Sub Destination supports alternate topics specified at the event level in the __topicOut field. So (for example) if a Pub/Sub Destination is configured to send to main topic topic1, and Cribl Stream receives an event with __topicOut: topic2, then Cribl Stream will override the main topic and send this event to topic2.

However, a topic specified in the event’s __topicOut field must already exist on Pub/Sub. If it does not, Cribl Stream cannot dynamically create the topic, and will drop the event. On the Destination’s Status tab, the Dropped metric tracks the number of events dropped because a specified alternate topic did not exist.

(This example uses the short topic name, and that works as long as the topic resides in the project where your GCP credentials originated. If the topic resides in a different project, you must use the fully-qualified topic name, for example __topicOut: projects/<my-project-id>/topics/topic2.)

Troubleshooting

​

The Destination’s configuration modal has helpful tabs for troubleshooting:

Live Data: Try capturing live data to see real-time events as they flow through the Destination. On the Live Data tab, click Start Capture to begin viewing real-time data.

Logs: Review and search the logs that provide detailed information about the delivery process, including any errors or warnings that may have occurred.

Test: Ensures that the Destination is correctly set up and reachable. Verify that sample events are sent correctly by clicking Run Test.

You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify delivery issues. Analyze the graphs showing events and bytes in/out over time.