Google Cloud Logging

Cribl Stream supports sending log data to Google Cloud Logging, a real-time log storage and management service with search, analysis and alerting.

Type: Streaming | TLS Support: Yes | PQ Support: Yes

Configuring Cribl Stream to Output to Google Cloud Logging​

From the top nav, click Manage, then select a Worker Group to configure. Next, you have two options:

To configure via the graphical QuickConnect UI, click Routing > QuickConnect (Stream) or Collect (Edge). Next, click Add Destination at right. From the resulting drawer’s tiles, select Google Cloud > Logging. Next, click either Add Destination or (if displayed) Select Existing. The resulting drawer will provide the options below.

Or, to configure via the Routing UI, click Data > Destinations (Stream) or More > Destinations (Edge). From the resulting page’s tiles or the Destinations left nav, select Google Cloud > Logging. Next, click Add Destination to open a New Destination modal that provides the options below.

General Settings​

Output ID: Enter a unique name to identify this Google Cloud Logging definition. If you clone this Destination, Cribl Stream will add -CLONE to the original Output ID.

Log location type: Select one of the following.

  • Project: Displays a Project ID expression field. Here, enter a JavaScript expression to compute the value of the project ID to associate log entries with.
  • Organization: Displays a Organization ID expression field. Enter a JS expression to compute the value of the organization ID to associate log entries with.
  • Billing Account: Displays a Billing account ID expression field. Enter a JS expression to compute the value of the billing account ID to associate log entries with.
  • Folder: Displays a Folder expression field. Enter a JS expression to compute the value of the folder ID to associate log entries with.

Log name expression: Ener a JavaScript expression to compute the log name’s value.

Field Mappings​

Payload format: Select one of the following.

  • Text: This default option displays a Payload text expression field. Here, enter JavaScript expression to compute the payload’s value. Must evaluate to a JavaScript string value; if validation fails, this will default to a JSON string representation of the event.

  • JSON: Displays a Payload object expression field. Enter a JS expression to compute the payload’s value. Must evaluate to a JavaScript object value; if validation fails, this will default to the entire event.

    This Destination compresses all payloads, using gzip.

Log labels: Click Add label to apply one or more labels to log entries. On each row of the resulting table, define each label with:

  • Label: Arbitrary name for the label.
  • Value: JavaScript expression to compute the label’s value.

Resource type expression: JavaScript expression to compute the field value for the managed resource type. Must evaluate to one of the monitored resource types listed here. If not specified, falls back to global.

Resource labels: Click Add label to apply one or more labels to the managed resource. Each must correspond to a valid label for the resource type you’ve specified in the Resource type expression, or else Google Cloud Logging will drop it. On each row of the resulting table, define each label with:

  • Label: Arbitrary name for the label.
  • Value: JavaScript expression to compute the label’s value.

Severity expression: JavaScript expression to compute the severity field’s value. Must evaluate to one of the LogSeverity values listed here. Case-insensitive. If not specified, falls back to DEFAULT.

Insert ID expression: JavaScript expression to compute the insert ID field’s value.

Optional Settings​

Backpressure behavior: Whether to block, drop, or queue events when all receivers are exerting backpressure. Defaults to Block.

Tags: Optionally, add tags that you can use to filter and group Destinations in Cribl Stream’s Manage Destinations page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.

Authentication​

Use the Authentication method drop-down to select one of these options:

  • Auto: This option uses the GOOGLE_APPLICATION_CREDENTIALS environment variable, and requires no configuration here.

  • Manual: This default option displays a Service account credentials field for you to enter the contents of your service account credentials file (a set of JSON keys), as downloaded from Google Cloud. To insert the file itself, click the upload button at this field’s upper right. As an alternative, you can use environment variables, as outlined here.

  • Secret: This option exposes a drop-down in which you can select a stored secret that references the service account credentials described above. A Create link is available to store a new, reusable secret.

Persistent Queue Settings​

This tab is displayed when the Backpressure behavior is set to Persistent Queue.

On Cribl-managed Cribl.Cloud Workers (with an Enterprise plan), this tab exposes only the destructive Clear Persistent Queue button (described below in this section). A maximum queue size of 1 GB disk space is automatically allocated per PQ‑enabled Destination, per Worker Process. The 1 GB limit is on outbound uncompressed data, and no compression is applied to the queue.

This limit is not configurable. If the queue fills up, Cribl Stream will block outbound data. To configure the queue size, compression, queue-full fallback behavior, and other options below, use a hybrid Group.

Max file size: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, etc. Defaults to 1 MB.

Max queue size: The maximum amount of disk space that the queue is allowed to consume on each Worker Process. Once this limit is reached, this Destination will stop queueing data and apply the Queue‑full behavior. Required, and defaults to 5 GB. Accepts positive numbers with units of KB, MB, GB, etc. Can be set as high as 1 TB, unless you’ve configured a different Max PQ size per Worker Process in Group Settings.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. To this value, Cribl Stream will append /<worker‑id>/<output‑id>.

Compression: Codec to use to compress the persisted data, once a file is closed. Defaults to None; Gzip is also available.

Queue-full behavior: Whether to block or drop events when the queue is exerting backpressure (because disk is low or at full capacity). Block is the same behavior as non-PQ blocking, corresponding to the Block option on the Backpressure behavior drop-down. Drop new data throws away incoming data, while leaving the contents of the PQ unchanged.

Clear Persistent Queue: Click this “panic” button if you want to delete the files that are currently queued for delivery to this Destination. A confirmation modal will appear - because this will free up disk space by permanently deleting the queued data, without delivering it to downstream receivers. (Appears only after Output ID has been defined.)

Strict ordering: The default Yes position enables FIFO (first in, first out) event forwarding. When receivers recover, Cribl Stream will send earlier queued events before forwarding newly arrived events. To instead prioritize new events before draining the queue, toggle this off. Doing so will expose this additional control:

  • Drain rate limit (EPS): Optionally, set a throttling rate (in events per second) on writing from the queue to receivers. (The default 0 value disables throttling.) Throttling the queue’s drain rate can boost the throughput of new/active connections, by reserving more resources for them. You can further optimize Workers’ startup connections and CPU load at Group Settings > Worker Processes.

Processing Settings​

Post‑Processing​

Pipeline: Pipeline to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the Cribl Stream Pipeline that processed the event). Supports wildcards. Other options include:

  • cribl_host – Cribl Stream Node that processed the event.
  • cribl_input – Cribl Stream Source that processed the event.
  • cribl_output – Cribl Stream Destination that processed the event.
  • cribl_route – Cribl Stream Route (or QuickConnect) that processed the event.
  • cribl_wp – Cribl Stream Worker Process that processed the event.

Advanced Settings​

Request concurrency: Maximum number of concurrent requests before blocking. This is set per Worker Process. Defaults to 5.

Connection timeout: Amount of time (in milliseconds) to wait for the connection to establish before retrying. Defaults to 10000 (10 sec.).

Request timeout: Amount of time (in seconds) to wait for a request to complete before aborting it. Defaults to 30 sec.

Max body size (KB): Maximum size of the request body before compression. Defaults to 4096 KB. The actual request body size might exceed the specified value because the Destination adds bytes when it writes to the downstream receiver. Cribl recommends that you experiment with the Max body size value until downstream receivers reliably accept all events.

Max events per request: Maximum number of events to include in the request body. Defaults to 0 (unlimited).

Flush period (sec): Maximum time between requests. Low values could cause the payload size to be smaller than the configured Max body size. Defaults to 1 sec.

Throttle request rate: Enter a maximum number of requests to send per second, as necessary to comply with Google Cloud’s API limits. This field defaults to empty (no throttling), and accepts throttling rates as high as 2000 requests per second.

Environment: Optionally, specify a single Git branch on which to enable this configuration. If this field is empty, the config will be enabled everywhere.

Google Cloud Roles and Permissions​

Your Google Cloud service account must have at least the following role:

  • Logs Writer