Google Cloud Storage is a non-streaming Destination type.

Configuring Cribl LogStream to Output to Google Cloud Storage Destinations

Select Data > Destinations, then select Google Cloud > Cloud Storage from the Data Destinations page's tiles or left menu.

Next, click Add New to open the Cloud Storage > New Destination modal, which provides the following fields.

General Settings

Output ID: Enter a unique name to identify this Cloud Storage definition.

Bucket name: Name of the destination bucket. This value can be a constant. or a JavaScript expression that can be evaluated only at init time. E.g., referencing a Global Variable: myBucket-${C.vars.myVar}.

Region: Region where the bucket is located.

Staging location: Filesystem location in which to locally buffer files before compressing and moving to final destination. Cribl recommends that this location be stable and high-performance.

Add Output ID: Whether to append output's ID to staging location. Defaults to Yes.

Key prefix: Root directory to prepend to path before uploading. Enter a constant, or a JS expression enclosed in single quotes, double quotes, or backticks.

Partitioning expression: JavaScript expression to define how files are partitioned and organized. If left blank, Cribl LogStream will fall back to event.__partition. Defaults to `${host}/${sourcetype}`. Partitioning by time is also possible, e.g., `${host}/${C.Time.strftime(_time, '%Y-%m-%d')}/${sourcetype}`

Data format: Format of the output data. Defaults to JSON.

File name prefix expression: The output filename prefix. Must be a JavaScript expression (which can evaluate to a constant), enclosed in quotes or backticks. Defaults to CriblOut.

Compress: Select the data compression format to use before moving data to final destination. Defaults to none. Cribl recommends setting this to gzip.

Backpressure behavior: Select whether to block or drop events when all receivers in this group are exerting backpressure. Defaults to Block.

Authentication

Authentication is via HMAC (Hash-based Message Authentication Code). To create a key and secret, see Google Cloud's Managing HMAC Keys for Service Accounts documentation.

Access key: Enter the HMAC access key.

Secret key: Enter the HMAC secret.

Processing Settings

Post‑Processing

Pipeline: Pipeline to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the LogStream Pipeline that processed the event). Supports c* wildcards. Other options include:

• cribl_host – LogStream Node that processed the event.
• cribl_wp – LogStream Worker Process that processed the event.
• cribl_input – LogStream Source that processed the event.
• cribl_output – LogStream Destination that processed the event.

Advanced Settings

Endpoint: The Google Cloud Storage service endpoint. Typically, there is no reason to change the default https://storage.googleapis.com endpoint.

Object ACL: Select an Access Control List to assign to uploaded objects. Defaults to private.

Storage class: Select a storage class for uploaded objects.

Signature version: Signature version to use for signing requests. Defaults to v4.

Max file size (MB): Maximum uncompressed output file size. Files of this size will be closed and moved to final output location. Defaults to 32.

Max file open time (sec): Maximum amount of time to write to a file. Files open for longer than this limit will be closed and moved to final output location. Defaults to 300.

Max file idle time (sec): Maximum amount of time to keep inactive files open. Files open for longer than this limit will be closed and moved to final output location. Defaults to 30.

Max open files: Maximum number of files to keep open concurrently. When exceeded, the oldest open files will be closed and moved to final output location. Defaults to 100.

📘

Cribl LogStream will close files when either of the Max file size (MB) or the Max file open time (sec) conditions are met.

Internal Fields

Cribl LogStream uses a set of internal fields to assist in forwarding data to a Destination.

Field for this Destination:

• __partition