DreamObjects Destination
The DreamObjects Destination sends data to a DreamObjects bucket through the S3-compatible API.
Type: Non-Streaming | TLS Support: Yes | PQ Support: No
Configure a DreamObjects Destination
- On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
- To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Destination and select the Destination you want from the list, choosing either Select Existing or Add New.
- To configure via the Routes, select Data > Destinations (Stream) or More > Destinations (Edge). Select the Destination you want. Next, select Add Destination.
- In the New Destination modal, configure the following under General Settings:
- Output ID: Enter a unique name to identify this DreamObjects definition. If you clone this Destination, Cribl Stream adds
-CLONEto the original Output ID. - Description: Optionally, enter a description.
- DreamObjects endpoint: DreamObjects S3-compatible cluster endpoint URL. For example,
https://s3.us-east-005.dream.io. Use anhttps://URL so certificate checks in Advanced Settings apply. - DreamObjects bucket name: JavaScript expression that resolves to the Destination bucket name. The expression must evaluate to a string, and Cribl Stream evaluates it only when the Destination initializes. For a fixed bucket name, enter a quoted string such as
'my-bucket'. For a name from a global variable, use a JavaScript string expression such asmyBucket-${C.vars.myVar}.Event-level variables are not available for this field, because Cribl Stream evaluates the bucket name only at Destination initialization. To use event-level variables in file paths, set them in the Partitioning expression field. That expression is evaluated for each file. For expression syntax, see Cribl Expression Syntax.
- Staging location: Filesystem location in which to buffer files, before compressing and moving them to the final destination. Use performant and stable storage. Defaults to
$CRIBL_HOME/state/outputs/staging.The Staging location field is not displayed or available on Cribl-managed Worker Groups in Cribl.Cloud.
- Data format: Format of the output data. Defaults to
JSON.RawandParquetare also available. When you selectParquet, configure Parquet Settings. - You can configure the following Optional Settings:
- Key prefix: Prefix to prepend to files before uploading. Enter a JavaScript expression, enclosed in quotes or backticks, that can evaluate to a constant. Cribl Stream evaluates it only at initialization. For example,
myKeyPrefix-${C.vars.myVar}. Defaults to an empty string. - Partitioning expression: JavaScript expression that defines how files are partitioned and organized. The default is date-based:
C.Time.strftime(_time ? _time : Date.now()/1000, '%Y/%m/%d'). If this field is blank, Cribl Stream uses the event’s__partitionfield when that field is present. Otherwise it uses the root directory of the staging location. - Compression: Data compression format to apply before the file is delivered. Defaults to
gzip. This setting is not available when Data format isParquet. - File name prefix expression: JavaScript expression that defines the output file name prefix. The expression can evaluate to a constant, and you enclose it in single quotes, double quotes, or backticks. Defaults to
`CriblOut`. - File name suffix expression: JavaScript expression that defines the output file name suffix. The expression can evaluate to a constant, and you enclose it in single quotes, double quotes, or backticks. Defaults to
`.${C.env["CRIBL_WORKER_ID"]}.${__format}${__compression === "gzip" ? ".gz" : ""}`.__formatisjsonorraw.__compressionisnoneorgzip.To keep files from being overwritten, Cribl Stream appends a random sequence of 6 characters to the end of their names. This also applies to prefix and suffix expressions in file names.
For example, if you set the File name prefix expression to
CriblExecand set the File name suffix expression to.csv, the file name might display asCriblExec-adPRWM.csv. - Backpressure behavior: Select whether to block or drop events when all receivers are exerting backpressure. Causes might include an accumulation of too many files that need to be closed. Defaults to
Block. See Destination backpressure behavior. - Tags: Optionally, add tags that you can use to filter and group Destinations on the Destinations page. These tags aren’t added to processed events. Use a tab or hard return between tag names.
- Key prefix: Prefix to prepend to files before uploading. Enter a JavaScript expression, enclosed in quotes or backticks, that can evaluate to a constant. Cribl Stream evaluates it only at initialization. For example,
- Output ID: Enter a unique name to identify this DreamObjects definition. If you clone this Destination, Cribl Stream adds
- Optionally, you can adjust the Authentication, Processing, Parquet, and Advanced settings outlined in the following sections.
- Select Save, then Commit & Deploy.
Authentication
This Destination authenticates with an access key and a secret key stored as a key pair. See the DreamObjects hostname for how to create those keys.
Authentication method: Authentication method. Defaults to Secret.
Secret key pair: Select or create a stored secret that references your access key and secret key. A Create link stores a new secret. See securing secrets.
Processing Settings
Post-Processing
Pipeline: Pipeline or Pack to process data before sending the data out using this output.
System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_pipe (identifying the Cribl Stream Pipeline that processed the event). Supports wildcards. Available fields include:
cribl_host- Worker Node that processed the event.cribl_input- Source that processed the event.cribl_output- Destination that processed the event.cribl_pipe- Pipeline that processed the event.cribl_route- Route (or QuickConnect) that processed the event.cribl_wp- Worker Process that processed the event.cribl_group- Worker Group that processed the event.cribl_mode- Distributed mode of the instance that processed the event.
Parquet Settings
To write out Parquet files, note that:
- You can use the Cribl Stream CLI
parquetcommand to view a Parquet file, its metadata, or its schema. - See Parquet Schemas for pointers on how to avoid problems such as data mismatches.
Automatic schema: Toggle on to automatically generate a Parquet schema based on the events of each Parquet file that Cribl Stream writes. When toggled off (the default), configure these settings:
- Parquet schema: Select a schema from the drop-down.
If you need to modify a schema or add a new one, follow the instructions in Parquet Schemas. These steps propagate the freshest schema back to this drop-down.
- Parquet version: Determines which data types are supported, and how they are represented. Defaults to
2.6.2.4and1.0are also available. - Data page version: Serialization format for data pages. Defaults to
V2. If your toolchain includes a Parquet reader that does not supportV2, useV1. - Group row limit: The number of rows that every group contains. The final group can contain a smaller number of rows. Defaults to
10000. - Page size: Target memory size for page segments. Set lower values to improve reading speed, or set higher values to improve compression. Enter a positive integer smaller than the Row group size value, with appropriate units. Defaults to
1 MB. - Log invalid rows: Toggle on to output up to 20 unique rows that were skipped due to data format mismatch. Log level must be set to
debugfor output to be visible. - Write statistics: Leave toggled on (the default) if you have Parquet tools configured to view statistics. These profile an entire file in terms of minimum and maximum values within data, numbers of nulls, and so on.
- Write page indexes: Leave toggled on (the default) if your Parquet reader uses statistics from Page Indexes to enable page skipping. One Page Index contains statistics for one data page.
- Write page checksum: Toggle on if you have configured Parquet tools to verify data integrity using the checksums of Parquet pages.
- Metadata (optional): The metadata of files the Destination writes includes the properties you add here as key-value pairs. For example, one way to tag events as belonging to the OCSF category for security findings is to set Key to
OCSF Event Classand Value to2001.
Advanced Settings
File size limit (MB): Maximum uncompressed output file size. Files of this size are closed and moved to the final output location. Defaults to 32.
Compression level: Compression level to apply before moving files to the final destination. Defaults to Best Speed. This setting is not available when Data format is Parquet.
Open file limit: Maximum number of files to keep open concurrently. When this limit is exceeded, on any individual Worker Process, Cribl Stream closes the oldest open files and moves them to the final output location. Defaults to 100.
Staging file limit: Maximum number of files that the Destination allows to wait for upload before it applies backpressure. Defaults to 100. Minimum is 10. Maximum is 4200.
File open time limit (sec): Maximum amount of time to write to a file. Files open for longer than this limit are closed and moved to the final output location. Defaults to 300.
Idle time limit (sec): Maximum amount of time to keep inactive files open. Files open for longer than this limit are closed and moved to the final output location. Defaults to 30.
Cribl Stream closes files when any of the file limits are met.
Disk space protection: How to handle events when disk space is below the global Min free disk space limit. Select Block (default) or Drop. See Min free disk space.
Concurrent file parts upload limit: Maximum number of parts to upload in parallel per file. A value of 1 sends the whole file at once. When set to 2 or above, the Destination uses multipart uploads. Minimum part size is 5 MB. Defaults to 4. The highest allowed value is 10.
Add output ID: When toggled on (default), adds the Output ID field’s value to the staging location’s file path. This puts each Destination’s files in its own path.
Remove empty staging directories: When toggled on (the default), Cribl Stream deletes empty staging directories after moving files. This prevents the proliferation of orphaned empty directories. When toggled on, configure these settings:
- Staging cleanup period: How often, in seconds, to delete empty directories. Defaults to
300seconds (every 5 minutes). Minimum interval is10seconds. Maximum is86400seconds (every 24 hours). - Directory batch size: How many directories are scanned per batch during automatic cleanup of empty staging directories. Set between
10and10000. Defaults to1000. Higher values speed up cleanup and increase memory usage.
Force close on shutdown: Toggle on to close all staged files during an orderly shutdown. This uploads in-progress files immediately, regardless of Idle time limit (sec), File open time limit (sec), or File size limit (MB). Use this option to minimize data loss risk in ephemeral or auto-scaling environments. Files can close before they reach their configured thresholds, which can produce a larger number of small files. Ensure that your Worker Group Drain timeout is high enough to allow all staged data to finalize before shutdown completes. Defaults to toggled on.
Enable dead-lettering: Toggle on to set a maximum number of retries, and to move files to a designated directory when write failures exceed that limit. This prevents data flow blockage and excessive error logging for undeliverable files. Defaults to toggled off. When toggled on, configure these settings:
- Dead-letter location: Storage location for files that fail to reach their final destination after the retry limit is exceeded. Defaults to
$CRIBL_HOME/state/outputs/dead-letter. - Retry limit: How many times the system attempts to move a file to its intended location before the file is placed in the dead-letter directory. Defaults to
20.
Enable retry backoff: Toggle on to apply exponential backoff with jitter when file uploads fail repeatedly. Defaults to toggled on.
Reuse connections: When toggled on (default), the Destination keeps open HTTP connections alive and reuses them across requests. This reduces connection overhead.
Reject unauthorized certificates: When toggled on (default), Cribl Stream rejects certificates that cannot be verified against a valid certificate authority (CA), such as self-signed certificates. These checks apply when the endpoint uses https://. Toggle off only when the endpoint uses a self-signed or internal CA certificate.
Verify if bucket exists: Toggle this off when your credentials can write objects in the bucket and cannot list the bucket. Defaults to toggled on.
Environment: If you’re using GitOps, use this field to specify a single Git branch on which to enable this configuration. If empty, the configuration is enabled everywhere.
S3 API Permissions
DreamObjects uses S3-compatible access for object operations. The access key must include these actions:
s3:ListBucket- Needed unless Verify if bucket exists is toggled off.s3:PutObjects3:AbortMultipartUploadands3:ListMultipartUploadParts- Needed when Concurrent file parts upload limit is set to2or above. The default is4.
See the DreamObjects documentation for how to configure S3 credentials and access permissions.
Internal Fields
Cribl Stream uses a set of internal fields to assist in forwarding data to this Destination.
Field for this Destination:
__partition: Used when Partitioning expression is blank.
Proxying Requests
To proxy HTTP or HTTPS requests, see System Proxy Configuration.
Troubleshooting
The Destination’s configuration modal has helpful tabs for troubleshooting:
Live Data: Try capturing live data to see real-time events as they flow through the Destination. On the Live Data tab, click Start Capture to begin viewing real-time data.
Logs: Review and search the logs that provide detailed information about the delivery process, including any errors or warnings that may have occurred.
Test: Ensures that the Destination is correctly set up and reachable. Verify that sample events are sent correctly by clicking Run Test.
You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify delivery issues. Analyze the graphs showing events and bytes in/out over time.
Common Issues
Bucket does not exist - self-signed certificate
Example error text:
message: Bucket does not exist - self signed certificate
stack: Error: self signed certificate
at TLSSocket.onConnectSecure (node:_tls_wrap_1535:34)
at TLSSocket.emit (node:events:513:28)
at TLSSocket.emit (node:events:489:12)
at TLSSocket._fini... Show more
tmpPath: /opt/cribl/dreamobjects_bucket/cribl/2023/03/28/CriblOut-XTNGh5.0.json.tmpThis error can occur when:
- The Destination uses a self-signed certificate that Cribl Stream does not trust yet.
- The access key cannot list the bucket.
- The authentication timestamp on the sender is out of sync with the server.
Recommendation
Toggle off Advanced Settings > Reject unauthorized certificates, or add the sender’s certificate to the NODE_EXTRA_CA_CERTS path. Confirm that the access key can list the bucket.
Misconfigured or Nonexistent Buckets
If the bucket does not exist or is misconfigured, the Destination logs follow this pattern:
- At initialization, the Destination logs an error that the bucket cannot be found. This error is not logged when Advanced Settings > Verify if bucket exists is toggled off.
- Each time the Destination closes a staged file and tries to upload it, the Destination logs an error, until the number of files exceeds Advanced Settings > Staging file limit.
After the Destination starts to apply backpressure:
- The Destination logs a backpressure warning.
- Events stop flowing through the Destination.
- The Destination stops creating staging files.
- The Destination keeps trying to upload the staged files and does not log further failures.