/ / ·

Known Issues

This page lists known issues affecting Cribl Stream and/or Cribl Edge.

2023-11-09 - All Versions through Current - High-volume UDP data may be dropped in Cribl.Cloud

Problem: Sending large numbers of UDP events per second can cause Cribl.Cloud to drop some of the data. This results from restrictions of the UDP protocol and concerns Sources that support UDP: Metrics, Raw UDP, SNMP Trap, and Syslog.

Workaround: To minimize the risk of data loss, deploy a hybrid Stream Worker Group with customer-managed Worker Nodes as close as possible to the UDP senders.

Cribl also recommends tuning the OS UDP buffer size.

2023-07-13 – All Versions through Current – When configuring an S3 Collector, JavaScript expressions break the “Auto-populate from” option for the Path field [CRIBL-18844]

Problem: When automatically populating an S3 Collector from an S3 Destination, the collector Path field won’t resolve JavaScript expressions, even if the expression was valid on the Destination that the field was populated from. The S3 Collector path is treated like a literal string and the software fails to warn you that the path is invalid.

Workaround: None.

Fix: In Cribl Edge 4.3.

2023-07-07 – v.4.1.3 – When configuring an HTTP Source, enabling Source PQ changes the inputId in events [CRIBL-18668]

Problem: When Source-side Persistent Queueing is enabled on an HTTP Source, the trailing colon is dropped from __inputId. This will break Routes and filters that are based on the Input ID if it is copied from the configuration page. Live captures will also stop working until you modify them.

Workaround: None.

Fix: In Cribl Edge 4.3.

2023-06-29 – All Versions through Current – (Linux) System Metrics Source does not emit “Per interface” metrics in “All” mode [CRIBL-18473]

Problem: In the (Linux) System Metrics Source > Host Metrics tab, selecting All does not emit Per Interface metrics.

Workaround: In the Host Metrics tab, select Custom > Network > Custom then toggle Per interface metrics to Yes.

Fix: In Cribl Edge 4.2.

2023-06-26 – All Versions through Current – High CPU usage in File Monitor Source’s Manual mode [CRIBL-18400]

Problem: In the File Monitor Source and the Explore > Files tab UI, the Manual mode does not honor the Max depth setting. The API process consumes high CPU resources because the discovery logic (used in the File Monitor Source and Files tab) recurses in the directory tree.

Fix: In Cribl Edge 4.2.

2023-06-26 – All Versions through Current – Fleet Upgrade Errors [CRIBL-18374]

Problem: A Fleet’s upgrade from a Leader can result in errors when the Edge Nodes use different host/port/tls settings than Worker Nodes.

Workaround: The Edge Nodes and Workers must connect to the leader using the host/port/tls connection details. If this is not possible, upgrade Edge Nodes separately using the boostrap script.

Fix: Version TBD.

2023-06-21 – v.3.5.2–v.4.1.3 – Using the Rename Function gives unexpected results due to skipped internal fields [CRIBL-18285]

Problem: When internal fields are present in the Rename fields list, the Rename Function may incorrectly assign field keys or values.

Workaround: This function is not intended to operate on internal fields. Avoid this operation.

Fix: In Cribl Edge 4.2.

2023-06-15 – v.4.1.3 – Amazon CloudWatch Destinations log error while flushing events older than 24 hours [CRIBL-18184]

Problem: Amazon CloudWatch Destination doesn’t filter batches of events to ensure all events in the batch are within 24 hours of each other as required by the API, resulting in the batch being rejected by Cloudwatch.

Workaround: Add a Post-Processing Pipeline with a Drop Function using a Filter Expression of _time<(Date.now() / 1000) - 86400.

Fix: In Cribl Edge 4.2.

2023-06-14 – v.4.1.2–4.1.3 – The Office 365 Activity Source misses events [CRIBL-18164]

Problem: The Office 365 Activity Source misses events due to the current collection system. For services such as Exchange, SharePoint, OneDrive, and Teams, Microsoft indicates that audit record availability is typically 60 to 90 minutes after an event occurs. Our current collector methodology does not account for this availability window. Instead, the Source completes runs as scheduled and collects only events it finds during the configured time range, which can lead to missed events.

Workaround: None.

Fix: In Cribl Edge 4.2.

2023-06-08 – v.4.1.2 – AES-256-GCM security vulnerability [CRIBL-18038]

Problem: The AES-256-GCM encryption option introduced in v.4.1.2 included a security vulnerability.

Workaround: Do not use this option in v.4.1.2.

Fix: In Cribl Edge 4.1.3.

2023-06-01 – v.4.0.0–4.1.3 – GroupEdit is not able to commit changes [CRIBL-17939]

Problem: Users having the GroupEdit policy are able to make changes (e.g. create new Sources) but are unable to Commit their changes. These users should be able to Commit, but not Deploy, changes.

Workaround: Apply the GroupFull policy for users that need to be able to Commit changes. These users will also have the ability to deploy a Worker Group or Fleet.

Fix: In Cribl Edge 4.2.

2023-06-01 – v.4.1.0–4.1.2 – Kubernetes Logs Event Breaker Incorrectly Breaks Events [CRIBL-17907]

Problem: The Kubernetes Logs Event Breaker incorrectly breaks events with out-of-order timestamps.

Workaround: In the Event Breaker, move the logic that adjusts _raw into a Pre-Processing Pipeline.

Fix: In Cribl Edge 4.1.3.

2023-05-27 – v.4.1.1–4.1.2 – Perf bug: RPC consuming excessive CPU compiling expressions [CRIBL-17779]

Problem: In 4.1.1, to improve product security, we changed how we manage our RPC traffic from Leaders and Workers. This change can have a negative impact in larger environments with many Worker Processes. Following a 4.1.1 or 4.1.2 upgrade, the Leader can become non-responsive to UI requests due to a steady 100% CPU utilization. When this happens, you cannot manage or monitor Cribl Edge environments.

Workaround: Roll back to a previous stable version, such as 4.1.0.

Fix: In Cribl Edge 4.1.3.

2023-05-26 – v.4.1.2–4.1.3 – Edge Nodes don’t update from Cribl Edge Leader [CRIBL-17792]

Problem: In Cribl Edge 4.1.2–4.1.3, if you define the baseURL key on a Leader, Edge Nodes don’t get the latest config version from the Leader. The log may also contain checksum mismatch warnings.

Workaround: Clear URL base path in Settings > General Settings > Advanced or specify an empty baseURL key in cribl.yml. If you need to define a base URL, do not upgrade to the affected versions.

Fix: In Cribl Edge 4.2.

2023-05-19 – v.4.1.2 – CrowdStrike Destination’s “LogScale endpoint” field hidden from UI [CRIBL‑17715]

Problem: The CrowdStrike Falcon LogScale Destination’s LogScale endpoint field is hidden from the 4.1.2 configuration modal, but can be restored.

Workaround: Follow these steps, in sequence, for each LogScale Destination.
1. Populate the LogScale config modal, and save once.
2. Reopen the modal, then select Manage as JSON.
3. Change the "loadbalanced": key’s value from true to false.
4. Copy this default nested element: "url": "https://cloud.us.humio.com/api/v1/ingest/hec",
5. Paste it above the whole "urls" element (typically at line 21).
6. Click OK to restore the visual UI, with the LogScale endpoint field now visible.
7. Enter your actual endpoint URL, and save the config.

Fix: In Cribl Edge 4.1.3.

2023-05-16 – v.4.1.1–4.1.2 – Expanding Status of Output Router Destination triggers error [CRIBL-17632]

Problem: When you attempt to expand a node on the Status tab for an Output Router Destination, the page crashes and the error Cannot convert undefined or null to object appears.

Fix: In Cribl Edge 4.1.3.

2023-05-12 – v.4.0–4.1.3 – Windows Event Forwarder Source causes duplicate events when experiencing backpressure [CRIBL-17614]

Problem: In certain cases where the Windows Event Forwarder Source experiences backpressure from a downstream Pipeline or Destination, it will return an incorrect error message to the client that sent the events. This causes that client to re-send the same events and results in duplicate events ingested in Stream.

Workaround: In cases where the cause of the backpressure can be resolved, this issue is mitigated. However if the backpressure cannot be completely removed, there is no additional workaround.

Fix: In Cribl Edge 4.2.

2023-05-12 – v.4.0–4.1.2 – The Kubernetes Logs Source drops events [CRIBL-17602]

Problem: Events from a running container will stall when the underlying container runtime rotates the log file. Any further rotation of log files will result in data loss.

Workaround: Restart the Kubernetes Logs Source to reconnect.

Fix: In Cribl Edge 4.1.3.

2023-05-05 – v.4.1.1 – S3 ingestion slows after upgrade [CRIBL-17317]

Problem: After upgrading to Cribl Edge 4.1.1, ingesting using any JSON Array Event Breaker (such as AWS CloudTrail) will slow or stop due to high CPU usage.

Workaround: Roll back to a previous stable version, such as 4.1.0.

Fix: In Cribl Edge 4.1.2.

2023-04-27 – v.4.1.1 – Edge Nodes integrated with `systemd/initd` can fail upon reconfig [CRIBL‑17264]

Problem: Where 4.1.1 on-prem or hybrid Edge Nodes (on Linux) are integrated with systemd/initd, deploying new config bundles to those Edge Nodes can stop the Edge Nodes from processing data. The root cause is an incorrect backup step, which (depending on permissions) can also consume large amounts of disk space on their hosts.

Precondition: Cribl-managed Cloud instances are unaffected. This problem has been observed only if at least one of the following directories is populated, or exists with restricted permissions:

/data/
/default/cribl/
/default/edge/
/local/cribl/
/local/edge/
/default/<any-installed-Pack-name>/
/local/<any-installed-Pack-name>/

Workaround: Skip v.4.1.1 or roll back to your last stable version. To run v.4.1.1, this workaround is available on systemd’s Service section: 1. In the script /etc/systemd/system/cribl.service, add the line: WorkingDirectory=/opt/cribl, (This is the default path - specify your own path equivalent to $CRIBL_HOME). 2. Then reload the systemd daemon: systemctl daemon-reload. 3. Then restart the Cribl Stream instance using systemctl restart <service name>.

Fix: In Cribl Edge 4.1.2.

2023-04-14 – v.2.2–4.1.x – Requests from Office 365 Message Trace Source failed intermittently [CRIBL-16929]

Problem: Microsoft has fixed this Office 365 Message Trace API problem with this patch (Microsoft login required). Before this patch, requests failed intermittently, often with Non-whitespace errors.

Fix: Fixed by Microsoft (not a Cribl problem).

2023-04-13 – v.4.1.1 – Monitoring incorrectly shows metrics for disconnected Subscriptions [CRIBL‑16926]

Problem: Selecting Monitoring > Data > Subscriptions will falsely show statistics even for Subscriptions that are not connected to a Destination, and therefore have no data flow.

Workaround: Ignore these Monitoring statistics.

Fix: In Cribl Edge 4.1.2.

2023-04-03 – v.4.1.0 – Windows Event Logs Source should have a configurable Max Event Size [CRIBL-16549]

Problem: Events aren’t broken properly when collecting logs from the PowerShell event log. The Source contains a Max Event Limit of 51200, which is a hard-coded breaker config. This limit should be configurable.

Fix: In Cribl Edge 4.1.1.

2023-03-31 – v.4.1.0-4.1.1 – Cribl.Cloud API Credential’s Roles aren’t honored on tokens [SAAS‑3681]

Problem: API Bearer tokens obtained through the Cribl.Cloud portal are all granted the Admin Role, regardless of the scope specified on the parent Credential.

Workaround: Use the pre-4.0 workaround to obtain Bearer tokens from the in-app API Reference.

Fix: In Cribl Edge 4.1.2.

2023-03-30 v.4.1.0 – Splunk Load Balanced Destination incorrectly re‑creates all connections during DNS resolution [CRIBL‑16494]

Problem: When Minimize in-flight data loss is enabled, the Splunk Load Balanced Destination re-creates all outbound connections during DNS resolution. This will cause the Destination to report a blocked status until the outbound connections refresh. If persistent queues (PQ) are enabled, PQ will engage while the Destination is blocked.

Workaround: Select the Destination’s Manage as JSON option, to add the key-value pair: "maxFailedHealthChecks": 1.

Fix: In Cribl Edge 4.1.1.

2023-03-28 – v.4.1.0 – Add Kubernetes and Docker modals in the UI aren’t checking the `CRIBL_BOOTSTRAP_HOST` environment variable [CRIBL-16432]

Problem: The Add Windows and Linux modals check for a bootstrap hostname to see if a user has configured a host override with the CRIBL_BOOTSTRAP_HOST environment variable. The Add Kubernetes and Docker modals aren’t checking the environment variable as expected.

Fix: In Cribl Edge 4.1.1.

2023-03-23 – v.4.1.0 – Leader’s config deployments to pre-4.1 Workers silently fail [CRIBL‑16339]

Problem: A 4.1.x Leader requires Workers to also be upgraded to – v.4.1.x. If you attempt to deploy configs to Workers running earlier versions, the intended upgrade prompt might not appear. In this case, the deploy will silently hang.

Workaround: If you haven’t enabled automatic upgrades of on-prem or hybrid Workers, upgrade all Fleets to 4.1.x before you deploy to them.

Fix: In Cribl Edge 4.1.1.

2023-03-22 – v.4.1.0-4.1.1 – Cribl Stream Database Collector cannot connect with SQL Server using AD authentication [CRIBL-16304]

Problem: On the indicated versions, the Database Collector cannot connect with SQL Server using Active Directory (AD) authentication. (Only local auth works.)

Workaround: Reverting to Cribl Stream v.4.0.4 restores the Database Collector’s ability to connect using AD authentication.

Fix: In Cribl Stream 4.1.2.

2023-03-21 – v.4.1.0 – Unable to bind `CRIBL_DIST_LEADER_URL` to an IPv6 address [CRIBL-16284]

Problem: Due to an issue with the CRIBL_DIST_LEADER_URL environment variable, Cribl will not bind to a specified IPv6 address.

Workaround: Manually update the settings in the two relevant configuration files, instance.yml and cribl.yml, for RPC and UI communication respectively.

For the UI, manually define the host setting in the cribl.yml file:

api:
  host: "::"

To configure the instance as a Leader node and listen on all IPv6 and IPv4 addresses, manually configure the instance.yml file:

distributed:
  mode: master
  master:
    host: "::"
    port: 4200
    tls:
      disabled: true
    authToken: criblmaster

Fix: Planned for Cribl Edge 4.2.2.

2023-03-20 – v.3.5.0–4.1.0 – System Activity drawer on Windows Edge Nodes displays no data [CRIBL-16194]

Problem: When you navigate to a Fleet’s List View and click a row containing a Windows Edge Node, the System Activity drawer displays no data.

Fix: In Cribl Edge 4.1.1.

2023-03-20 – v.4.1.0-4.2.1 – If a deployment fails, Workers will not automatically revert to the previous version [CRIBL‑16203]

Problem: When a deployment fails, Workers cannot revert default/cribl to a previous version because that directory is no longer backed up. The Worker will enter a broken state if you provide an invalid deploy bundle, because it cannot revert to the last valid state.

Workaround: To resolve any broken Workers, you must deploy a valid configuration. The Worker will resume working once it picks up the new configuration.

Fix: Planned for Cribl Edge 4.2.2.

2023-03-15 – v.4.1.0 – QuickConnect Pipeline Settings button fails [CRIBL‑16142]

Problem: When adding a new Pipeline using QuickConnect, clicking the gear (⚙️) button fails to open Pipeline Settings.

Workaround: After saving the Pipeline in QuickConnect, use Manage > Processing > Pipelines to select your new Pipeline. Click the gear button here to access Pipeline Settings as expected.

Fix: In Cribl Edge 4.1.1.

2023-03-14 v.4.0.0–4.1.0 – Duplicate data with Event Hubs [CRIBL‑16102]

Problem: When Minimize Duplicates is enabled, the Azure Event Hubs Source assigns partitions to multiple Edge Nodes in the same Fleet, which generates duplicate data.

Workaround: Toggle Minimize Duplicates to No in Azure > Event Hubs > Advanced Settings.

Fix: In Cribl Edge 4.1.1.

2023-03-01 – v.4.0.4 – Kafka-based Sources are omitting the `_time` field [CRIBL‑15696]

Problem: Kafka-based Sources (Kafka, Azure Event Hubs, Confluent Cloud) are not sending the _time field.

Workaround: If Cribl Edge is not retrieving the _time field, then in the affected Sources’ config modals, use the Processing Settings > Fields tab to re-create _time. This new _time field will add the current (ingestion) time instead of the message time.

Fix: In Cribl Edge 4.1.

2023-02-24 – v.4.1.0-4.2.1 – The Chain Function has a 10% impact on performance [CRIBL-15538]

Problem: Using the Chain Function to chain data processing from one Pipeline or Pack to another degrades performance by about 10%, compared to running the original Pipeline or Pack directly.

Fix: Planned for Cribl Edge 4.2.2.

2023-02-23 – v.4.0.x – DNS Resolution reconnects Cribl TCP connections every time [CRIBL‑15363]

Problem: All Cribl TCP connections will reconnect every time DNS Resolution occurs, even when reconnection is not necessary.

Fix: In Cribl Edge 4.1.

2023-02-23 – v.4.0.4–4.1.2 – Enable Automatic Upgrades deletes remote repo’s Git Settings [CRIBL‑15502]

Problem: Enabling the Leader’s Upgrade > Enable Automatic Upgrade setting deletes the corresponding remote repo’s stored Git Settings.

Workaround: Reconfigure your repo at Git Settings > Remote.

Fix: In Cribl Edge 4.1.3.

2023-02-22 – v.4.0.4–4.1.0 – Sending data to an inactive Kafka, Confluent Cloud, or Azure Event Hubs Destination triggers an endless series of log errors [CRIBL‑15455]

Problem: If you accidentally make a Kafka, Confluent Cloud, or Azure Event Hubs Destination inactive by putting a mismatched value in the Advanced Settings > Environment setting, Cribl Edge will send an endless series of errors to the log file.

Workaround: Correct the Environment value or leave it empty.

Fix: In Cribl Edge 4.1.1.

2023-02-20 – v.4.0.x–4.1.0 – Logging-level changes do not take effect for services [CRIBL‑15365]

Problem: Changing logging levels in the Leader’s Settings has no effect on the logs for the Connections, Lease Renewal, Metrics, or Notifications services.

Fix: In Cribl Edge 4.1.1.

2023-02-20 – v.4.0.3–4.1.0 – On-prem Edge Nodes automatically upgrade, ignoring Leader settings [CRIBL‑15367]

Problem: Upgrading a Leader Node causes its on-prem Edge Nodes to automatically upgrade, even when the Leader’s Enable automatic upgrades option is set to No (the default). This problem does not affect Cribl-managed Cloud Edge Nodes.

Workaround: 1. Toggle the UI’s Enable automatic upgrades slider to Yes and save the configuration; then slide it back to No and save again. Or: 2. Edit local/cribl/cribl.yml to explicitly set upgradeSettings.disableAutomaticUpgrade to true.

Fix: In Cribl Edge 4.1.1.

2023-02-14 – v.4.0.0–4.0.4 – Pipeline is deselected in QuickConnect after modification [CRIBL-15246]

Problem: If you add a Pipeline in the QuickConnect UI, then modify it in the Edit Pipeline modal, the Pipeline is deselected in the Add Pipeline to Connection modal. If you then close the Add Pipeline to Connection modal, the Pipeline will disappear from the Route.

Workaround: Select the modified Pipeline in the Add Pipeline to Connection modal before closing it.

Fix: In Cribl Edge 4.1.

2023-02-14 – v.3.5.0 through Current – Cannot clear Messages drawer while in GitOps Push mode [CRIBL‑15239]

Problem: When in GitOps Push mode, you cannot clear messages from the Messages drawer. Cribl Edge returns a Forbidden error.

Fix: Version TBD.

2023-02-06 – v.4.0.x–4.1.0 – Event Breaker timestamp extraction fails after configured time [CRIBL‑15107]

Problem: Event Breakers’ timestamp extraction stops working after the Rule’s configured Future timestamp allowed value (if set). All subsequent events received get the current time as their timestamp.

Workaround: Set a sufficiently large value for Future timestamp allowed – e.g., one year from the date you re/configure the rule. Alternately, restore normal timestamp extraction by restarting Worker Processes.

Fix: In Cribl Edge 4.1.1.

2023-02-05 – v.3.0.3–4.0.4 – `CRIBL_DIST_WORKER_PROXY` env var is ignored when Leader Node’s Master URL is set via `instance.yml` [CRIBL-15540]

Problem: Setting the Cribl Edge Leader’s Master URL via instance.yml causes Edge Nodes to ignore the CRIBL_DIST_WORKER_PROXY env var. Instead of trying to connect to the proxy configured in CRIBL_DIST_WORKER_PROXY, Edge Nodes will try to connect to other entities (e.g., Leader Nodes) directly, producing unexpected results.

Workaround: Set the Cribl Edge Leader’s Master URL via the CRIBL_DIST_LEADER_URL env var, rather than via instance.yml.

Fix: In Cribl Edge 4.1.

2023-02-03 – v.4.0.0–4.0.4 – Metrics blocklist can’t be changed on a global level [CRIBL-15081]

Problem: The Metrics blocklist cannot be modified from Settings > Global Settings > General Settings > Limits.

Workaround: Edit the limits.yml file’s metricsFieldsBlacklist element. Add the event fields for which you want to disable metrics collection; remove any event fields for which you want to restore metrics collection.

Fix: In Cribl Edge 4.1.

2023-01-26 – v.3.5.0–4.0.3 – Preview Full > Send Out does not capture events [CRIBL-14914]

Problem: Selecting a Sample Data file’s Preview Full > Send out option captures no events. This bug has been observed when capturing on Destinations.

Fix: In Cribl Edge 4.0.4.

2023-01-25 – v.3.5.x through Current – Cascading problems when Windows Event Logs Source collects from `ForwardedEvents` channel [CRIBL‑14869]

Problem: Using the Windows Event Logs Source to collect events from a Windows ForwardedEvents channel can cause misattribution of logs to the local machine that hosts Cribl Edge, and undercollection of local logs from other logging channels (such as Security, System, or Application).

Workaround: Exclude the ForwardedLogs channel from your Event Logs selection.

Fix: Version TBD.

Problem: After you enable GitOps Push mode, the resulting red confirmation banner contains a dead link labeled GitOps Workflow.

Workaround: To switch off Push mode, navigate to Settings > Global > Git Settings, and then set GitOps workflow back to None.

Fix: In Cribl Edge 4.1.

2023-01-12 – v.3.5.x through Current – Recently added Worker Nodes fail to appear on the Monitoring page [CRIBL‑14627]

Problem: When Worker Nodes are added, there may be a delay before the Worker count is updated on the Monitoring page.

Workaround: Refresh your web browser.

Fix: Version TBD.

2023-01-11 – Multiple versions through 4.1.x – Kafka-based Sources’ rebalancing is logged with exaggerated severity [CRIBL-14609]

Problem: The Kafka, Azure Event Hubs, and Confluent Cloud Sources log REBALANCE_IN_PROGRESS events at the error level, even though only frequent rebalancing indicates a system-level or processing issue.

Workaround: Treat infrequent rebalancing events as warn.

Fix: Depends on a change to the underlying kafkajs library.

2023-01-10 – v.3.5.x through Current – Configuration updates not shared between primary and standby Leaders [CRIBL‑12814, CRIBL‑14556]

Problem: With high availability/failover enabled, updates to the active Leader’s configuration are not automatically synced to the standby Leader. Workers might not be able to connect to the standby Leader when it takes over.

Workaround: Use the filesystem to explicitly sync the updated instance.yml file across the two Leaders’ hosts.

Fix: In – v.4.0.3, as an intermediate fix, enabling HA will prevent further UI-based changes to Distributed Settings. This will enforce config changes via edits to portable instance.yml files. Version TBD for automatic synchronization between Leaders’ hosts.

2023-01-02 – v.4.0.0–4.0.2 – Google Cloud Chronicle Destination drops changes in distributed deployments with custom log types [CRIBL‑14453]

Problem: A new Google Cloud Chronicle Destination with a custom log type might fail to save changes. This issue affects only distributed deployments.

Fix: In Cribl Edge 4.0.3.

2022-12-15 – v.3.5.4–4.0.3 – Changes to a Lookup table on the Leader don’t always propagate to Edge Nodes [CRIBL-14299]

Problem: Modifying a Lookup table on the Leader doesn’t always propagate the changes to Edge Nodes, even after clicking Commit and Deploy.

Workaround: Manually restart the Edge Nodes to refresh Lookup tables.

Fix: In Cribl Edge 4.0.4.

2022-12-13 – v.4.0.0 through Current – Default commit message missing for non-admin users [CRIBL‑14239]

Problem: For users who have Roles as high as owner_all, but not admin, the Commit modal fails to display any Default commit message saved in Git Settings.

Workaround: Enter (or paste) a message per commit.

Fix: Version TBD.

2022-12-09 – v.4.0.1–4.0.2 – Users assigned the `owner_all` role cannot perform commits [CRIBL-14180]

Problem: When a user with the owner_all role tries to perform a commit, the commit fails with the UI displaying a Forbidden modal.

Workaround: Modify the GroupFull policy, as follows:

As a user with the owner_all role, try a POST /version/commit API call with a request body of { "message": "test: hello" }. The commit should fail, and the request payload should be { message: "test: hello" }.
If $CRIBL_HOME/local/cribl/policies.yml does not exist, copy $CRIBL_HOME/default/cribl/policies.yml to $CRIBL_HOME/local/cribl/policies.yml.
If your OS is Linux, run the command chmod 0744 policies.yml.

In $CRIBL_HOME/local/cribl/policies.yml, edit the GroupFull policy to match the following:

GroupFull:
  args:
    - groupName
  template:
    - PATCH /master/groups/${groupName}/deploy
    - GroupEdit ${groupName}
    - POST /version/commit
    - GET /version
    - GET /version/*

Now retry the POST /version/commit API call, again with a request body of { "message": "test: hello" }. The commit should succeed, and the request payload should be { message: "test: hello", "group": "default", "effective": true }.

Fix: In Cribl Edge 4.0.3.

2022-12-08 – v.4.0.1–4.0.2 – Landing page not displaying system metrics from Edge Nodes when teleporting from the Leader [CRIBL-14169]

Problem: When you teleport from a Leader running v.4.0.1 or 4.0.2 to an Edge Node that’s running v.4.0.0 or earlier, the system metrics for the landing page will not populate. This also affects the system metrics shown in the Node drawer for the honeycomb and Node List View pages.

Workaround: Upgrade Edge Nodes to v.4.0.2 or higher.

Fix: In Cribl Edge 4.0.3.

2022-12-07 – v.4.0.1 – Google Cloud Pub/Sub Source and Destination can break on field validation [CRIBL‑14160]

Problem: The Google Cloud Pub/Sub Source and Destination strictly validated entries in the Topic ID and Subscription ID fields. If you entered a full path (rather than the expected ID substring) in these fields, this strict validation broke the integration and broke the UI.

Workaround: Trim Topic ID and Subscription ID field values to just the ID.

Fix: Validation is rolled back in Cribl Edge 4.0.2.

2022-12-01 – v.4.0.2 – CrowdStrike FDR Source needs 6-hour visibility timeout [CRIBL‑14067]

Problem: With its default Visibility timeout seconds setting of 600 (10 minutes), the CrowdStrike FDR Source can accumulate large backlogs when pulling data from Crowdstrike buckets.

Workaround: Set the Visibility timeout seconds to 21600 (6 hours), as CrowdStrike recommends. Leave both Max messages and Num receivers at their default 1 settings.

Fix: In Cribl Edge 4.0.3.

2022-12-01 – v.4.0.0 – Amazon S3 Source stops receiving data after upgrade [CRIBL‑14093]

Problem: Upgrading to Cribl Edge 4.0.0 can cause the Amazon S3 Source to stop receiving data. This is due to a race condition between (premature) SQS messaging versus the Source’s initialization.

Workaround: Skip v.4.0.0 (or temporarily roll back to v.3.5.4).

Fix: In Cribl Edge 4.0.1.

2022-11-30 – v.4.0.2 – QuickConnect-configured Sources’ misleading “Enabled” status while disconnected [CRIBL-14041]

Problem: As you configure a new Source from the QuickConnect UI, the Source’s Enabled slider will initially switch to Yes. This is misleading, because the Source is not yet connected to a Destination, so no data can flow.

Workaround: Save the Source in its config drawer. This resets the Enabled slider to its accurate No status, until you connect the Source to a Destination.

Fix: In Cribl Stream 4.0.3.

2022-11-28 v.4.0–4.0.2 – “Unknown config version” error can appear when deploying changes [CRIBL‑13910]

Problem: Attempting to commit and deploy a change can trigger errors of the form Unknown config version: "[hash]".

Workaround: On the Leader’s host, upgrade the git client to v.1.9.1 or later.

Fix: In Cribl Edge 4.0.3.

2022-11-28 – v.4.1.0 through Current – Diag bundles might fail to download from teleported Edge Nodes [CRIBL-13999]

Problem: When you’re remotely accessing a Edge Node’s UI, diag bundles might fail to download from Global Settings > Diagnostics.

Workaround: Refresh the page and Export the bundle again. Alternatively, log directly into the Edge Node’s UI before creating the diag.

Fix: Version TBD.

2022-11-23 – All versions through 4.1.0 – Edge Nodes show incorrect configurations after upgrade or commit/deploy from Leader [CRIBL‑13863, CRIBL‑15507]

Problem: When you commit and deploy changes from the Leader, Edge Nodes sometimes fail to automatically restart with the correct configuration changes.

Workaround: Manually restart the Edge Nodes.

Fix: In Cribl Edge 4.1.1.

2022-11-23 – v.4.0.0 – Teleported Edge Nodes don’t display Distributed Settings or Diagnostics Settings [CRIBL‑13868]

Problem: When displayed via remote access from the Leader (“teleporting”), a Worker’s/Edge Node’s Settings UI omits the Distributed Settings and Diagnostics left-nav links. This blocks remote access to features like configuring TLS communications between Edge Node and Leader.

Workaround: Access the Worker’s UI directly on its host’s port 9000 (or other configured port). As a precondition, you might need to undo any Disable UI Access setting on the parent Fleet.

Fix: In Cribl Edge 4.0.1.

2022-11-22 – v.3.2.0–4.0.0 – Chain Function creates extra Pipelines and Functions during initialization [CRIBL‑13860]

Problem: Under certain circumstances, when initializing, the Chain Function created multiple unneeded Pipelines and Functions.

Fix: In Cribl Edge 4.0.1.

2022-11-21 – v.4.0.0 – Metrics service unavailable [CRIBL‑13826]

Problem: The Monitoring and Fleet pages don’t load, due to the systemd-tmpfiles service cleaning some of the Cribl socket files from the /tmp/ directory.

Workaround: Stop the host system from cleaning the socket files from the /tmp/cribl-* directory. For example, on an Amazon EC2 instance, add a new tmp.conf file to /etc/tmp.conf with the line X /tmp/cribl-*. Then restart the tmpfiles cleanup service with: systemctl restart systemd-tmpfiles-clean.service. Finally, restart the Cribl server.

Fix: This issue no longer applies to Cloud.Cloud as of 4.0.1. On-prem customers should use the workaround, which will be added to the deployment documentation as a standard practice.

2022-11-18 – v.4.0.0–4.0.3 – License usage visible only for current day [CRIBL‑13811]

Problem: In the listed versions, selecting Monitoring > System > License displays usage only for the current day.

Fix: In Cribl Edge v.4.0.4.

2022-11-16 – versions 3.4.0–4.0.4 – Source PQ in Smart mode can trigger excessive memory usage and OOM failures [CRIBL‑13761]

Problem: Enabling Source-side Persistent Queues in Smart mode can trigger excessive memory usage, leading to out-of-memory failures and Worker Node instability.

Workaround: If you encounter OOM failures (most likely with high throughput), set PQ to Always On mode instead.

Fix: In Cribl Edge 4.1.

2022-11-14 – version 4.0.0 – `Forbidden` error banners mistakenly displayed to non-admin users [CRIBL‑13681]

Problem: Non-admin users (with the reader_all Role) are mistakenly shown a continuous string of Forbidden error banners.

Workaround: Available upon request (documented in the ticket), but cumbersome. Cribl recommends upgrading instead.

Fix: In Cribl Edge 4.0.1.

2022-11-13 – v.4.0.0–4.0.2 – Edge/Kubernetes Logs Source repeats logs due to incorrect timestamp [CRIBL-13900]

Problem: The Kubernetes Logs Source is collecting redundant information for containers that don’t emit their own timestamps. This Source assumes “current time” when the timestamps are missing, causing it to incorrectly stream older logs during restarts.

Fix: In Cribl Edge 4.0.3.

2022-11-08 – v.4.0.0 – Cribl Edge honeycomb display doesn’t render null values [CRIBL-13595]

Problem: When you view Edge Nodes in Map View, a honeycomb displays values for each of the metrics you select in the Measure drop-down. When a value is null for a particular the Edge Node, the honeycomb should display null; instead, the value incorrectly renders as ul i.

Fix: In Cribl Edge 4.0.1.

2022-11-08 – v.4.0.0 – Broken in-app doc links [DOC‑100]

Problem: The following Sources have broken in-app help links: Kubernetes Logs, Kubernetes Metrics, Windows Event Logs, and Windows Metrics.

Workaround: Use the above links to access these Sources’ online documentation.

Fix: In Cribl Edge 4.0.1.

2022-11-07 – v.4.0.0 – Spurious “Secret decrypt failed with error" log messages appear after upgrade [CRIBL‑13554]

Problem: After upgrading to Cribl Edge 4.0.0, multiple Secret decrypt failed with error messages might appear in logs. These are spurious, and you can disregard them.

Fix: In Cribl Edge 4.0.1.

2022-11-07 – v.4.0.0 – Upgrade status shown on Group Upgrade page doesn’t match Global Settings [CRIBL-13566]

Problem: The Enable automatic upgrades status shown on the Settings > Group Upgrade page doesn’t match the slider selection on the Settings > Global Settings > Upgrade tab.

Workaround: View the Enable automatic upgrades slider on the Settings > Global Settings > Upgrade tab to verify the current upgrade status.

Fix: In Cribl Edge 4.0.1.

2022-11-04 – v.3.5.1–4.0.4 – Avg Thruput (Bytes Per Second) not displayed for internal logs or metrics [CRIBL‑13530]

Problem: No Avg Thruput (Bytes Per Second) data is displayed on Cribl Internal logs and metrics Sources’ Charts tab, nor for these Sources on their attached Routes. This is by design, because Cribl Edge does not perform BPS calculations for internal logs or metrics. But the visual disparity can be confusing.

Fix: In Cribl Edge 4.1.

2022-11-03 – v.4.0.0 – Sorting by column on Manage > Edge Node page results in blank list [CRIBL‑13505]

Problem: When you click a column header on the Manage > Edge Node page, the list goes blank and rows are not sorted.

Workaround: Click the heading a couple of times to refresh the list.

Fix: In Cribl Edge 4.0.1.

2022-11-01 – All versions through Current – Source PQ in Smart mode could cause premature backpressure [CRIBL-13414]

Problem: Source-side persistent queuing enabled in Smart mode might trigger backpressure before the configured maximum queue size is reached. For TCP Sources, this will send backpressure back to the sender. For Sources using the UDP protocol, it will cause events to be dropped.

Fix: None, works as designed.

2022-11-01 – v.4.0.0 through Current – Default AppScope Config file can’t be restored when in use [CRIBL‑13400]

Problem: When the default AppScope Config file is assigned to an AppScope Source filter, you can’t restore it to its default settings. You’ll see an error message that the AppScope Config file is currently in use.

Workaround: On the AppScope Source’s AppScope Filter Settings tab, delete the default AppScope Config from the Allowlist. Then, edit the AppScope Config Knowledge object to restore its default settings. You can now add this AppScope Config, with its restored settings, back to the AppScope Source’s Allowlist.

Fix: Version TBD.

2022–10-20 – v.4.0.0 through Current – Pipelines with Clone and other Functions can show inconsistent total processing times [CRIBL-13125]

Problem: For Pipelines containing both a Clone Function and an async Functions (like Redis), the Pipeline Diagnostics modal’s Pipeline Profile > Process Time graph will sum up the duration of all Functions’ processing times. This sum can exceed the Pipeline’s total duration displayed in the Summary.

Fix: Version TBD.

2022-11-03 – v.4.0.0 – Git settings don’t immediately populate Commit modals [CRIBL‑13501]

Problem: After you save Git settings (such as a default commit message), the new settings do not immediately appear in Commit/Git Changes modals.

Workaround: A hard or soft refresh will close the modal, but when you reopen it, it will reflect your new settings.

Fix: In Cribl Edge 4.0.1.

2022-10-12 – All versions through 3.5.4 – Event preview can hide some fields [CRIBL‑12914]

Problem: Preview panes can hide some fields at the bottom of each event.

Workaround: Resize your browser window to give the preview pane more depth.

Fix: In Cribl Edge 4.0.

2022-10-11 – All versions through 4.1 – GitOps Push mode doesn’t support ad hoc Collection jobs [CRIBL‑12868]

Problem: If you’ve enabled the GitOps Push workflow, you will be unable to run ad hoc Collection jobs.

Workaround: There are two options. 1. Temporarily disable the Push workflow in your environment. 2. Create a scheduled Collection job (with a relaxed cron schedule) on your dev branch, and push it to production through your Push workflow.

Fix: Version TBD.

2022-10-04 – All versions through 3.5.4 – File Monitor Source fails with high-volume logs and file rotation [CRIBL‑12762]

Problem: The File Monitor Source might stop reading logs from an open file, if the log rotation service renames that file.

Workaround: Restart the File Monitor Source.

Fix: In Cribl Edge 4.0.

2022-10-03 – All versions through 4.0.4 – New Help drawers open under pinned Help drawers [CRIBL-12737]

Problem: If you’ve pinned a Help drawer open, it’s possible to open additional Help drawers behind the pinned drawer.

Workaround: Close the pinned drawer to see the newly opened drawer.

Fix: In Cribl Edge 4.1.

2022–09-26 – All versions through 4.0.0 – Manage Groups/Fleets pages count Packs in Pipeline totals [CRIBL‑12602]

Problem: On the Manage Groups and Manage Fleets pages, the Pipelines column overstates the actual number of Pipelines, because it includes Packs in each Fleet’s total count.

Workaround: For each Fleet, click the link in its Pipelines column to see the Fleet’s actual list of Pipelines.

Fix: In Cribl Edge 4.0.1.

2022-09-19 – v.4.0.0 – Subfleet search doesn’t return parent Fleets [CRIBL‑12465]

Problem: On the Manage Fleets page’s Fleets tab, searching for a Subfleet name returns only the Subfleet, without information about its parent Fleets.

Fix: In Cribl Edge 4.0.1.

2022-09-06 – v.3.3.0–3.5.4 – Load-balanced Destinations with PQ enabled: Buffer flush errors and memory leaks [CRIBL‑12203]

Problem: Destination logs might show errors of the form: Attempted to flush previously flushed buffer token. No data has been lost, but this will lead to a memory leak until the Edge Node is restarted. This is most likely to occur when there are more available downstream hosts than a Max connections limit configured on the Destination.

Workaround: To prevent the bug from occurring, set Max connections to the default 0 (enabling unlimited connections). This will ensure that all discovered hosts enter the connection pool. If you encounter the bug, restarting the affected Edge Nodes (or letting an OOM the killer do so) will clear the memory leak.

Fix: In Cribl Stream 4.0.

2022-08-29 – All versions through 3.5.3 – Sources reject connections on ACME certs with no Subject [CRIBL‑12097]

Problem: With multiple Sources, if you configure TLS mutual authentication using an ACME certificate with an empty Subject field, Cribl Edge will reject connections – even though RFC 6125 allows for Subject to be empty. This affects: Splunk Sources, Cribl internal Sources, Syslog, TCP, TCP JSON, Metrics, HTTP/S, Amazon Firehose, Elasticsearch API, DataDog, Grafana, Loki, Prometheus, and Windows Event Forwarder.

Workaround: Enter any value in the certificate’s CN field. Any entry will cause CCribl Edge to match on the certificate’s SAN (subjectAltName) extension.

Fix: In Cribl Edge 4.0.

2022-08-20 – v.3.2.0–4.0.3 – Deleting or modifying default Mapping Rule reclassifies Cribl-managed Cloud Workers as hybrid Workers [CRIBL‑11983]

Problem: Cribl.Cloud’s UI currently allows deleting or modifying the default Mapping Rule. By doing so, an admin can inadvertently reclassify Cribl-managed Cloud Workers as hybrid Workers. These might not be supported by your Cribl.Cloud plan.

Workaround: If you have an Enterprise plan, create a hybrid Worker Group to manage the resulting hybrid Workers.

Fix: In Cribl Edge 4.0.4.

2022-08-19 – v.3.5.1–3.5.4 – Using GitOps environment variables crashes the Leader [CRIBL‑11972]

Problem: Where deployments rely on CRIBL_GIT_* environment variables, the Leader might crash on startup.

Workaround Restart the Leader, disabling the environment variables if necessary.

Fix: In Cribl Edge 4.0.

2022-08-27 – All versions through 4.1.3 – Azure Event Hubs Destination with PQ drops events when inbound data is interrupted [CRIBL-17661, replaces CRIBL‑12649]

Problem: When Azure Event Hubs (and other Kafka-based Destinations) have persistent queues (PQs) configured, an interruption of inbound data flow (e.g., due to network issues) can cause the Destination to start dropping events.

Workaround: On the Source whose events are being interrupted, configure an Always On persistent queue. This buffering will cause the Destination to drop fewer events. Note that persistent queuing will not engage until Cribl Edge has exhausted all attempts to send the data to the Kafka receiver.

Fix: In Cribl Edge 4.2.

2022-08-25 – Versions 3.4.1 through Current – Splunk Load Balanced Destination degradation with acks enabled [CRIBL‑12066]

Problem: On a Splunk Load Balanced Destination, enabling the Minimize in-flight data loss (acknowledgments) field can cause high CPU drain and backpressure.

Workaround: On the Splunk LB Destination’s Advanced Settings tab, disable Minimize in-flight data loss.

Fix: Planned for Cribl Edge 4.2.2.

2022-08-16 – All versions through Current – Splunk HEC shows higher outbound data volume than other Splunk Destinations

Problem: Events sent to the Splunk HEC Destination will show higher outbound data volume than the same events sent to the Splunk Single Instance or Splunk Load Balanced Destinations, which use the S2S binary protocol.

Fix: Version TBD.

2022-08-16 – v.3.1.2–3.5.2 – Auto timestamping randomly begins using current time [CRIBL‑11925]

Problem: After a random interval, the auto timestamping feature might insert the current time into the _time field instead of using the timestamp from the event.

Workarounds: Restart the Stream instances to temporarily remedy the problem, or enable an Event Breaker rule’s Manual Timestamp Format option.

Fix: In Cribl Edge v.3.5.3.

Problem: Changing the Target Type while configuring a new Notification target crashes the New Target modal.

Workaround: Set the Target type once only while the modal is open.

Fix: In Cribl Edge 3.5.3.

2022-08-08 – v.3.x through Current – `cidr-matcher` does not support IPv6 addresses [CRIBL‑11767]

Problem: The version of cidr-matcher supported by Cribl only supports IPv4 CIDR matching.

Fix: Version TBD.

2022-08-03 – v.3.5.0–3.5.1 – Cribl HTTP and Cribl TCP Sources are not enabled on Cribl.Cloud [SAAS‑1905]

Problem: Cribl.Cloud’s ⚙️ Network Settings >: Data Sources tab lists Ingest Address ports for the new Cribl HTTP and Cribl TCP Sources. However, these ports are not yet enabled.

Workaround: If you need these ports enabled before the next maintenance release, contact Cribl Support or your Solutions Engineer.

Fix: In Cribl Edge 3.5.2.

2022-08-02 – v.3.3.0–3.5.1 – Managed Edge Nodes mistakenly display “Unregistered” button [CRIBL‑11652]

Problem: Managed Edge instances erroneously display an Unregistered button. Managed Edge Nodes cannot be registered, because they pull their registration/licensing information from the Leader. So this button should not appear.

Workaround: Ignore the scary button. Your managed Edge Node has inherited its Leader’s registration.

Fix: In Cribl Edge 3.5.2.

Problem: On Cribl.Cloud, Cribl Edge’s General Settings > Upgrade & Share Settings display a Sharing and live help toggle, even though you cannot turn off telemetry on Cribl- or customer-managed (hybrid) Cloud Workers.

Workaround: Ignore this toggle. The UI controls enable you to slide the toggle to No and save the result, but this will have no effect.

Fix: In Cribl Edge 3.5.2.

2022-07-21 – v.3.5.0–3.5.2 – Job Inspector shows higher Bytes In than Monitoring dashboards [CRIBL‑11482]

Problem: The Job Inspector sometimes displays a higher byte count than other Monitoring dashboards display for the same collection job. The Job Inspector overstates the throughput because it disregards any Event Breakers and Filter expressions applied between initial collection and Pipelines.

Workaround: Rely on the Monitoring metrics to judge accurate data flow through Pipelines to downstream services.

Fix: In Cribl Edge 3.5.3.

2022-07-21 – v.3.4.0–3.5.1 – Data loss with Source-side persistent queueing enabled [CRIBL‑11478]

Problem: With Source-side PQ enabled, events got stuck and did not process through Pipelines. The root cause was a Rename Function that used wildcards to rename required internal fields (specifically, __pqSliceId and __offset).

Workaround: Where Rename Functions potentially rename internal fields (especially via Rename expressions that include wildcards), either disable PQ on Sources that feed the parent Pipeline, or test and narrow the expression to affect only the desired fields.

Fix: In Cribl Edge 3.5.2.

2022-07-21 – v3.5.1-3.5.4 – Fake Pack appears after upgrading to v.3.5.1. [CRIBL‑11481]

Problem: After an upgrade to v.3.5.1., a new Pack appears on the Manage Packs page. This Pack has no Display Name and no contents.

Workaround: Delete the ghost Pack.

Fix: Not observed as of Cribl Edge 4.0.

2022-07-20 – v3.5.0–3.5.1 – Manage Edge Nodes page fails to lazy-load Workers when scrolled [CRIBL‑11474]

Problem: Scrolling the Manage Edge Nodes page fails to load more than 50 Edge Nodes.

Fix: In Cribl Edge 3.5.2.

2022-07-20 – v3.5.0–4.0.4 – Cribl Edge/Windows: Upgrading ignores existing mode (etc.) settings [CRIBL‑11467]

Problem: When rerunning Windows installers on a system that hosts a previous Cribl version as a managed Edge Node, the installer does not read the distributed mode or other settings. The new version might install as a Leader instance.

Workaround: Run the new version’s installer using the same mode and other options you used when installing the preceding version.

Fix: In Cribl Edge 4.1.

2022-07-19 – v.3.5.0–3.5.1 – Cribl Edge/Windows: Syslog Source disallows UDP binding [CRIBL‑11439]

Problem: On Cribl Edge/Windows, attempting to bind the Syslog Source to a UDP port might trigger an error of the form: bind EADDRINUSE 0.0.0.0:<port number>. The reason is that Edge currently does not fully support the socket.bind on Windows.

Fix: In Cribl Stream 3.5.2.

2022-07-13 – v.3.5.0–3.5.1 – Cribl Edge: Fleet > List View tab doesn’t correctly filter Edge Nodes [CRIBL‑11183]

Problem: A Fleet Home page’s List View tab doesn’t correctly filter Edge Nodes. This tab displays all the Edge Nodes across all Fleets, instead of only those assigned to the Fleet.

Workaround: The Map View page is filtered correctly.

Fix: In Cribl Edge 3.5.2.

2022-07-04 – All versions through v.3.5.3 – No extended characters in Publish Metrics Function > Event field name field [CRIBL‑8968, CRIBL‑10984]

Problem: A Publish Metrics Function’s Event field name values should contain only letters, numbers, underscores (_), and . characters (to separate names in nested structures). Using other characters will cause the parent Pipeline to stop sending events. Cribl Edge 3.5.0 and above check for valid characters when you save the Function’s configuration, but in prior versions, the invalid config will save without an error message – the failure will happen at runtime, with errors echoed only in the logs.

Workaround: Ensure that Event field name values contain no extended characters.

Fix: In Cribl Edge 3.5.4.

2022-07-01 – All versions through 3.5.0 – S3-based Sources’ unread Region triggers auth errors [CRIBL‑10981]

Problem: On Amazon S3–based Sources, if you concatenate the AWS Region into the Queue field’s URL or ARN, the SQS client is correctly configured to that region, but the S3 client is not. You will see authentication errors of the form: The AWS Access Key ID you provided does not exist in our records.

Workaround: Explicitly set the Region drop-down, even if the URL/ARN already contains the same Region.

Fix: In Cribl Edge 3.5.1.

2022-06-30 – All versions through Current – Persistent queue with compression requires oversize max queue limit [CRIBL‑10965]

Problem: If you configure persistent queueing to both enable Compression and set a Max queue size limit, set that limit optimistically and monitor the results. Due to an error in computing the compression factor, Cribl Edge will not fully use Max queue size values below or equal to the volume’s available disk space. This can lead to a mostly empty disk, lost data (either blocked or dropped), and/or excessive backpressure.

Workaround: With Compression enabled, set the Max queue size to a value higher than the volume’s total available physical disk space (disregarding compression). Alternatively, leave Max queue size empty, to impose no limit. Monitor overall throughput and the queue size (if PQ engages), to verify that Cribl Edge is maximizing the queue.

Fix: Version TBD.

2022-06-29 – v.3.4.2–3.5.1 – Can’t install a new Cribl Edge instance as a named user [CRIBL‑10907, CRIBL‑11457]

Problem: Bootstrapping a new Leader or Worker with a command of this form fails, with no systemd service created: cribl boot‑start enable ‑u <username>

The symptom is an error of the form: error: found user=0 as owner for path=/opt/cribl/local, expected uid=1001. This does not affect upgrades to existing instances.

Workaround: Issue the [sudo] chown -R <username>: /opt/cribl command a second time. Then reattempt the cribl boot-start command.

Fix: In Cribl Edge 3.5.2.

2022-06-28 – v.3.5.x through Current – Cribl Edge/Windows Limitation: Cannot upgrade Edge Nodes from the Leader’s UI [CRIBL‑10870]

Problem: Cribl Edge/Windows does not support upgrading Edge Nodes via the Leader’s UI.

Workaround: Rerun the Windows installer on each Edge Node, specifying the same installation options/parameters you used when installing the preceding version.

Fix: In Cribl Edge 4.3.

2022-06-27 – v.3.3.0–3.5.1 – Edge vs. Stream conflict in `/tmp` directory [CRIBL‑10806]

Problem: If you install Cribl Edge and Cribl Stream on the same machine or VM – e.g., running Edge as root and Stream as a cribl user – both services attempt to use the tmp/cribl_temp subdirectory. Starting Stream after Edge will throw an error of the form: EPERM: operation not permitted, rmdir '/tmp/cribl_temp'. Starting Edge after Stream will change the folder’s ownership, blocking subsequent restarts of Stream.

Workaround: For either Cribl Edge or Cribl Stream, set the CRIBL_TMP_DIR variable to a separate base subdirectory like /tmp/stream/ or /tmp/edge/ before starting the app.

Fix: Cribl Edge 3.5.2 and later set separate tmp/ subdirectories by default.

2022-06-24 – v.3.5.0 – Leader takes excessive time to report healthy status [CRIBL‑10768, CRIBL‑11127]

Problem: Upgrading from v.3.4.x to v.3.5.0 tripled the latency before some Leaders reported healthy status. The presumed cause was that load time increased due to an accumulation of persistent metrics, which progressively increased with more Worker Nodes and more uptime.

Workaround: If upgrading is not possible or insufficient, move the existing metrics subdirectory away from $CRIBL_HOME/state/metrics/. Cribl Edge will re-create that as an empty subdirectory, and begin accruing fresh metrics there.

Fix: In Cribl Stream 3.5.1.

2022-06-23 – v.3.5.0 – Logs not viewable at Worker Group level [CRIBL‑10665]

Problem: If you select Monitoring > Logs, and then select a Worker Group from the drop-down at upper left, you might see no logs at the Group level. Instead, you will see an error banner of the form:
ENOENT: no such file or directory, scandir '/opt/cribl_data/failover/log/group/<group‑name>'

Workaround: From the drop-down at upper left, select individual Worker Nodes to view their logs.

Fix: In Cribl Edge 3.5.1.

2022-06-09 – v.4.0.x–4.1.0 – Edge Settings mistakenly display Process Count controls [CRIBL‑10402]

Problem: Edge Fleets’ Fleet Settings > Worker Processes page incorrectly includes editable Process Count and Minimum Process Count controls. Each Edge Node is locked to 1 Worker Process, so changes to these fields’ values will have no effect.

Workaround: Ignore these two fields.

Fix: In Cribl Edge 4.1.1.

2022-06-01 – v.3.4.2-3.5.0 – Bootstrapping fails to create SystemD service file, or starts wrong service [CRIBL‑10250]

Problem: Bootstrapping a 3.4.2 Worker fails. The terminal displays no Enabled Cribl to be managed by ... confirmation message.

Workaround: Explicitly run the boot‑start CLI command.

Fix: In Cribl Edge 3.5.1.

2022-05-31 – v.3.4.2 – Cloning/loading Groups page breaks with `Config Helper service is not available...` error [CRIBL‑10228, CRIBL‑10229]

Problem: After cloning a Group containing at least one Pack, the UI hangs with a spinning pinwheel and an error banner reading Config Helper service is not available.... The root cause is that cloning the Group failed to copy over the Pack, leaving the new Group with broken references to it.

Workaround: Use the filesystem to manually copy missing Packs from the original Group to the cloned Group. If these Packs’ contents are not needed in the new Group, it’s sufficient to just create the parallel subdirectory with: mkdir $CRIBL_HOME/groups/$destGroup/default/$packName.

Fix: In Cribl Edge 3.5.

2022-05-19 – v.3.3.0–3.5.1 – System Metrics Source skips filesystem events on LVM volumes [CRIBL‑10092]

Problem: When monitoring LVM logical volumes, the System Metrics Source omits node_filesystem_* events.

Fix: In Cribl Edge 3.5.2.

2022-05-06 – v.3.5.0-3.5.4 – Running ~1,000 Edge Nodes crashes Leader [CRIBL‑9859]

Problem: Attempting to bring up approximately 1,000 Edge Nodes (even in small batches) can crash the Leader.

Workaround: To support up to 1,000 Nodes, apply this Node.js setting:
export NODE_OPTIONS=--max-old-space-size=8192

Fix: In Cribl Edge 4.0.

2022-04-28 – v.3.4.0–3.4.1 – REST Collector Misinterprets `.` character in Response Attributes [CRIBL‑9708]

Problem: Where a REST Collector’s Response Attributes contain a . character, it misinterprets this as an attribute nested within an object, not a literal character.

Workaround: If possible, use other Response Attributes to fetch the next page. Otherwise, skip this version, or downgrade to a known well-behaved version.

Fix: In Cribl Edge 3.4.2.

2022-04-27 – v.3.4.1 – Git Changes drop-down returns `Invalid Date` [CRIBL‑9698]

Problem: After opening the left nav’s Changes fly-out, the Version drop-down can display Invalid Date instead of commits’ dates.

Workaround: Upgrade your installed git client to v.2.1.1 or newer; or skip this Cribl Edge version.

Fix: In Cribl Edge 3.4.2.

2022-04-26 – v.3.4.1 – HTTP 500 error when git is absent [CRIBL‑9684]

Problem: A 500–Internal Server Error error banner can indicate that git is not installed on Cribl Edge’s host.

Workaround: Install git on the Leader or single instance. On affected Edge Nodes, install git, followed by a git init and an empty push to a new master branch. A simpler workaround for Edge Nodes is to just enable remote access (Stream, Edge) from the Leader.

Fix: In Cribl Edge 3.4.2.

2022-04-22 – v.3.3.0-3.4.1 – Datadog Agent API key ignored when forwarding metrics to Datadog [CRIBL‑9633]

Problem: On a Datadog Destination, when Allow API key from events is set to Yes, the API key from a Datadog Agent Source is not correctly emitted with metric events sent to Datadog. Instead, Cribl Edge simply sends the static API key configured in the Destination. (This affects metrics events only; other data types correctly forward the event’s API key.)

Workaround: Set Allow API key from events. to No. Configure the desired API key in the Destination’s static API key field. (If you need multiple API keys, can create multiple Destinations, and route events to each based on the __agent_api_key field value. This value contains the Agent API key per event.)

Fix: In Cribl Edge 3.4.2.

2022-04-21 – v.3.4.1 – Failed restart, `Uncaught (in promise)` error, after changing Authentication settings [CRIBL‑9614]

Problem: After changing Authentication settings, or configuring an external auth provider, Cribl Edge might fail to restart. Indications are errors of the form: Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'workerRemoteAccess'), or: Uncaught (in promise) TypeError: l.api is undefined.

Workaround: Directly edit the cribl.yml file’s auth: section.

Fix: In Cribl Edge 3.4.2.

Problem: On certain iPads, we’ve seen the Cribl.Cloud login page’s left text column repeated twice more across the display. These unintended overlaps prevent you from selecting (or tabbing to) the Log in with Google button.

Workaround: If you encounter this, the only current workarounds are to either use Google SSO on a desktop browser, or else use a different login method.

Fix: No planned fix.

2022-04-19 – v.3.3.0–3.4.2 – Spurious buffer token flush error in TCP-based Destinations with PQ enabled [CRIBL‑9565]

Problem: This error message can mistakenly appear in logs: Attempted to flush previously flushed buffer token. You can ignore it on TCP-based, load-balanced Destinations (Splunk Load Balanced, TCP JSON, Syslog/TCP, and Cribl Stream) with persistent queueing enabled.

Fix: In Cribl Edge 3.5.

2022-04-18 – All versions through Current – Upgrade Version List is Limited to Latest Version [CRIBL‑9568]

Problem: The Choose Version drop-down list displays the latest version only. This creates an issue when you want to incrementally upgrade between prior versions.

Fix: Version TBD.

2022-04-18 – v.3.3.1–4.0.0 – Lookup Functions intermittently fail [CRIBL‑9539]

Problem: Lookup Functions within some Pipelines were skipped up to ~20% of the time. Restarting Cribl Edge resolves this temporarily, but the failure eventually resurfaces as the new session proceeds.

Workaround: Where a Lookup Function fails, substitute an Eval Function, building a ternary JS expression around a C.Lookup method.

Fix: In Cribl Edge 4.0.1.

2022-04-15 – All versions through 3.4.1 – Git permission errors [CRIBL‑9530]

Problem: Multiple Linux distro’s have backported a permissions-restriction patch from git-2.35.1 to earlier versions, notably including Ubuntu 20.04 LTS. If you see errors of the form fatal: unsafe repository ('/opt/cribl' is owned by someone else) on the command line or in the Cribl Edge UI, this indicates an ownership mismatch (current user versus file base) on the directory corresponding to $CRIBL_HOME or $CRIBL_VOLUME_DIR.

Workaround: Use chown to recursively set permissions on all files in /opt/cribl/ (or in or $CRIBL_VOLUME_DIR’s target directory) to match the user running Cribl Edge.

Fix: In Cribl Edge 3.4.2.

2022-04-15 – v.3.4.0 – Workers report incomplete metrics [CRIBL‑9524]

Problem: After upgrading to v.3.4.0, Worker Nodes failed to report all metrics. Missing metrics were logged with a warning of the form: failed to report metrics, and with a reason of the form: Cannot read property 'size' of undefined.

Fix: In Cribl Edge 3.4.1.

2022-04-14 – v.3.4.0-3.4.1 – Monitoring visualizations lack color indicators [CRIBL‑9510]

Problem: When you hover over the Monitoring page’s graphs, the expected red/yellow/green status indicators are missing.

Fix: In Cribl Edge 3.4.2.

2022-04-07 – v.3.4.0–3.4.1 – Chain Function referencing a Pack triggers errors on Pack’s Functions [CRIBL‑9235]

Problem: After you define a Chain Function that references a Pack, Functions on Pipelines within that Pack will throw errors of the form: Failed to load. Function <Function‑name> is missing. Please fix, disable, or remove the Function. You might also find that you cannot add more Functions within the Pack. (However, data might continue to flow, despite these errors.)

Workaround: Remove the Chain Function, or its Pack reference. Refactor your flow logic to avoid this combination.

Fix: In Cribl Edge 3.4.2.

2022-04-05 – v.3.4.0 through Current – Upgrade to 3.4.0 disables Worker/Node UI access [CRIBL‑9175]

Problem: Upon upgrading to v.3.4.0, even if the (prior, global) Worker/Node UI access option was set to Yes, the new per–Fleet toggles are all set to No by default.

Workaround: On the Manage Fleets page, re-enable the UI access toggles as desired. If these toggles are not displayed (with a Free license), edit groups.yml to add the key-value pair workerRemoteAccess: true within each desired Fleet.

Fix: Version TBD.

2022-03-29 – v.3.4.0 – `this.dLogger.isSilly is not a function` errors [CRIBL‑9019]

Problem: Errors of this form have been observed with multiple triggers: 1. Disabling a customized Source or Destination (it remains enabled). 2. Importing and deleting a Pack. 3. Changing a channel’s logging level.

Workaround: Delete and re-create affected Sources and Destinations. (No workaround has been identified for the Pack or logging errors.)

Fix: In 3.4.1.

2022-03-23 – v.3.4.0 – Monitoring page error in distributed environments with many Edge Nodes [CRIBL‑8938]

Problem: The Monitoring page displays an error where distributed environments have more than 30 Edge Nodes.

Fix: In 3.4.1.

2022-03-23 – Collect parameters values not evaluated as expressions [CRIBL‑8932]

Problem: In a REST Collector’s Collect parameters table, entering a parameter name like “earliest” as plain text will cause it to be interpreted as a literal string. This will not be evaluated as the earliest time-range parameter.

Workaround: Backtick the parameter name, e.g.: `earliest`.

Fix: Tooltip clarified in Cribl Edge 4.0.

2022-03-23 – Upgrading v.3.3.1 Leader to v.3.4.0 blocks upgrade to Workers [CRIBL‑8918]

Problem: When you explicitly upgrade a Leader Node to 3.4.0, you can’t push upgrades to Workers.

Workaround: Automatic upgrades work as expected. At Settings > Global Settings > System > Upgrade, set Disable Automatic upgrades to No.

Fix: In 3.4.1.

2022-03-17 – All versions through 3.4.0 – Failed systemd restarts due to erroneous `ExecStopPost` command [CRIBL‑8807]

Problem: On systemd, the Cribl service can stop with an unsuccessful return code. This was caused by a malformed ExecStopPost command in Cribl’s provided cribl.service file.

Workaround: In cribl.service, delete the whole ExecStopPost=... line, and change the Restart parameter’s value from on-failure to always. (The first deletion necessitates the second change.)

Fix: Versions 3.4.1 and later install a cribl.service file incorporating both the above changes. If you are upgrading from v.3.4.0 or earlier, and you notice dead Workers, check and update your existing cribl.service configuration to match the changes above.

2022-03-17 – v.3.4.0 – Source PQ re-engages while draining, causing blockage/backpressure [CRIBL‑8800]

Problem: A Source with Persistent Queueing enabled can mistakenly re-engage queueing while still draining its queue – leading to blocked throughput and backpressure. This occurs after a Destination goes offline and back online, once or more, leaving the Source in an undetermined queue state.

Workaround: Wait 60 seconds for another queue drain event, to see if PQ behavior goes back to normal. If the problem persists, restart Cribl Stream.

Fix: In Cribl Edge 3.4.1.

2022-03-17 – v.3.4.0 – Monitoring pages break without git [CRIBL‑8799]

Problem: Unless git is installed locally, two Monitoring pages fail to display, with errors of the form Versioning not supported. The pages are Monitoring > Overview and Monitoring > System > Licensing. To resolve the errors, git must be installed even on single-instance deployments.

Workaround: Install git.

Fix: In 3.4.1.

2022-03-16 – v.3.2.2-3.4.0 – Upgrading from earlier versions degrades performance [CRIBL‑8784]

Problem: After upgrading to any of the indicated versions, customers with active Splunk Destinations noticed that CPU load increased, triggering backpressure more readily.

Workaround: Skip these versions, or downgrade to a known well-behaved version.

Fix: In 3.4.1.

2022-03-11 – v.3.4.0 – In Free versions, Worker/Node UI access can’t be accessed via UI [CRIBL‑8707, CRIBL‑8945]

Problem: In v.3.4, Cribl moved the Worker/Node UI access toggle from global ⚙️ Settings > System > Distributed Settings to per–Fleet toggles on the Manage Fleets page. But with a Free license, these controls aren’t displayed in the UI at all.

Workaround: Directly edit groups.yml to add the key-value pair workerRemoteAccess: true within each desired Fleet.

Fix: In 3.4.1.

2022-03-08 – versions 3.3.1 through Current – GitOps + License expiration = Catch-22 [CRIBL‑8600]

Problem: If your Enterprise license expires while you have enabled the GitOps Push workflow, you will encounter the following block: Cribl Stream is in read-only mode, triggering a Forbidden error when you try to update your license key. But you also cannot reset the workflow from Push to None, because the expired license disables GitOps features.

Workaround: Contact Cribl Support for help updating your license.

Fix: Version TBD.

2022-03-07 – versions 3.2.x-3.4.x – Undercounted `total.in_bytes` dimension metrics [CRIBL‑8572]

Problem: The cribl.logstream.total.in_bytes dimension sent to Destinations can show a lower data volume (against license quota) than cribl.logstream.index.in_bytes. This latter index.in_bytes dimension displays the correct license usage, and matches what’s reported on the Monitoring dashboard.

Workaround: Within the cribl_metrics_rollup Pipeline that ships with Cribl Stream, disable the Rollup Metrics Function.

Fix: Couldn’t reproduce the error.

2022-03-03 – versions 3.3.x – Elasticsearch API Source (distributed mode) stops receiving data after upgrade to 3.3.x [CRIBL‑8515]

Problem: After upgrading a distributed deployment to version 3.3.x, the ElasticSearch API Source might stop receiving data. This occurs when your existing Source config’s Authentication type was set to Auth Token. The root cause is a 3.3.x schema change that unset this auth type to None.

Workaround: In the Elasticsearch API Source configuration, reset the Authentication type to Auth Token. Then commit and deploy the restored config.

Fix: In Cribl Stream 3.4.

2022-02-24 – versions 3.3.x – After upgrade to 3.3.0, ElasticSearch Destination can’t write to indexes whose name includes `-` [CRIBL‑8451]

Problem: After upgrade to version 3.3.x, LogStream cannot send data to Elasticsearch indexes whose name includes a hyphen (‑).

Workaround: In the Elasticsearch Destinations’s Index or Data Stream field, surround these index names in “quotes” or backticks.

Fix: LogStream 3.3.1 added format validation and error-checking.

2022-02-18 – versions 3.3.x – Worker/Leader communications blocked by untrusted TLS certificate after upgrade to 3.3.0 [CRIBL‑8361]

Problem: With TLS enabled on Worker/Leader communications, after an upgrade from LogStream 3.2.x to v.3.3.x, Workers might fail to communicate with the Leader due to an untrusted SSL certificate on the Leader. Logs will show connection errors of the form: "unable to verify the first certificate" or self signed certificate in certificate chain.

Workaround: On each affected Worker: In the instance.yml file’s’ distributed: master: tls: section, set: "rejectUnauthorized: false". Then restart Cribl Stream.

Fix: In 3.4.1.

2022-02-18 – versions 3.3.x – Can’t upgrade Workers to 3.3.x via UI [CRIBL‑8346]

Problem: Attempting to upgrade on-prem Workers to version 3.3.x via the UI can fail, with an error of the form: Group <group‑name> is a managed group.

Workaround: Edit the Leader’s $CRIBL_HOME/local/cribl/groups.yml file to delete all instances of the key-value pair onPrem: false. Then, resave the file and reload the Leader.

Fix: In Cribl Stream 3.4.

2022-02-17 – versions 3.2.2-3.3.x – REST Collector pagination fails on duplicate/nested attribute names [CRIBL‑8352]

Problem: The REST Collector does not consistently differentiate between multiple (nested) attributes that share the same name. This can break pagination that relies on a Response Body Attribute.

Workaround: If possible, select a different Pagination method, or specify a unique attribute name.

Fix: In Cribl Stream 3.4.

2022-02-17 – v.3.3.0 – Do not configure InfluxDB Destination on LogStream 3.3.0 [CRIBL‑8354]

Problem: Configuring InfluxDB Destinations can fail on LogStream 3.3.0. This problem has been observed only on distributed deployments, and data does flow to InfluxDB. But the InfluxDB config will not appear in LogStream’s UI, and the broken config will block editing of other Destinations.

Workaround: If you need to send data to InfluxDB, skip v.3.3.0, and wait for the next release. If you encounter the broken UI, edit outputs.yml to remove any influxdb_output sections, and then restart LogStream. This will unblock UI-based editing of other Destinations.

Fix: In LogStream 3.3.1.

2022-02-10 – All versions, 3.2.1+ – Syslog Source’s previewed data doesn’t proceed through Routes [CRIBL‑8210]

Problem: The Syslog Source’s General Settings > Input ID field’s default filter expression is __inputId.startsWith('syslog:in_syslog:'). The same filter appears in the Preview Full tab’s Entry Point drop-down, except without the final colon. This means that data sent using Preview Full’s version of the filter will not go through Routes that use the Input ID version.

Workaround: When sending data from Preview Full to a Route, if both use the syslog:in_syslog filter, edit the Route’s filter to remove the final colon. This will make the filter identical in both places, routing data correctly.

2022-02-10 – version 3.3.0-3.5.x – Slow-Mo Notification Notifications [CRIBL‑8205]

Problem: After you create a Notification on a Destination, the new Notification might take up to several minutes to appear on the Destination config modal’s Notifications tab.

Details: This delay has not been reproduced consistently. Some Notifications appear immediately.

Fix: In Cribl Edge 4.0.

2022-02-08 – versions 3.3.x – Missing event from Datadog Agent v.7.33.0+ [CRIBL‑8132]

Problem: If ingesting metrics from Datadog Agent 7.33.0 or later (i.e., you have set the DD_DD_URL environment variable or the dd_url YAML config key), LogStream’s Datadog Agent Source will not ingest agent_metadata events. This is due to a breaking change in Datadog Agent 7.33.0, which split out part of the prior /intake/ endpoint into a new /api/v1/metadata endpoint.

Workaround: Use Datadog Agent v.7.32.4 or earlier.

Fix: In Cribl Stream 3.4.

2022-02-08 – All versions through 3.3.x – Syslog and Metrics Sources’ default filter expression needs correction [CRIBL‑8115]

Problem: In the Syslog and Metrics Sources’s Logs tab, the auto-generated filter expression (channel =='input:in_syslog') is not specific enough and yields no results, because it doesn’t address these Sources’ two channels (input:in_syslog:tcp and input:in_syslog:udp).

Workaround: Replace this default filter expression with channel.startsWith('input:<input_ID>'). For example: channel.startsWith('input:in_syslog:').

Fix: In Cribl Stream 3.4.

2022-02-07 – v.3.2.2-3.4.0 – Okta integration fails after upgrading from v.3.1.3 to v.3.2.2 [CRIBL‑8105]

Problem: Okta (OpenID Connect) authentication on Cribl Stream fails behind a proxy, when using environment variables (http_proxy or https_proxy).

Workaround: Configure the proxy to work in transparent mode, to bypass the http*_proxy variables. If the proxy is required, leave the User Info URL empty, and Cribl Stream can obtain user details from the JWT token.

Fix: In 3.4.1.

2022-02-04 – v.3.2.2-3.4.0 – Mask Function slows down post-processing Pipeline [CRIBL‑8043]

Problem: A post-processing Pipeline with a Mask Function can slow down drastically, showing high CPU usage on Workers.

Workaround: If practical, promote the same Mask Function to a pre-processing or processing Pipeline.

Fix: In Cribl Stream 3.4.1.

2022-2-03 – All versions through 3.4.0 – JSON Serialize Function in post-processing Pipeline won’t process [CRIBL‑8013]

Problem: A JSON Serialize Function in a post-processing Pipeline will throw errors of the form: Failed to process event in Function: Maximum call stack size exceeded.

Workaround: Promote the Serialize Function to a processing Pipeline, upstream from the Destination.

Fix: In 3.4.1.

2022-02-02 – Versions 3.0.0 through Current – Upgrading a Pack overwrites Lookup and sample-data customizations [CRIBL‑7998]

Problem: Upgrading a Pack overwrites any customizations you’ve made to the Pack’s original/default Lookup tables and sample-data files.

Workaround: Duplicate the Pack’s default Lookup tables and sample-data files, and customize your duplicate copies. Upgrading the Pack will not overwrite your local copies.

Fix: Version TBD.

2022-01-11 – v.3.2.2 – Git Collapse Actions disaggregates [CRIBL‑7638]

Problem: With Git Settings > Collapsed actions enabled, the combined Commit & Push button appears in the modal as expected, but then saving a new change splits it back into two separate buttons: Commit and Deploy.

Workaround: Disable the Collapse actions setting, then commit and deploy separately.

Fix: In LogStream 3.3.

2022-01-19 – v.3.2.2 – Usernames containing certain letters cause some API requests to fail [CRIBL‑7728]

Problem: Usernames containing certain letters can cause some API requests to fail with an Invalid character in header content ["cribl-user"] error. This can happen even when the username is valid for an Identity Provider.

Workaround: In your usernames, make sure that the letters you use are limited to the letters in Latin alphabet no. 1 as defined in the ISO/IEC 8859-1 standard.

Fix: In LogStream 3.3.

2022-01-18 – v.3.2.0-3.4.0 – Diagnostics > System Info page omits some entries [CRIBL‑7731]

Problem: In current Cribl Stream versions, the Diagnostics > System Info page/tab omits certain statistics (sysctl, ulimits, network stats, etc.) that were previously displayed below the cpus and nets elements.

Workaround: The hidden information is nevertheless available within diagnostic bundles.

Fix: In Cribl Stream 3.4.1.

2022-01-04 – v.3.2.x – Enabling GitOps deletes LogStream’s `bin`, `logs`, and `pid`subdirectories [CRIBL‑7513]

Problem: Enabling GitOps with LogStream 3.2.x, via either CLI or UI, can cause the deletion of the bin, logs, and pidsubdirectories. This disables LogStream. The root cause is that git versions 2.13.1 and lower did not fully respect paths listed in .gitignore.

Workaround: Upgrade your git client to v.2.13.2 or higher, to correct the underlying behavior.

Fix: In LogStream 3.3.

2021-12-21 – v.3.2.2-4.0.x – Chain Function degrades CPU load and throughput [CRIBL‑7445]

Problem: Chaining Pipelines via the Chain Function can increase CPU load, and can signficantly slow down data throughput.

Workaround: Consolidate all Functions – per processing scenario – into a single Pipeline.

Fix: In Cribl Edge 4.1.

2021-12-15 – v.3.2.1-3.2.2 – Backpressure when writing to S3 and other file-based Destinations [CRIBL‑7364]

Problem: On file-based Destinations, enabling the Remove staging dirs toggle can trigger a race condition when inbound events per second reach a high rate. This leads to backpressure.

Workaround: Disable LogStream’s native Remove staging dirs option. Instead, as the user running LogStream, set up a cron job like this:
crontab -e
0 1 * * * find <stagingdir> -type d -empty -mtime +1 -delete

Fix: In LogStream 3.3.

2021-12-13 – v.2.4.4-3.1.1 – Upgrades via UI delete sample files [CRIBL‑7347]

Problem: Using the UI to upgrade LogStream versions prior to 3.1.2 will silently delete sample data files created by users. This affects using a Leader’s UI (any version) to upgrade Workers running LogStream version 3.1.1 or earlier. It also affects using the UI to upgrade the Leader itself, or a Single-Instance deployment, from v.3.1.1 or earlier to any newer version.

Workarounds: 1. Upgrade pre-3.1.2 versions via the filesystem. 2. If you choose to upgrade pre-3.1.2 versions via the UI, first save to the filesystem any custom sample files that you want to preserve. (Or, to copy directly from the filesystem, LogStream stores sample files internally at $CRIBL_HOME/data/samples, and stores an index of all sample files in /$CRIBL_HOME/local/cribl/samples.yml.)

Fix: Does not affect Workers running Cribl Stream (LogStream) 3.1.2 or higher. Does not affect self-upgrades of Leaders or Single Instances running v.3.1.2 or higher.

2021-12-10 – All Versions – API Reference is temporarily unstyled

Problem: Our documentation’s API Reference has temporarily lost some visual styling, while we swap in a new rendering component.

Workaround: Manually expand accordions, as needed.

Fix: Published as of 12/20/2021.

2021-12-09 – v.3.2.1 – File-based Destinations display an unavailable Persistent Queue option [CRIBL‑7293]

Problem: File-based Destinations’ Backpressure behavior drop-down misleadingly displays a Persistent Queue option, which is not really available on these Destinations. (Applies to Filesystem and some Amazon, Azure, and Google Cloud Destinations.) Selecting this option displays an error, and the configuration cannot be saved.

Workaround: Don’t fall for clicking that fake Persistent Queue option.

Fix: In LogStream 3.2.2.

2021-12-07 – v.3.2.1 – System Metrics Source’s documentation doesn’t open in Help drawer

Problem: After clicking the System Metrics Source’s Help link, the drawer displays the error: “Unable to load docs. Please check LogStream’s online documentation instead.” This Source’s documentation is also missing from the 3.2.1 docs PDF.

Workaround: Go to the live System Metrics docs page.

Fix: In LogStream 3.2.1, as duplicate of CRIBL-6319.

2021-12-07 – v3.1.2-3.1.3 – Increasing CPU load over time [CRIBL‑7268]

Problem: CPU load increases over time, with a slow increase in memory usage. Restarting LogStream temporarily resolves these symptoms. The root cause is needlessly collecting a high volume of full-fidelity metrics from the CriblMetrics source.

Workaround: Disable the CriblMetrics Source.

Fix: Upgrade to LogStream 3.2.1, which provides an option to disable the CriblMetrics Source’s Full Fidelity toggle.

2021-11-24 – v3.2.0 through Current – With processing Pipelines in QuickConnect, Preview Full doesn’t show Functions’ results [CRIBL‑7113]

Problem: If a processing Pipeline has been inserted between a QuickConnect Source and Destination, selecting the Pipelines page, and then selecting Preview Full with a data sample, doesn’t show the Pipeline’s effects on the OUT tab. This affects only Pipelines inserted in a QuickConnect connection line. (Attaching a pre-processing Pipeline to a QuickConnect Source doesn’t inhibit Preview.)

Workarounds: 1. Remove the Pipeline from QuickConnect, and re-create the same Source ‑> Pipeline ‑> Destination architecture under Routing > Data Routes. 2. Rely on Preview Simple.

Fix: Version TBD.

2021-11-24 – v3.2.0 – Data from Collectors and Collector-based Sources isn’t reaching Routes [CRIBL‑7109]

Problem: Routes are not recognizing data from Collectors and from the following Collector-based Sources: Prometheus Scraper, Office 365 Activity, Office 365 Services, and Office 365 Message Trace. This data will not flow through Routes, but will be sent to the default Destination(s).

Workarounds: 1. In Collectors’ config modals, open the Result Routing tab, Disable the Send to Routes default, and directly specify a Pipeline and Destination. (This option is not available in Prometheus Scraper or in the Office 365 Sources.) 2. Skip v.3.2.0.

Fix: In LogStream 3.2.1.

2021-11-17 – v.3.2.0 – TLS certs that use passphrases won’t decrypt private keys [CRIBL‑7049]

Problem: Sources whose TLS config uses a (known good) passphrase will fail to decrypt private keys. You will see a connect error, or an error of the form: TLS validation error, is passphrase correct?

Workarounds: 1. Edit the Leader’s or single instance’s inputs.yml file to insert a plaintext TLS passphrase. (See paths here.) Then commit and deploy the new config (distributed mode), or reload or restart the LogStream server (single instance). 2. Use a cert and key that do not require a passphrase. 3. Skip v.3.2.0.

Fix: In LogStream 3.2.1.

2021-11-17 – v.3.2.0 – Only one Chain Function works per Pipeline [CRIBL‑7044]

Problem: If you add more than one Chain Function to a Pipeline, only the first will take effect. Chain Functions lower in the stack will simply pass the data down to the next Function.

Workaround: Design your data flow to require at most one Chain Function per Pipeline.

Fix: In 3.2.1.

2021-11-17 – v.3.2.0 – Exporting a Pack to another Group requires a Leader restart [CRIBL‑7043]

Problem: Exporting a Pack to a different Worker Group via the UI succeeds, but opening the Pack on the target Group fails with a Cannot read property...undefined error.

Workaround: To resolve the error and make the target Pack accessible, restart the Leader. To prevent the error, export and import the Pack as a file.

Fix: In LogStream 3.2.1.

2021-11-17 – v.2.1.0 through Current – Re-enabling a Function group mistakenly re-enables all its Functions [CRIBL‑7053]

Problem: When you change a Function group from disabled to enabled, all of its Functions are enabled, regardless of their individual enabled/disabled states when the group was disabled.

Workaround: Avoid disabling and re-enabling Functions as a group (e.g., for testing or stepwise debugging purposes).

Fix: Version TBD.

2021-11-16 – v.3.2.0 – QuickConnected Source goes to wrong Destination [CRIBL‑7013, CRIBL‑7047]

Problem: Some QuickConnect connections send data to an unintended Destination. We’ve observed this in single-instance deployments that include the Cribl-supplied cribl_metrics_rollup Pipeline, or include other Pipelines/Packs with stateful Functions like Rollup Metrics or Aggregations. Data will either flow through Routes instead of your specified QuickConnect Destination, or will continue flowing to the original QuickConnect Destination after you drag the connection to a different Destination.

Workaround: Use the Data Routes interface to manage the Pipeline and stateful Functions indicated above. If your QuickConnect data doesn’t oblige a changed QuickConnect Destination, restart LogStream. This will stop data flow to the unintended Destination, and redirect it to the intended Destination.

Fix: In Cribl Edge 3.2.1.

2021-11-10 – v.3.2.0 – GitOps `Push` workflow displayed unsupported Revert button [CRIBL‑6961]

Problem: When a GitOps Push workflow has placed the UI in read-only mode, the Commit UI displayed a Revert button, even though reverting changes was not supported. Pressing the button simply triggered an error message.

Fix: In Cribl Edge/LogStream 3.2.1.

2021-10-11 – v.2.x-3.2.1 – Collectors do not filter correctly by date/time range [CRIBL‑6440]

Problem: Collectors can retrieve data from undesired time ranges, instead of from the range specified. If paths are organized by date/time, this can also cause Collectors to retrieve data from the wrong paths. The root cause is that in the Collector’s Run and Schedule configuration modals, time ranges are set based on the browser’s time zone, whereas LogStream’s backend assumes UTC.

Workaround: Offset the Earliest and Latest date/time values based on your browser’s offset from UTC.

Fix: In Cribl Edge/LogStream 3.2.2 and later, the Run and Schedule modals provide a Range Timezone drop-down to prevent these errors.

2021-10-06 – v.3.1.2 – Monitoring page omits Collector Sources’ data [CRIBL‑6412]

Problem: With functioning Office 365 Activity and Office 365 Services Sources, the Job Inspector reports data being retrieved, and Monitoring > Data > Destinations reports data being sent out. However, Monitoring > Data > Sources falsely reports no data being received. The problem is isolated to this Sources page. It might also affect the Office 365 Message Trace and Prometheus Scraper Sources.

Workaround: Use the Source modal’s Live Data tab, the Monitoring > System > Job Inspector page, and/or the Monitoring > Data > Destinations page to monitor throughput.

Fix: In LogStream 3.1.3.

2021-10-02 – v.3.2.x through Current – Kafka with Kerberos causes Workers’ continuous restart [CRIBL‑6674]

Problem: Configuring the Kafka Source or Destination with Kerberos authentication can send LogStream Workers into a continuous loop of restarts.

Workaround (multi-step):
1. In krb5.conf, set dns_lookup_realm = false and dns_lookup_kdc = false.
2. From krb5.conf’s [realms] section, extract the realm name (from the element name) and the FQDN (from the kdc key’s value, stripping any port number).
3. Run nslookup on either or both of the realm name and the FQDN. E.g.: nslookup mysubdomain.confluent.io and nslookup kdc.kerberos-123.local.
4. Add at least one of the returned IP addresses (which might be identical) to the etc/hosts file, as values in the format your platform expects.

Fix: Version TBD.

2021-09-30 – v.3.1.2 – High CPU load with CriblMetrics Source [CRIBL‑6319]

Problem: Enabling the CriblMetrics Source causes high event count and high CPU load.

Workaround: Adjust the cribl_metrics_rollup pre-processing Pipeline to roll up a wider Time Window of metrics.

Fix: Upgrade to LogStream 3.2.1, which provides an option to disable the CriblMetrics Source’s Full Fidelity toggle.

2021-09-29 – v.3.0.2-3.1.2 – Memory leak with multiple Collectors [CRIBL‑6310]

Problem: Configuring multiple Collectors can lead to gradual but cumulative memory leaks. Due to a caching error, the memory can be recovered only by restarting LogStream.

Workaround: Restart LogStream on the affected Worker Nodes.

Fix: In LogStream 3.1.3.

2021-09-21 – v.3.1.1 – Azure Blob Storage Source/Destination authentication error on upgrade from v.3.0.4 [CRIBL‑6236]

Problem: Upon upgrading the Azure Blob Source and/or Destination from v.3.0.4 to v.3.1.1, you might receive an error of the form: Unable to extract accountName with provided information.

Workaround: Change the key, and then reset it back to your desired connection string.

Fix: Not observed as of Cribl LogStream 3.1.1.

2021-09-15 – v.3.0.0–3.1.3 – High CPU usage with Google Cloud Pub/Sub Source [CRIBL‑6182]

Problem: The Google Cloud Pub/Sub Source substantially increases CPU usage, which stays high even after data stops flowing. This causes throughput degradation, and more-frequent failed to acknowledge errors.

Workaround: Configure the Google Cloud Pub/Sub Source’s Advanced Settings > Max backlog to 1000.

Fix: In 3.2.0, which defaults the Max backlog to 1000, and also relaxes the retry interval from 10 seconds to 30 seconds.

2021-09-14 – v.3.1.1 – Modifying Collector > Preview > Capture settings can break Capture elsewhere [CRIBL‑6166]

Problem: Modifying the capture settings in a Collector’s Run > Preview > Capture modal can improperly modify the Filter expression in Capture modals for other Collectors, Sources, and Routes/Pipelines.

Fix: In LogStream 3.1.2.

2021-09-10 – v.3.1.1 – Worker Group certificate name drop-down shows certificates from the Leader [CRIBL‑6142]

Problem: When configuring certificates at Groups > <group‑name> > Settings > API Server Settings > TLS, certificates configured on the Leader incorrectly appear on the Certificate name drop-down.

Fix: In LogStream 3.1.2.

2021-09-10 – v.3.1.1 – Git Collapsed Actions broken in 3.1.1 [CRIBL‑6134]

Problem: With Collapsed Actions enabled, clicking the Commit & Push button has no effect. (The Commit & Deploy button works properly.)

Workaround: Disable Collapsed Actions, to restore separate Commit and Git Push buttons.

Fix: In LogStream 3.1.2.

2021-09-08 – All versions through v.3.1.1 – UDP support is currently IPv4-only [CRIBL‑6106, CRIBL‑6115]

Problem: Where Sources and Destinations connect over UDP, they currently support IPv4 only, not IPv6. This applies to Syslog, Metrics, and SNMP Trap Sources; and to Syslog, SNMP Trap, StatsD, StatsD Extended, and Graphite Destinations.

Workaround: Integrate via IPv4 if possible.

Fix: UDP Sources and Destinations gained IPv6 support in LogStream 3.1.2.

2021-09-02 – v.3.1.0-3.1.1 – Google Pub/Sub authentication via proxy environment variable fails [CRIBL‑6086]

Problem: When LogStream’s Google Cloud Pub/Sub Source and Destination attempt authentication through a proxy, using the https_proxy environment variable, they send an HTTP request to http://www.googleapis.com:443/oauth2/v4/token. This request fails with 504/502 errors. The root cause is a mismatch in dependency libraries, whose correction has been identified, but requires broader testing.

Workaround: Configure the proxy in transparent mode, to avoid relying on environment variables.

Fix: In 3.1.2.

2021-08-24 – v.3.1.0 – High CPU load with LogStream version 3.1.0 [CRIBL‑6039, CRIBL‑6044]

Problem: LogStream 3.1.0 added code-execution safeguards that inadvertently increased CPU load, and decreased throughput, with several Functions and most expressions.

Workaround: Downgrade to version 3.0.4.

Fix: Upgrade to version 3.1.1 or later.

2021-08-18 – v.3.1.0 – Clone Collector option broken [CRIBL‑5977]

Problem: Clicking a 3.1.0 Collector modal’s Clone Collector button simply closes the modal. (If you have unsaved changes, you’ll first be challenged to confirm closing the parent modal – but the expected cloned modal won’t open.)

Workaround: Click + Add New to re-create your original Collector’s config from scratch, adding any desired modifications.

Fix: In LogStream 3.1.1.

2021-08-18 – All versions through 3.1.0 – Tabbed code blocks broken on in-app docs [CRIBL‑5972]

Problem: When docs with tabbed code blocks are opened in the Help drawer, the default (leftmost) tab seizes focus. Other tabs will not display when clicked.

Workaround: Click the blue/linked page title atop the Help drawer to open the same page on docs.cribl.io, where all tabs can be selected.

Fix: In LogStream 3.1.1.

2021-08-14 – v.3.1.0 – Splunk Load Balanced Destination does not migrate auth type [CRIBL‑5940]

Problem: In a Splunk Load Balanced Destination with Indexer discovery enabled and a corresponding Auth token defined, upgrading to LogStream 3.1.0 corrupts the Auth token field’s value.

Workaround: Set the Authentication method to Manual and resave the token’s value.

Fix: In 3.1.1.

2021-08-11 – v.3.1.0 – `C.Secret()` values are undefined in Collectors [CRIBL‑5926]

Problem: Calling the C.Secret() internal method within a Collector field resolves incorrectly to an undefined substring. E.g., in URL fields, C.Secret() values will resolve to /undefined/ path substrings.

Workarounds: 1. Use C.vars and a Global Variable, instead of using this method. 2. Root cause is that C.Secret() in Collectors and Pipeline Functions has access only to secrets that were created before the last restart. Therefore, restart Worker Processes to refresh the method’s access.

Fix: In 3.1.1.

2021-08-10 – v.3.1.0 – Pre-processing Pipelines break Flows display [CRIBL‑5909]

Problem: Attaching a pre-processing Pipeline to a Source breaks the Monitoring > Flows (beta) page’s display. Attempting to remove Sources/Destinations from that page’s selectors throws a cryptic Sankey error.

Workaround: Temporarily detach pre-processing Pipelines if you want to check Flows.

Fix: Upgrade to version 3.1.1 or later.

2021-07-29 – v.3.0.2-3.0.4 – Upgrades via UI require broader permissions [CRIBL‑5774]

Problem: Upgrading from v.3.0.x via the UI requires the cribl user to be granted write permission on the parent directory above $CRIBL_HOME. The symptom is an error message of the form: Upgrade failed: EACCES: permission denied, mkdir '/opt/unpack.xxxxxxx.tmp'.

Workaround: Either adjust permissions, or upgrade via the filesystem. For complete instructions, see Upgrading.

Fix: Does not affect LogStream 3.1 or higher.

2021-07-26 – v.3.0.x–3.1.0 – Packs with orphaned lookups block access to Worker Groups [CRIBL‑5738]

Problem: If a Pack references a lookup file that’s missing from the Pack, pushing the Pack to a Worker Group will block access to the Group’s UI. You will see an error message of the form: “The Config Helper service is not available because a configuration file doesn’t exist… Please fix it and restart LogStream.”

Workaround: On the Leader Node, review the config helper logs ($CRIBL_HOME/log/groups/<group>/*.log) to see which references are broken. (In a single-instance deployment, see $CRIBL_HOME/log/*.log.) Then manually resolve these references in the Pack’s configuration.

Fix: Upgrade to version 3.1.1 or later.

2021-07-20 – v.3.0.3-3.4.0 – Can’t add Functions to a Pipeline named `config` [CRIBL‑5706]

Problem: You cannot add Functions to a Pipeline if the Pipeline is named config, because this name conflicts with the reserved route for the Create Pipeline dialog.

Workarounds: Don’tcha name your Pipelines config.

Fix: In Cribl Edge 3.4.1.

2021-07-06 – All versions through Current – Duplicate Workers/Worker GUIDs [CRIBL‑5611]

Problem: Multiple Workers have identical GUIDs. This creates problems in Monitoring, upgrading and versioning, etc., because all Workers show up as one.

Cause: This is caused by configuring one Worker and then copying its cribl/ directory to other Workers, to quickly bootstrap a deployment.

Workaround: Don’t do this! Instead, use the Bootstrap Workers from Leader endpoint.

Fix: Version TBD.

2021-07-02 – v.3.0.2–3.0.3 – Sample file’s last line not displayed upon upload [CRIBL‑5595]

Problem: When uploading (attaching) a sample data file, the file’s final line is not displayed in the Add Sample Data modal.

Workarounds: This is a UI bug only. LogStream correctly processes the complete sample data, which should show up when viewing the sample afterwards (e.g., within a Pipeline’s preview pane).

Fix: In LogStream 3.1.0.

2021-07-02 – All versions through 3.0.x – Date fields misleadingly preview with string symbol [CRIBL‑5594]

Problem: In Preview or Capture, incoming events like _raw will be displayed in the right pane with an α symbol that indicates string data. However, calling new Date() and then C.Time.strptime() methods in an Eval Function will return null on the OUT tab.

Cause: Due to the nature of JSON serialization, the incoming event’s Date field is misleadingly subsumed under the event’s α string symbol. It’s actually a structured type, not a string…yet.

Workaround: If you see unexpected null results, stringify the datetime field as you extract it, e.g.: new Date().toISOString(). Feeding the resulting field to Time methods should return datetime strings as expected.

Fix: In LogStream 3.1.0.

2021-06-22 – v.3.0.0–3.0.3 – Internal `C.Text.relativeEntropy()` method – broken typeahead and preview [CRIBL‑5534]

Problem: The C.Text.relativeEntropy() internal method is missing from JavaScript expressions’ typeahead drop-downs. You can manually type or paste in the method, and save your Function and Pipeline, but LogStream’s right Preview pane will (misleadingly) always show the method returning 0.

Workarounds: Use other means (such as the Live button) to preview and verify that the method is (in fact) returning valid results.

Fix: In LogStream 3.1.0.

2021-05-20 – v.3.0.0 – Multiple Functions Break LogStream 3.0 Pipelines [CRIBL‑5311, CRIBL‑5766]

Problem: After upgrade to LogStream 3.0.0, including any of the following Functions in a Pipeline can break the Pipeline: GeoIP, Redis, DNS Lookup, Reverse DNS, Tee. Symptom is an error of the form: Pipeline process timeout has occurred. Less seriously, including these Functions in a Pipeline can suppress Preview’s display of fields/values.

Workarounds: If you use these Functions in your Pipelines, stay with (or restore) a pre-3.0 version until LogStream 3.0.1 is available.

Fix: CRIBL-5311 fixed in LogStream 3.0.1. CRIBL-5766 fixed in LogStream 3.2.1.

2021-05-19 – v.3.0.0 – Leader’s Changes fly-out stays open after Commit

Problem: In the Leader’s left nav, the Changes fly-out remains stuck open after you commit pending changes.

Workarounds: Hover or click away. Then hover or click back to reopen the fly-out.

Fix: In LogStream 3.0.1.

2021-05-18 – v.3.0.0 – Packs > Export in “Merge” mode omits schemas and custom Functions

Problem: Exporting a Pack with the export mode set to Merge omits schemas and custom Functions configured within the Pack’s Knowledge > Schemas.

Workarounds: 1. Change the export mode to Merge safe, and export again. 2. If that doesn’t preserve the schema and Functions, revert to Merge export mode; install the resulting Pack onto its target(s); and then manually copy/paste the schema(s) and Functions from the source Pack’s UI to the target Pack’s UI.

Fix: In LogStream 3.0.1.

2021-05-17 – v.3.0.0–3.2.0 – Can’t Enable KMS on Worker Group after installing license

Problem: Enabling HashiCorp Vault or AWS KMS on a Worker Group, after installing a LogStream license on the same Group, fails with a spurious External KMS is prohibited by the current license error message.

Workaround: On the Leader, navigate to Settings > Worker Processes. Restart the affected Worker Group’s CONFIG_HELPER process. Then return to that Worker Group’s Security > KMS Settings, re-enter the same KMS configuration, and save.

2021-05-10 – v.2.4.5-3.0.3 – Elasticsearch Destination, with Auto version discovery, doesn’t send Authorization header

Problem: When the Elasticsearch Destination has Basic Authentication enabled, and its Elastic version field specifies Auto version discovery, LogStream fails to send the configured username and password credentials along with its API initial request. Elasticsearch responds with an HTTP 401 error.

Workaround: Explicitly set the Elastic version to either 7.x or 6.x (depending on your Elasticsearch cluster’s version); then stop and restart LogStream to pick up this configuration change.

Fix: In LogStream 3.1.0.

2021-05-04 – v.2.4.5-3.0.1 – Office 365 Message Trace Source skips events

Problem: The Event Breaker Rule provided for the Office 365 Message Trace Source mistakenly presets the Default timezone to ETC/GMT‑0. This setting causes LogStream to discover events but not collect them.

Workaround: Reset the Rule’s Default timezone to UTC, then click OK and resave the Ruleset.

Fix: In 3.0.2.

2021-05-03 – v.2.4.4–3.0.1 – Rollup Function suppresses `sourcetype` metrics

Problem: sourcetype metrics can be suppressed when the Cribl Internal > CriblMetrics Source is enabled and the cribl_metrics_rollup pre-processing Pipeline is attached to a Source.

Workarounds: Disabling the pre-processing pipeline restores sourcetype and any other missing data. However, without the rollup, a much higher data volume will be sent to the indexing tier.

Fix: In LogStream 3.0.2.

2021-04-20 – v.2.4.3–2.4.5 – Orphaned S3 staging directories

Problem: Using the S3 Destination, defining a partitioning expression with high cardinality can proliferate a large number (up to millions) of empty directories. This is because LogStream cleans up staged files, but not staging directories.

Workaround: Programmatically or manually delete stale staging directories (e.g., those older than 30 days).

Fix: In LogStream 3.0.2.

2021-04-12 – v.2.4.4-3.0.0 – Splunk Sources do not support multiple-metric events

Problem: LogStream’s Splunk Sources do not support multiple-measurement metric data points. (LogStream’s Splunk Load Balanced Destination does.)

Fix: In LogStream 3.0.1.

2021-04-07 – v.2.4.2–2.4.5 – Google Cloud Storage Destination fails to upload files > 5 MB

Problem: The Google Cloud Storage Destination might fail to put objects into GCS buckets. This happens with files larger than 5 MB, and causes the Google Cloud API to report a vague Invalid argument error.

Workaround: Set the Max file size (MB) to 5 MB. Also, reduce the Max file open time (sec) limit from its default 300 (5 minutes) to a shorter interval, to prevent files from growing to the 5 MB threshold. (Tune this limit based on your observed rate of traffic flow through the Destination.)

Fix: In LogStream 3.0.0.

Problem: The Log in with local user option is displayed to users even when you have disabled Settings > Authentication > Allow local auth for an OpenID Connect identity provider.

Workaround: Advise users to ignore this button. Although visible, it will not function.

Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.0–2.4.4 – Splunk TCP and LB Destinations’ Workers trigger OOM errors and restart

Problem: With a Splunk TCP or Splunk Load Balanced Destination created after upgrading to LogStream 2.4.x, Workers’ memory consumption may grow without bound, leading to out-of-memory errors. The API Process will restart the Workers, but there might be temporary outages.

Workaround: Toggle the Destination’s Advanced Settings > Minimize in‑flight data loss slider to No. This will preserve Processes killed by OOM conditions.

Fix: In LogStream 2.4.5.

2021-03-31 – v.2.4.4-2.4.5 – OpenID Connect authentication always shows local-auth fallback

Problem: Even if OpenID Connect external authentication is configured to disable Allow local auth, LogStream’s login page displays a Log in with local user button.

Workaround: Do not click that button.

Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.4 – Authentication options mistakenly display Cribl Cloud

Problem: The Settings > Authentication > Type drop-down offers a Cribl Cloud option, which is not currently functional. Attempting to configure and save this option could lock the admin user out of LogStream.

Workaround: Do not select, configure, or save that option.

Fix: In LogStream 2.4.5.

2021-03-30 – v.2.4.4 – Can’t disable some Sources from within their config modals

Problem: In configuration modals for the Azure Blob Storage and Office 365 Message Trace Sources, the Enabled slider cannot be toggled off, and its tooltip doesn’t appear.

Workaround: Disable your configured Source (where required) from the Manage Blob Storage Sources or the Manage Message Trace Sources page.

Fix: In LogStream 2.4.5.

2021-03-29 – v.2.4.x – SpaceOut Destination is broken

Problem: Within the SpaceOut game, you cannot shoot, and your player is immortal.

Workaround: There are other video games. After we defeat COVID, you’ll even be able to buy a PS5.

Fix: Restored in LogStream 2.4.5.

2021-03-24 – v.2.4.x – Cribl App for Splunk blocks admin password changes, configuration changes, and Splunk-based authentication

Problem: Attempting to change the admin password via the UI triggers a 403/Forbidden message. You can reset the password by editing users.json, but can’t save configuration changes to Settings, Pipelines, etc., because RBAC Roles are not properly applied.

Workaround: Using a 2.3.x version of the App enables local authentication and enables changes to Cribl/LogStream passwords and configuration/settings.

Fix: In LogStream 2.4.4.

2021-03-22 – v.1.7.0-2.4.3 – Azure Event Hubs Destination: Compression must be manually disabled

Problem: LogStream’s Azure Event Hubs Destination provides a Compression option that defaults to Gzip. However, compressed Kafka messages are not yet supported on Azure Event Hubs.

Workaround: Manually reset Compression to None, then resave Azure Event Hubs Destinations.

Fix: In LogStream 2.4.4.

2021-03-17 – v.2.4.2-2.4.3 – Parser Function > List of Fields copy/paste fails

Problem: When copying/pasting List of Fields contents between Parser Functions via the Copy button, the paste operation inserts unintended metadata instead of the original field references.

Workaround: Manually re-enter the second Parser Function’s List of Fields.

Fix: In LogStream 2.4.4.

2021-03-13 – v.2.4.3 – UI can’t find valid TLS .key files, blocking Master restarts and Worker reconfiguration

Problem: After upgrading to v.2.4.3, the UI fails to recognize valid TLS .key files, displaying spurious error messages of the form: “File does not exist: $CRIBL_HOME/local/cribl/auth/certs/<keyname>key.” An affected Master will not restart. Affected Workers will restart, but will not apply changes made through the UI.

Workaround: Ideally, specify an absolute path to each key file, rather than relying on environment variables. If you’re locked out of the UI, you’ll need to manually edit the referenced paths within these configuration files in LogStream subdirectories: local/cribl/cribl.yml (General > API Server TLS settings) and/or local/_system/instance.yml (Distributed > TLS settings). Contact Cribl Support if you need assistance. A more drastic workaround is to disable TLS for the affected connections.

Fix: In LogStream 2.4.4.

2021-03-12 – v.2.4.2-2.4.5 – Redis Function with specific username can’t authenticate against Redis 6.x ACLs

Problem: The Redis Function, when used with a specific username and Redis 6.x’s Access Control List feature, fails due to authentication problems.

Workaround: In the Function’s Redis URL field, point to the Redis default account, either with a password (e.g., redis://default:Password1@192.168.1.20:6379) or with no password (redis://192.168.1.20:6379). Do not specify a user other than default.

Fix: In LogStream 3.0.

2021-03-09 – v.2.4.3 – Splunk Destinations’ in-app docs mismatch UI’s current field order

Problem: For the Splunk Single Instance and Splunk Load Balanced Destinations, the in-app documentation omits the UI’s Advanced Settings section. Some fields are documented out-of-sequence, or are omitted.

Workaround: Refer to the UI’s tooltips, to the corrected Splunk Single Instance and Splunk Load Balanced online docs, and/or to the corrected PDF.

Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – Enabling Git Collapse Actions breaks Commit & Deploy

Problem: After enabling Settings > Distributed Settings > Git Settings > General > Collapse Actions, selecting Commit & Deploy throws a 500 error.

Workaround: Disable the Collapse Actions setting, then commit and deploy separately.

Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – S3 Collector lacks options to reuse HTTP connections and allow-self signed certs

Problem: As of v.2.4.3, LogStream’s AWS-related Sources & Destinations provide options to reuse HTTP connections, and to establish TLS connections to servers with self-signed certificates. However, the S3 Collector does not yet provide these options.

Fix: In LogStream 2.4.4.

2021-03-04 – v.2.4.2 – Esc key closes both Event Breaker Ruleset modals

Problem: After adding a rule to a Knowledge > Event Breaker Ruleset, pressing Esc closes the parent Ruleset modal along with the child Rule modal.

Workaround: Close the Rule modal by clicking either its Cancel button or its close box.

Fix: In LogStream 2.4.3.

2021-03-04 – v.2.4.2 – Aggregations Function in post-processing Pipeline addresses wrong Destination

Problem: An Aggregations Function, when used in a post-processing Pipeline, sends data to LogStream’s Default Destination rather than to the Pipeline’s attached Destination.

Workaround: If applicable, use the Function in a processing or pre-processing Pipeline instead.

Fix: In LogStream 2.4.3.

2021-02-25 – v.2.4.2 – On Safari, Event Breaker shows no OUT events

Problem: When viewing an Event Breaker’s results on Safari, no events are displayed on the Preview pane’s OUT tab.

Workaround: Use another supported browser.

Fix: In LogStream 2.4.3.

2021-02-22 – v.2.4.3 – Collection jobs UI errors

Problem: Collection jobs are missing from the Monitoring > Sources page, even though they are returned by metric queries. Also, the Job Inspector > Live modal displays an empty, unintended Configure tab.

Workaround: Use the Job Inspector to access collection results. Ignore the Configure tab.

Fix: In LogStream 2.4.4.

2021-02-19 – v.2.4.2 – Upon upgrade, Git remote repo setting breaks, blocking Worker Groups

Problem: If a Git remote repo was previously configured, upgrading to LogStream v.2.4.2 throws errors of this form upon startup: Failed to initialize git repository. Config versioning will not be available...Invalid URL.... The Master cannot commit or deploy to any Worker Group.

Workarounds: 1. Downgrade back to v.2.4.1 (or your previous working version). 2. Switch from Basic authentication to SSH authentication against the repo, to remove the username from requests. (The apparent root cause is Basic/http auth using a valid URL and username, but missing a password.)

Fix: In LogStream 2.4.3.

2021-02-19 – v.2.4.0-2.4.2 – Splunk (S2S) Forwarder access control blocks upon upgrade to LogStream 2.4.x

Problem: If Splunk indexers have forwarder tokens enabled, and worked with LogStream 2.3.x before, upgrading to LogStream 2.4.x causes data to stop flowing.

Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.

Fix: In LogStream 2.4.3.

2021-02-10 – v.2.4.0-2.4.1 – With Splunk HEC Source, JSON payloads containing embedded objects trigger high CPU usage

Problem: Splunk HEC JSON payloads containing nested objects trigger high CPU usage, due to a flaw in JSON parsing.

Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.

Fix: In LogStream 2.4.2.

2021-01-30 – v.2.4.0 – Worker Nodes cannot connect to Master

Problem: Worker Nodes cannot connect to the Master after the Master is upgraded to v.2.4.0.

Workaround: Disable compression on the Workers. You can do so through the Workers’ UI at System Settings > Distributed Settings > Master Settings > Compression, or by commenting out this line in each Worker’s cribl.yml config file:

compression: gzip

Fix: In LogStream 2.4.1.

2021-01-25 – v.2.4.0 – S3 collection stops working due to auth secret key issues.

Problem: S3 collection stops after upgrade to 2.4.0 due to secret key re-encryption.

Workaround: Re-configure S3, save and re-deploy.

Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Google Cloud Storage Destination Needs Extra Endpoint to Initialize

Problem: The Google Cloud Storage Destination fails to initialize, displaying an error of the form: Bucket does not exist!

Workaround: In the outputs.yml file, under your cribl-gcp-bucket key endpoint, add: https://storage.googleapis.com. (in a single-instance deployment, locate this file at $CRIBL_HOME/local/cribl/outputs.yml. In a distributed deployment, locate it at $CRIBL_HOME/groups/<group name>/local/cribl/outputs.yml.)

Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Worker Groups’ Settings > Access Management Is Absent from UI

Problem: In this release, the Worker Groups > <group‑name> > System Settings UI did not display the expected Access Management, Authentication, and Local Users sections.

Workaround: Manually edit the users.json file.

Fix: In LogStream 2.4.1.

Problem: On the Routes page, selecting Capture New in the right pane does not copy custom Filter expressions to the resulting Capture Sample Data modal. That modal’s Filter Expression field always defaults to true.

Workarounds: 1. Bypass the Capture New button. Instead, from the Route’s own ••• (Options) menu, select Capture. This initiates a capture with the Filter Expression correctly populated. 2. Copy/paste the expression into the Capture Sample Data modal’s Filter Expression field. Or, if the expression is displayed in that field’s history drop-down, retrieve it.

Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Destinations’ Documentation Doesn’t Render from UI

Problem: Clicking the Help link in a Destination’s configuration modal displays the error message: “Unable to load docs. Please check LogStream’s online documentation instead.”

Workarounds: 1. Go directly to the online Destinations docs, starting here. 2. Follow the UI link to the docs landing page, click through to open or download the current PDF, and scroll to its Destinations section.

Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – `Esc` Key Doesn’t Consistently Close Modals

Problem: Pressing Esc with focus on a modal’s drop-down or slider doesn’t close the modal as expected. (Pressing Esc with focus on a free-text field, combo box, or nothing does close the modal – displaying a confirmation dialog first, if you have unsaved changes.)

Workarounds: Click the X close box at upper right, or click Cancel at lower right.

Fix: In LogStream 2.4.1.

2020-12-17 – v.2.3.0-2.4.0 – Free-License Expiration Notice, Blocked Inputs

Problem: LogStream reports an expired Free license, and blocks inputs, even though Free licenses in v.2.3.0 do not expire.

Workaround: This is caused by time-limited Free license key originally entered in a LogStream version prior to 2.3.0. Go to Settings > Licensing, click to select and expand your expired Free license, and click Delete license. LogStream will recognize the new, permanent Free license, and will restore throughput.

Fix: In LogStream 2.4.1.

2020-11-14 – v.2.3.3 – Null Fields Redacted in Preview, but Still Forwarded

Problem: Where event fields have null values, LogStream (by default) displays them as struck-out in the right Preview pane. The preview is misleading, because the events are still sent to the output.

Workaround: If you do want to prevent fields with null values from reaching the output, use an Eval Function, with an appropriate Filter expression, to remove them.

Fix: Preview corrected in LogStream 2.3.4.

2020-10-27 – v.2.3.2 – Cannot Name or Save New Event Breaker Rule

Problem: After clicking Add Rule in a new or existing Event Breaker Ruleset, the Event Breaker Rule modal’s Rule Name field is disabled. Because Rule Name is mandatory field, this also disables saving the Rule via the OK button.

Fix: In LogStream 2.3.3.

2020-10-12 – v.2.3.1 – Deleting One Function Deletes Others in Same Group

Problem: After inserting a new Function into a group and saving the Pipeline, deleting the Function also deletes other Functions lower down in the same group.

Fix: In LogStream 2.3.2.

Workaround: Move the target Function out of the group, resave the Pipeline, and only then delete the Function.

2020-09-27 – v.2.3.1 – Enabling Boot Start as Different User Fails

Problem: When a root user tries to enable boot-start as a different user (e.g., using /opt/cribl/bin/cribl boot-start enable -u <some‑username>), they receive an error of this form:

error: found user=0 as owner for path=/opt/cribl, expected uid=NaN. 
Please make sure CRIBL_HOME and its contents are owned by the uid=NaN by running: 
[sudo] chown -R NaN:[$group] /opt/cribl 

Fix: In LogStream 2.3.2.

Workaround: Install LogStream 2.2.3 (which you can download here), then upgrade to 2.3.1.

Problem: Upon upgrading an earlier, licensed LogStream installation to v.2.3.0, the Worker Groups tab might be absent from the Master Node’s top menu.

Fix: In LogStream 2.3.1.

Workaround: Click the Home > Worker Groups tile to access Worker Groups.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 on RHEL 6, RHEL 7

Problem: Upon upgrading to v.2.3.0, LogStream might fail to start on RHEL 6 or 7, with an error message of the following form. This occurs when the user running LogStream doesn’t match the LogStream binary’s owner. LogStream 2.3.0 applies a restrictive permissions check using id -un <uid>, which does not work with the version of id that ships with these RHEL releases.

id: 0: No such user
ERROR: Cannot run command because user=root with uid=0 does not own executable 

Fix: In LogStream 2.3.1.

Workaround: Update your RHEL environment’s id version, if possible.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 with OpenId Connect

Problem: Upon upgrading an earlier LogStream installation to v.2.3.0, OIDC users might be unable to restart the LogStream server.

Fix: In LogStream 2.3.1.

Workaround: Edit $CRIBL_HOME/default/cribl/cribl.yml to add the following lines to its the auth section:

filter_type: email_whitelist
scope: openid profile email

2020-06-11 – v.2.1.x – Can’t switch from Worker to Master Mode

Problem: In a Distributed deployment, attempting to switch Distributed Settings from Worker to Master Mode blocks with a spurious “Git not available…Please install and try again” error message.

Fix: In LogStream 2.3.0.

Workaround: To initialize git, switch first from Worker to Single mode, and then from Single to Master mode.

Problem: Entering valid credentials on the login page (e.g., http://localhost:9000/login) yields only a spinner.

Fix: In LogStream 2.3.0.

Workaround: Trim /login from the URL.

2020-02-22 – v.2.1.x – Deleting resources in `default/`

Problem: In a Distributed deployment, deleting resources in default/ causes them to reappear on restart.

Workaround/Fix: In progress.

2019-10-22 – v.2.0.0 – In-product upgrade issue on v2.0

Problem: Using in-product upgrade feature in v.1.7 (or earlier) fails to upgrade to v.2.0, due to package-name convention change.

Workaround/Fix: Download the new version and upgrade per steps laid out here.

2019-08-27 – v.1.7 – In-product upgrade issue on v.1.7

Problem: Using in-product upgrade feature in v.1.6 (or earlier) fails to upgrade to v.1.7 due to package name convention change.

Workaround/Fix: Download the new package and upgrade per steps laid out here.

2019-03-21 – v.1.4 – S3 stagePath issue on upgrade to v.1.4+

Problem: When upgrading from v.1.2 with a S3 output configured, stagePath was allowed to be undefined. In v.1.4+, stagePath is a required field. This might causing schema violations when upgrading older configs.

Workaround/Fix: Reconfigure the output with a valid stagePath filesystem path.

Known Issues

2023-11-09 - All Versions through Current - High-volume UDP data may be dropped in Cribl.Cloud​

2023-07-13 – All Versions through Current – When configuring an S3 Collector, JavaScript expressions break the “Auto-populate from” option for the Path field [CRIBL-18844]​

2023-07-07 – v.4.1.3 – When configuring an HTTP Source, enabling Source PQ changes the inputId in events [CRIBL-18668]​

2023-06-29 – All Versions through Current – (Linux) System Metrics Source does not emit “Per interface” metrics in “All” mode [CRIBL-18473]​

2023-06-26 – All Versions through Current – High CPU usage in File Monitor Source’s Manual mode [CRIBL-18400]​

2023-06-26 – All Versions through Current – Fleet Upgrade Errors [CRIBL-18374]​

2023-06-21 – v.3.5.2–v.4.1.3 – Using the Rename Function gives unexpected results due to skipped internal fields [CRIBL-18285]​

2023-06-15 – v.4.1.3 – Amazon CloudWatch Destinations log error while flushing events older than 24 hours [CRIBL-18184]​

2023-06-14 – v.4.1.2–4.1.3 – The Office 365 Activity Source misses events [CRIBL-18164]​

2023-06-08 – v.4.1.2 – AES-256-GCM security vulnerability [CRIBL-18038]​

2023-06-01 – v.4.0.0–4.1.3 – GroupEdit is not able to commit changes [CRIBL-17939]​

2023-06-01 – v.4.1.0–4.1.2 – Kubernetes Logs Event Breaker Incorrectly Breaks Events [CRIBL-17907]​

2023-05-27 – v.4.1.1–4.1.2 – Perf bug: RPC consuming excessive CPU compiling expressions [CRIBL-17779]​

2023-05-26 – v.4.1.2–4.1.3 – Edge Nodes don’t update from Cribl Edge Leader [CRIBL-17792]​

2023-05-19 – v.4.1.2 – CrowdStrike Destination’s “LogScale endpoint” field hidden from UI [CRIBL‑17715]​

2023-05-16 – v.4.1.1–4.1.2 – Expanding Status of Output Router Destination triggers error [CRIBL-17632]​

2023-05-12 – v.4.0–4.1.3 – Windows Event Forwarder Source causes duplicate events when experiencing backpressure [CRIBL-17614]​

2023-05-12 – v.4.0–4.1.2 – The Kubernetes Logs Source drops events [CRIBL-17602]​

2023-05-05 – v.4.1.1 – S3 ingestion slows after upgrade [CRIBL-17317]​

2023-04-27 – v.4.1.1 – Edge Nodes integrated with systemd/initd can fail upon reconfig [CRIBL‑17264]​

2023-04-14 – v.2.2–4.1.x – Requests from Office 365 Message Trace Source failed intermittently [CRIBL-16929]​

2023-04-13 – v.4.1.1 – Monitoring incorrectly shows metrics for disconnected Subscriptions [CRIBL‑16926]​

2023-04-03 – v.4.1.0 – Windows Event Logs Source should have a configurable Max Event Size [CRIBL-16549]​

2023-03-31 – v.4.1.0-4.1.1 – Cribl.Cloud API Credential’s Roles aren’t honored on tokens [SAAS‑3681]​

2023-03-30 v.4.1.0 – Splunk Load Balanced Destination incorrectly re‑creates all connections during DNS resolution [CRIBL‑16494]​

2023-03-28 – v.4.1.0 – Add Kubernetes and Docker modals in the UI aren’t checking the CRIBL_BOOTSTRAP_HOST environment variable [CRIBL-16432]​

2023-03-23 – v.4.1.0 – Leader’s config deployments to pre-4.1 Workers silently fail [CRIBL‑16339]​

2023-03-22 – v.4.1.0-4.1.1 – Cribl Stream Database Collector cannot connect with SQL Server using AD authentication [CRIBL-16304]​

2023-03-21 – v.4.1.0 – Unable to bind CRIBL_DIST_LEADER_URL to an IPv6 address [CRIBL-16284]​

2023-03-20 – v.3.5.0–4.1.0 – System Activity drawer on Windows Edge Nodes displays no data [CRIBL-16194]​

2023-03-20 – v.4.1.0-4.2.1 – If a deployment fails, Workers will not automatically revert to the previous version [CRIBL‑16203]​

2023-03-15 – v.4.1.0 – QuickConnect Pipeline Settings button fails [CRIBL‑16142]​

2023-03-14 v.4.0.0–4.1.0 – Duplicate data with Event Hubs [CRIBL‑16102]​

2023-03-01 – v.4.0.4 – Kafka-based Sources are omitting the _time field [CRIBL‑15696]​

2023-02-24 – v.4.1.0-4.2.1 – The Chain Function has a 10% impact on performance [CRIBL-15538]​

2023-02-23 – v.4.0.x – DNS Resolution reconnects Cribl TCP connections every time [CRIBL‑15363]​

2023-02-23 – v.4.0.4–4.1.2 – Enable Automatic Upgrades deletes remote repo’s Git Settings [CRIBL‑15502]​

2023-02-22 – v.4.0.4–4.1.0 – Sending data to an inactive Kafka, Confluent Cloud, or Azure Event Hubs Destination triggers an endless series of log errors [CRIBL‑15455]​

2023-02-20 – v.4.0.x–4.1.0 – Logging-level changes do not take effect for services [CRIBL‑15365]​

2023-02-20 – v.4.0.3–4.1.0 – On-prem Edge Nodes automatically upgrade, ignoring Leader settings [CRIBL‑15367]​

2023-02-14 – v.4.0.0–4.0.4 – Pipeline is deselected in QuickConnect after modification [CRIBL-15246]​

2023-02-14 – v.3.5.0 through Current – Cannot clear Messages drawer while in GitOps Push mode [CRIBL‑15239]​

2023-02-06 – v.4.0.x–4.1.0 – Event Breaker timestamp extraction fails after configured time [CRIBL‑15107]​

2023-02-05 – v.3.0.3–4.0.4 – CRIBL_DIST_WORKER_PROXY env var is ignored when Leader Node’s Master URL is set via instance.yml [CRIBL-15540]​

2023-02-03 – v.4.0.0–4.0.4 – Metrics blocklist can’t be changed on a global level [CRIBL-15081]​

2023-01-26 – v.3.5.0–4.0.3 – Preview Full > Send Out does not capture events [CRIBL-14914]​

2023-01-25 – v.3.5.x through Current – Cascading problems when Windows Event Logs Source collects from ForwardedEvents channel [CRIBL‑14869]​

2023-01-17 – v.4.0.x–4.0.4 – GitOps Push/read-only confirmation banner has dead link [CRIBL‑14681]​

2023-01-12 – v.3.5.x through Current – Recently added Worker Nodes fail to appear on the Monitoring page [CRIBL‑14627]​

2023-01-11 – Multiple versions through 4.1.x – Kafka-based Sources’ rebalancing is logged with exaggerated severity [CRIBL-14609]​

2023-01-10 – v.3.5.x through Current – Configuration updates not shared between primary and standby Leaders [CRIBL‑12814, CRIBL‑14556]​

2023-01-02 – v.4.0.0–4.0.2 – Google Cloud Chronicle Destination drops changes in distributed deployments with custom log types [CRIBL‑14453]​

2022-12-15 – v.3.5.4–4.0.3 – Changes to a Lookup table on the Leader don’t always propagate to Edge Nodes [CRIBL-14299]​

2022-12-13 – v.4.0.0 through Current – Default commit message missing for non-admin users [CRIBL‑14239]​

2022-12-09 – v.4.0.1–4.0.2 – Users assigned the owner_all role cannot perform commits [CRIBL-14180]​

2022-12-08 – v.4.0.1–4.0.2 – Landing page not displaying system metrics from Edge Nodes when teleporting from the Leader [CRIBL-14169]​

2022-12-07 – v.4.0.1 – Google Cloud Pub/Sub Source and Destination can break on field validation [CRIBL‑14160]​

2022-12-01 – v.4.0.2 – CrowdStrike FDR Source needs 6-hour visibility timeout [CRIBL‑14067]​

2022-12-01 – v.4.0.0 – Amazon S3 Source stops receiving data after upgrade [CRIBL‑14093]​

2022-11-30 – v.4.0.2 – QuickConnect-configured Sources’ misleading “Enabled” status while disconnected [CRIBL-14041]​

2022-11-28 v.4.0–4.0.2 – “Unknown config version” error can appear when deploying changes [CRIBL‑13910]​

2022-11-28 – v.4.1.0 through Current – Diag bundles might fail to download from teleported Edge Nodes [CRIBL-13999]​

2022-11-23 – All versions through 4.1.0 – Edge Nodes show incorrect configurations after upgrade or commit/deploy from Leader [CRIBL‑13863, CRIBL‑15507]​

2022-11-23 – v.4.0.0 – Teleported Edge Nodes don’t display Distributed Settings or Diagnostics Settings [CRIBL‑13868]​

2022-11-22 – v.3.2.0–4.0.0 – Chain Function creates extra Pipelines and Functions during initialization [CRIBL‑13860]​

2022-11-21 – v.4.0.0 – Metrics service unavailable [CRIBL‑13826]​

2022-11-18 – v.4.0.0–4.0.3 – License usage visible only for current day [CRIBL‑13811]​

2022-11-16 – versions 3.4.0–4.0.4 – Source PQ in Smart mode can trigger excessive memory usage and OOM failures [CRIBL‑13761]​

2022-11-14 – version 4.0.0 – Forbidden error banners mistakenly displayed to non-admin users [CRIBL‑13681]​

2022-11-13 – v.4.0.0–4.0.2 – Edge/Kubernetes Logs Source repeats logs due to incorrect timestamp [CRIBL-13900]​

2022-11-08 – v.4.0.0 – Cribl Edge honeycomb display doesn’t render null values [CRIBL-13595]​

2022-11-08 – v.4.0.0 – Broken in-app doc links [DOC‑100]​

2022-11-07 – v.4.0.0 – Spurious “Secret decrypt failed with error" log messages appear after upgrade [CRIBL‑13554]​

2022-11-07 – v.4.0.0 – Upgrade status shown on Group Upgrade page doesn’t match Global Settings [CRIBL-13566]​

2022-11-04 – v.3.5.1–4.0.4 – Avg Thruput (Bytes Per Second) not displayed for internal logs or metrics [CRIBL‑13530]​

2022-11-03 – v.4.0.0 – Sorting by column on Manage > Edge Node page results in blank list [CRIBL‑13505]​

2022-11-01 – All versions through Current – Source PQ in Smart mode could cause premature backpressure [CRIBL-13414]​

2022-11-01 – v.4.0.0 through Current – Default AppScope Config file can’t be restored when in use [CRIBL‑13400]​

2022–10-20 – v.4.0.0 through Current – Pipelines with Clone and other Functions can show inconsistent total processing times [CRIBL-13125]​

2023-11-09 - All Versions through Current - High-volume UDP data may be dropped in Cribl.Cloud

2023-07-13 – All Versions through Current – When configuring an S3 Collector, JavaScript expressions break the “Auto-populate from” option for the Path field [CRIBL-18844]

2023-07-07 – v.4.1.3 – When configuring an HTTP Source, enabling Source PQ changes the inputId in events [CRIBL-18668]

2023-06-29 – All Versions through Current – (Linux) System Metrics Source does not emit “Per interface” metrics in “All” mode [CRIBL-18473]

2023-06-26 – All Versions through Current – High CPU usage in File Monitor Source’s Manual mode [CRIBL-18400]

2023-06-26 – All Versions through Current – Fleet Upgrade Errors [CRIBL-18374]

2023-06-21 – v.3.5.2–v.4.1.3 – Using the Rename Function gives unexpected results due to skipped internal fields [CRIBL-18285]

2023-06-15 – v.4.1.3 – Amazon CloudWatch Destinations log error while flushing events older than 24 hours [CRIBL-18184]

2023-06-14 – v.4.1.2–4.1.3 – The Office 365 Activity Source misses events [CRIBL-18164]

2023-06-08 – v.4.1.2 – AES-256-GCM security vulnerability [CRIBL-18038]

2023-06-01 – v.4.0.0–4.1.3 – GroupEdit is not able to commit changes [CRIBL-17939]

2023-06-01 – v.4.1.0–4.1.2 – Kubernetes Logs Event Breaker Incorrectly Breaks Events [CRIBL-17907]

2023-05-27 – v.4.1.1–4.1.2 – Perf bug: RPC consuming excessive CPU compiling expressions [CRIBL-17779]

2023-05-26 – v.4.1.2–4.1.3 – Edge Nodes don’t update from Cribl Edge Leader [CRIBL-17792]

2023-05-19 – v.4.1.2 – CrowdStrike Destination’s “LogScale endpoint” field hidden from UI [CRIBL‑17715]

2023-05-16 – v.4.1.1–4.1.2 – Expanding Status of Output Router Destination triggers error [CRIBL-17632]

2023-05-12 – v.4.0–4.1.3 – Windows Event Forwarder Source causes duplicate events when experiencing backpressure [CRIBL-17614]

2023-05-12 – v.4.0–4.1.2 – The Kubernetes Logs Source drops events [CRIBL-17602]

2023-05-05 – v.4.1.1 – S3 ingestion slows after upgrade [CRIBL-17317]

2023-04-27 – v.4.1.1 – Edge Nodes integrated with `systemd/initd` can fail upon reconfig [CRIBL‑17264]

2023-04-14 – v.2.2–4.1.x – Requests from Office 365 Message Trace Source failed intermittently [CRIBL-16929]

2023-04-13 – v.4.1.1 – Monitoring incorrectly shows metrics for disconnected Subscriptions [CRIBL‑16926]

2023-04-03 – v.4.1.0 – Windows Event Logs Source should have a configurable Max Event Size [CRIBL-16549]

2023-03-31 – v.4.1.0-4.1.1 – Cribl.Cloud API Credential’s Roles aren’t honored on tokens [SAAS‑3681]

2023-03-30 v.4.1.0 – Splunk Load Balanced Destination incorrectly re‑creates all connections during DNS resolution [CRIBL‑16494]

2023-03-28 – v.4.1.0 – Add Kubernetes and Docker modals in the UI aren’t checking the `CRIBL_BOOTSTRAP_HOST` environment variable [CRIBL-16432]

2023-03-23 – v.4.1.0 – Leader’s config deployments to pre-4.1 Workers silently fail [CRIBL‑16339]

2023-03-22 – v.4.1.0-4.1.1 – Cribl Stream Database Collector cannot connect with SQL Server using AD authentication [CRIBL-16304]

2023-03-21 – v.4.1.0 – Unable to bind `CRIBL_DIST_LEADER_URL` to an IPv6 address [CRIBL-16284]

2023-03-20 – v.3.5.0–4.1.0 – System Activity drawer on Windows Edge Nodes displays no data [CRIBL-16194]

2023-03-20 – v.4.1.0-4.2.1 – If a deployment fails, Workers will not automatically revert to the previous version [CRIBL‑16203]

2023-03-15 – v.4.1.0 – QuickConnect Pipeline Settings button fails [CRIBL‑16142]

2023-03-14 v.4.0.0–4.1.0 – Duplicate data with Event Hubs [CRIBL‑16102]

2023-03-01 – v.4.0.4 – Kafka-based Sources are omitting the `_time` field [CRIBL‑15696]

2023-02-24 – v.4.1.0-4.2.1 – The Chain Function has a 10% impact on performance [CRIBL-15538]

2023-02-23 – v.4.0.x – DNS Resolution reconnects Cribl TCP connections every time [CRIBL‑15363]

2023-02-23 – v.4.0.4–4.1.2 – Enable Automatic Upgrades deletes remote repo’s Git Settings [CRIBL‑15502]

2023-02-22 – v.4.0.4–4.1.0 – Sending data to an inactive Kafka, Confluent Cloud, or Azure Event Hubs Destination triggers an endless series of log errors [CRIBL‑15455]

2023-02-20 – v.4.0.x–4.1.0 – Logging-level changes do not take effect for services [CRIBL‑15365]

2023-02-20 – v.4.0.3–4.1.0 – On-prem Edge Nodes automatically upgrade, ignoring Leader settings [CRIBL‑15367]

2023-02-14 – v.4.0.0–4.0.4 – Pipeline is deselected in QuickConnect after modification [CRIBL-15246]

2023-02-14 – v.3.5.0 through Current – Cannot clear Messages drawer while in GitOps Push mode [CRIBL‑15239]

2023-02-06 – v.4.0.x–4.1.0 – Event Breaker timestamp extraction fails after configured time [CRIBL‑15107]

2023-02-05 – v.3.0.3–4.0.4 – `CRIBL_DIST_WORKER_PROXY` env var is ignored when Leader Node’s Master URL is set via `instance.yml` [CRIBL-15540]

2023-02-03 – v.4.0.0–4.0.4 – Metrics blocklist can’t be changed on a global level [CRIBL-15081]

2023-01-26 – v.3.5.0–4.0.3 – Preview Full > Send Out does not capture events [CRIBL-14914]

2023-01-25 – v.3.5.x through Current – Cascading problems when Windows Event Logs Source collects from `ForwardedEvents` channel [CRIBL‑14869]

2023-01-17 – v.4.0.x–4.0.4 – GitOps Push/read-only confirmation banner has dead link [CRIBL‑14681]

2023-01-12 – v.3.5.x through Current – Recently added Worker Nodes fail to appear on the Monitoring page [CRIBL‑14627]

2023-01-11 – Multiple versions through 4.1.x – Kafka-based Sources’ rebalancing is logged with exaggerated severity [CRIBL-14609]

2023-01-10 – v.3.5.x through Current – Configuration updates not shared between primary and standby Leaders [CRIBL‑12814, CRIBL‑14556]

2023-01-02 – v.4.0.0–4.0.2 – Google Cloud Chronicle Destination drops changes in distributed deployments with custom log types [CRIBL‑14453]

2022-12-15 – v.3.5.4–4.0.3 – Changes to a Lookup table on the Leader don’t always propagate to Edge Nodes [CRIBL-14299]

2022-12-13 – v.4.0.0 through Current – Default commit message missing for non-admin users [CRIBL‑14239]

2022-12-09 – v.4.0.1–4.0.2 – Users assigned the `owner_all` role cannot perform commits [CRIBL-14180]

2022-12-08 – v.4.0.1–4.0.2 – Landing page not displaying system metrics from Edge Nodes when teleporting from the Leader [CRIBL-14169]

2022-12-07 – v.4.0.1 – Google Cloud Pub/Sub Source and Destination can break on field validation [CRIBL‑14160]

2022-12-01 – v.4.0.2 – CrowdStrike FDR Source needs 6-hour visibility timeout [CRIBL‑14067]

2022-12-01 – v.4.0.0 – Amazon S3 Source stops receiving data after upgrade [CRIBL‑14093]

2022-11-30 – v.4.0.2 – QuickConnect-configured Sources’ misleading “Enabled” status while disconnected [CRIBL-14041]

2022-11-28 v.4.0–4.0.2 – “Unknown config version” error can appear when deploying changes [CRIBL‑13910]

2022-11-28 – v.4.1.0 through Current – Diag bundles might fail to download from teleported Edge Nodes [CRIBL-13999]

2022-11-23 – All versions through 4.1.0 – Edge Nodes show incorrect configurations after upgrade or commit/deploy from Leader [CRIBL‑13863, CRIBL‑15507]

2022-11-23 – v.4.0.0 – Teleported Edge Nodes don’t display Distributed Settings or Diagnostics Settings [CRIBL‑13868]

2022-11-22 – v.3.2.0–4.0.0 – Chain Function creates extra Pipelines and Functions during initialization [CRIBL‑13860]

2022-11-21 – v.4.0.0 – Metrics service unavailable [CRIBL‑13826]

2022-11-18 – v.4.0.0–4.0.3 – License usage visible only for current day [CRIBL‑13811]

2022-11-16 – versions 3.4.0–4.0.4 – Source PQ in Smart mode can trigger excessive memory usage and OOM failures [CRIBL‑13761]

2022-11-14 – version 4.0.0 – `Forbidden` error banners mistakenly displayed to non-admin users [CRIBL‑13681]

2022-11-13 – v.4.0.0–4.0.2 – Edge/Kubernetes Logs Source repeats logs due to incorrect timestamp [CRIBL-13900]

2022-11-08 – v.4.0.0 – Cribl Edge honeycomb display doesn’t render null values [CRIBL-13595]

2022-11-08 – v.4.0.0 – Broken in-app doc links [DOC‑100]

2022-11-07 – v.4.0.0 – Spurious “Secret decrypt failed with error" log messages appear after upgrade [CRIBL‑13554]

2022-11-07 – v.4.0.0 – Upgrade status shown on Group Upgrade page doesn’t match Global Settings [CRIBL-13566]

2022-11-04 – v.3.5.1–4.0.4 – Avg Thruput (Bytes Per Second) not displayed for internal logs or metrics [CRIBL‑13530]

2022-11-03 – v.4.0.0 – Sorting by column on Manage > Edge Node page results in blank list [CRIBL‑13505]

2022-11-01 – All versions through Current – Source PQ in Smart mode could cause premature backpressure [CRIBL-13414]

2022-11-01 – v.4.0.0 through Current – Default AppScope Config file can’t be restored when in use [CRIBL‑13400]

2022–10-20 – v.4.0.0 through Current – Pipelines with Clone and other Functions can show inconsistent total processing times [CRIBL-13125]

2022-11-03 – v.4.0.0 – Git settings don’t immediately populate Commit modals [CRIBL‑13501]