2021-05-20 – 3.0.0 – Multiple Functions Break LogStream 3.0 Pipelines

Problem: After upgrade to LogStream 3.0.0, including any of the following Functions in a Pipeline can break the Pipeline: GeoIP, Redis, DNS Lookup, Reverse DNS, Tee. Symptom is an error of the form: Pipeline process timeout has occurred. Less seriously, including these Functions in a Pipeline can suppress Preview's display of fields/values.
Workarounds: If you use these Functions in your Pipelines, stay with (or restore) a pre-3.0 version until LogStream 3.0.1 is available.
Fix: In LogStream 3.0.1.

2021-05-19 – 3.0.0 – Leader's Changes fly-out stays open after Commit

Problem: In the Leader's left nav, the Changes fly-out remains stuck open after you commit pending changes.
Workarounds: Hover or click away. Then hover or click back to reopen the fly-out.
Fix: In LogStream 3.0.1.

2021-05-18 – 3.0.0 – Packs > Export in "Merge" mode omits schemas and custom Functions

Problem: Exporting a Pack with the export mode set to Merge omits schemas and custom Functions configured within the Pack's Knowledge > Schemas.
Workarounds: 1. Change the export mode to Merge safe, and export again. 2. If that doesn't preserve the schema and Functions, revert to Merge export mode; install the resulting Pack onto its target(s); and then manually copy/paste the schema(s) and Functions from the source Pack's UI to the target Pack's UI.
Fix: In LogStream 3.0.1.

2021-05-10 – 2.4.5 – Elasticsearch Destination, with Auto version discovery, doesn't send Authorization header

Problem: When the Elasticsearch Destination has Basic Authentication enabled, and its Elastic version field specifies Auto version discovery, LogStream fails to send the configured username and password credentials along with its API initial request. Elasticsearch responds with an HTTP 401 error.
Workaround: Explicitly set the Elastic version to either 7.x or 6.x (depending on your Elasticsearch cluster's version); then stop and restart LogStream to pick up this configuration change.
Fix: Planned for LogStream 3.0.2.

2021-05-04 – 2.4.5 – Office 365 Message Trace Source skips events

Problem: The Event Breaker Rule provided for the Office 365 Message Trace Source mistakenly presets the Default timezone to ETC/GMT‑0. This setting causes LogStream to discover events but not collect them.
Workaround: Reset the Rule's Default timezone to UTC, then click OK and resave the Ruleset.
Fix: In 3.0.0.

2021-04-20 – v.2.4.3–2.4.5 – Orphaned S3 staging directories

Problem: Using the S3 Destination, defining a partitioning expression with high cardinality can proliferate a large number (up to millions) of empty directories. This is because LogStream cleans up staged files, but not staging directories.
Workaround: Programmatically or manually delete stale staging directories (e.g., those older than 30 days).
Fix: Planned for LogStream 3.1.0.

2021-04-12 – 2.4.4 – Splunk Sources do not support multiple-metric events

Problem: LogStream's Splunk Sources do not support multiple-measurement metric data points. (LogStream's Splunk Load Balanced Destination does.)
Fix: Planned for LogStream 3.0.1.

2021-04-07 – v.2.4.2–2.4.5 – Google Cloud Storage Destination fails to upload files > 5 MB

Problem: The Google Cloud Storage Destination might fail to put objects into GCS buckets. This happens with files larger than 5 MB, and causes the Google Cloud API to report a vague Invalid argument error.
Workaround: Set the Max file size (MB) to 5 MB. Also, reduce the Max file open time (sec) limit from its default 300 (5 minutes) to a shorter interval, to prevent files from growing to the 5 MB threshold. (Tune this limit based on your observed rate of traffic flow through the Destination.)
Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.4 – Local login option visible even when disabled

Problem: The Log in with local user option is displayed to users even when you have disabled Settings > Authentication > Allow local auth for an OpenID Connect identity provider.
Workaround: Advise users to ignore this button. Although visible, it will not function.
Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.0–2.4.4 – Splunk TCP and LB Destinations' Workers trigger OOM errors and restart

Problem: With a Splunk TCP or Splunk Load Balanced Destination created after upgrading to LogStream 2.4.x, Workers' memory consumption may grow without bound, leading to out-of-memory errors. The API Process will restart the Workers, but there might be temporary outages.
Workaround: Toggle the Destination's Advanced Settings > Minimize in‑flight data loss slider to No. This will preserve Processes killed by OOM conditions.
Fix: In LogStream 2.4.5.

2021-03-31 – v.2.4.4 – OpenID Connect authentication always shows local-auth fallback

Problem: Even if OpenID Connect external authentication is configured to disable Allow local auth, LogStream's login page displays a Log in with local user button.
Workaround: Do not click that button.
Fix: Planned for LogStream 3.0.0.

2021-03-31 – v.2.4.4 – Authentication options mistakenly display Cribl Cloud

Problem: The Settings > Authentication > Type drop-down offers a Cribl Cloud option, which is not currently functional. Attempting to configure and save this option could lock the admin user out of LogStream.
Workaround: Do not select, configure, or save that option.
Fix: In LogStream 2.4.5.

2021-03-30 – v.2.4.4 – Can't disable some Sources from within their config modals

Problem: In configuration modals for the Azure Blob Storage and Office 365 Message Trace Sources, the Enabled slider cannot be toggled off, and its tooltip doesn't appear.
Workaround: Disable your configured Source (where required) from the Manage Blob Storage Sources or the Manage Message Trace Sources page.
Fix: In LogStream 2.4.5.

2021-03-29 – v.2.4.x – SpaceOut Destination is broken

Problem: Within the SpaceOut game, you cannot shoot, and your player is immortal.
Workaround: There are other video games. After we defeat COVID, you'll even be able to buy a PS5.
Fix: Restored in LogStream 2.4.5.

2021-03-24 – v.2.4.x – Cribl App for Splunk blocks admin password changes, configuration changes, and Splunk-based authentication

Problem: Attempting to change the admin password via the UI triggers a 403/Forbidden message. You can reset the password by editing users.json, but can't save configuration changes to Settings, Pipelines, etc., because RBAC Roles are not properly applied.
Workaround: Using a 2.3.x version of the App enables local authentication and enables changes to Cribl/LogStream passwords and configuration/settings.
Fix: In LogStream 2.4.4.

2021-03-22 – v.1.7 through 2.4.3 – Azure Event Hubs Destination: Compression must be manually disabled

Problem: LogStream's Azure Event Hubs Destination provides a Compression option that defaults to Gzip. However, compressed Kafka messages are not yet supported on Azure Event Hubs.
Workaround: Manually reset Compression to None, then resave Azure Event Hubs Destinations.
Fix: In LogStream 2.4.4.

2021-03-17 – v.2.4.2, 2.4.3 – Parser Function > List of Fields copy/paste fails

Problem: When copying/pasting List of Fields contents between Parser Functions via the Copy button, the paste operation inserts unintended metadata instead of the original field references.
Workaround: Manually re-enter the second Parser Function's List of Fields.
Fix: In LogStream 2.4.4.

2021-03-13 – v.2.4.3 – UI can't find valid TLS .key files, blocking Master restarts and Worker reconfiguration

Problem: After upgrading to v.2.4.3, the UI fails to recognize valid TLS .key files, displaying spurious error messages of the form:
"File does not exist: $CRIBL_HOME/local/cribl/auth/certs/<keyname>key."
An affected Master will not restart. Affected Workers will restart, but will not apply changes made through the UI.
Workaround: Ideally, specify an absolute path to each key file, rather than relying on environment variables. If you're locked out of the UI, you'll need to manually edit the referenced paths within these configuration files in LogStream subdirectories: local/cribl/cribl.yml (General > API Server TLS settings) and/or local/_system/instance.yml (Distributed > TLS settings). Contact Cribl Support if you need assistance. A more drastic workaround is to disable TLS for the affected connections.
Fix: In LogStream 2.4.4.

2021-03-12 – v.2.4.2 – Redis Function with specific username can't authenticate against Redis 6.x ACLs

Problem: The Redis Function, when used with a specific username and Redis 6.x's Access Control List feature, fails due to authentication problems.
Workaround: In the Function's Redis URL field, point to the Redis default account, either with a password (e.g., redis://default:[email protected]:6379) or with no password (redis://192.168.1.20:6379). Do not specify a user other than default.
Fix: Planned for LogStream 3.0.1.

2021-03-09 – v.2.4.3 – Splunk Destinations' in-app docs mismatch UI's current field order

Problem: For the Splunk Single Instance and Splunk Load Balanced Destinations, the in-app documentation omits the UI's Advanced Settings section. Some fields are documented out-of-sequence, or are omitted.
Workaround: Refer to the UI's tooltips, to the corrected Splunk Single Instance and Splunk Load Balanced online docs, and/or to the corrected PDF.
Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – Enabling Git Collapse Actions breaks Commit & Deploy

Problem: After enabling Settings > Distributed Settings > Git Settings > General > Collapse Actions, selecting Commit & Deploy throws a 500 error.
Workaround: Disable the Collapse Actions setting, then commit and deploy separately.
Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – S3 Collector lacks options to reuse HTTP connections and allow-self signed certs

Problem: As of v.2.4.3, LogStream's AWS-related Sources & Destinations provide options to reuse HTTP connections, and to establish TLS connections to servers with self-signed certificates. However, the S3 Collector does not yet provide these options.
Fix: In LogStream 2.4.4.

2021-03-04 – v.2.4.2 – Esc key closes both Event Breaker Ruleset modals

Problem: After adding a rule to a Knowledge > Event Breaker Ruleset, pressing Esc closes the parent Ruleset modal along with the child Rule modal.
Workaround: Close the Rule modal by clicking either its Cancel button or its close box.
Fix: In LogStream 2.4.3.

2021-03-04 – v.2.4.2 – Aggregations Function in post-processing Pipeline addresses wrong Destination

Problem: An Aggregations Function, when used in a post-processing Pipeline, sends data to LogStream's Default Destination rather than to the Pipeline's attached Destination.
Workaround: If applicable, use the Function in a processing or pre-processing Pipeline instead.
Fix: In LogStream 2.4.3.

2021-02-25 – v.2.4.2 – On Safari, Event Breaker shows no OUT events

Problem: When viewing an Event Breaker's results on Safari, no events are displayed on the Preview pane's OUT tab.
Workaround: Use another supported browser.
Fix: In LogStream 2.4.3.

2021-02-22 – v.2.4.3 – Collection jobs UI errors

Problem: Collection jobs are missing from the Monitoring > Sources page, even though they are returned by metric queries. Also, the Job Inspector > Live modal displays an empty, unintended Configure tab.
Workaround: Use the Job Inspector to access collection results. Ignore the Configure tab.
Fix: In LogStream 2.4.4.

2021-02-19 – v.2.4.2 – Upon upgrade, Git remote repo setting breaks, blocking Worker Groups

Problem: If a Git remote repo was previously configured, upgrading to LogStream v.2.4.2 throws errors of this form upon startup: Failed to initialize git repository. Config versioning will not be available...Invalid URL.... The Master cannot commit or deploy to any Worker Group.
Workarounds: 1. Downgrade back to v.2.4.1 (or your previous working version). 2. Switch from Basic authentication to SSH authentication against the repo, to remove the username from requests. (The apparent root cause is Basic/http auth using a valid URL and username, but missing a password.)
Fix: In LogStream 2.4.3.

2021-02-19 – v.2.4.0, 2.4.1, 2.4.2 – Splunk (S2S) Forwarder access control blocks upon upgrade to LogStream 2.4.x

Problem: If Splunk indexers have forwarder tokens enabled, and worked with LogStream 2.3.x before, upgrading to LogStream 2.4.x causes data to stop flowing.
Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.
Fix: In LogStream 2.4.3.

2021-02-10 – v.2.4.0, 2.4.1 – With Splunk HEC Source, JSON payloads containing embedded objects trigger high CPU usage

Problem: Splunk HEC JSON payloads containing nested objects trigger high CPU usage, due to a flaw in JSON parsing.
Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.
Fix: In LogStream 2.4.2.

2021-01-30 – v.2.4.0 – Worker Nodes cannot connect to Master

Problem: Worker Nodes cannot connect to the Master after the Master is upgraded to v.2.4.0.
Workaround: Disable compression on the Workers. You can do so through the Workers' UI at System Settings > Distributed Settings > Master Settings > Compression, or by commenting out this line in each Worker's cribl.yml config file:

compression: gzip

Fix: In LogStream 2.4.1.

2021-01-25 – v.2.4.0 – S3 collection stops working due to auth secret key issues.

Problem: S3 collection stops after upgrade to 2.4.0 due to secret key re-encryption.
Workaround: Re-configure S3, save and re-deploy.
Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Google Cloud Storage Destination Needs Extra Endpoint to Initialize

Problem: The Google Cloud Storage Destination fails to initialize, displaying an error of the form: Bucket does not exist!
Workaround: In the outputs.yml file, under your cribl-gcp-bucket key endpoint, add: https://storage.googleapis.com. (in a single-instance deployment, locate this file at $CRIBL_HOME/local/cribl/outputs.yml. In a distributed deployment, locate it at $CRIBL_HOME/groups/<group name>/local/cribl/outputs.yml.)
Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Worker Groups' Settings > Access Management Is Absent from UI

Problem: In this release, the Worker Groups > <group‑name> > System Settings UI did not display the expected Access Management, Authentication, and Local Users sections.
Workaround: Manually edit the users.json file.
Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Route Filters Aren't Copied to Capture Modal

Problem: On the Routes page, selecting Capture New in the right pane does not copy custom Filter expressions to the resulting Capture Sample Data modal. That modal's Filter Expression field always defaults to true.
Workarounds: 1. Bypass the Capture New button. Instead, from the Route's own ••• (Options) menu, select Capture. This initiates a capture with the Filter Expression correctly populated. 2. Copy/paste the expression into the Capture Sample Data modal's Filter Expression field. Or, if the expression is displayed in that field's history drop-down, retrieve it.
Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Destinations' Documentation Doesn't Render from UI

Problem: Clicking the Helplink in a Destination's configuration modal displays the error message: "Unable to load docs. Please check LogStream's online documentation instead."
Workarounds: 1. Go directly to the online Destinations docs, starting here. 2. Follow the UI link to the docs landing page, click through to open or download the current PDF, and scroll to its Destinations section.
Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Esc Key Doesn't Consistently Close Modals

Problem: Pressing Esc with focus on a modal's drop-down or slider doesn't close the modal as expected. (Pressing Esc with focus on a free-text field, combo box, or nothing does close the modal – displaying a confirmation dialog first, if you have unsaved changes.)
Workarounds: Click the X close box at upper right, or click Cancel at lower right.
Fix: Planned for LogStream 2.4.2.

2020-12-17 – v.2.3.0+ – Free-License Expiration Notice, Blocked Inputs

Problem: LogStream reports an expired Free license, and blocks inputs, even though Free licenses in v.2.3.0 do not expire.
Workaround: This is caused by time-limited Free license key originally entered in a LogStream version prior to 2.3.0. Go to Settings > Licensing, click to select and expand your expired Free license, and click Delete license. LogStream will recognize the new, permanent Free license, and will restore throughput.
Fix: In LogStream 2.4.1.

2020-11-14 – v.2.3.3 – Null Fields Redacted in Preview, but Still Forwarded

Problem: Where event fields have null values, LogStream (by default) displays them as struck-out in the right Preview pane. The preview is misleading, because the events are still sent to the output.
Workaround: If you do want to prevent fields with null values from reaching the output, use an Eval Function, with an appropriate Filter expression, to remove them.
Fix: Preview corrected in LogStream 2.3.4.

2020-10-27 – v.2.3.2 – Cannot Name or Save New Event Breaker Rule

Problem: After clicking Add Rule in a new or existing Event Breaker Ruleset, the Event Breaker Rule modal's Rule Name field is disabled. Because Rule Name is mandatory field, this also disables saving the Rule via the OK button.
Fix: In LogStream 2.3.3.

2020-10-12 – v.2.3.1 – Deleting One Function Deletes Others in Same Group

Problem: After inserting a new Function into a group and saving the Pipeline, deleting the Function also deletes other Functions lower down in the same group.
Fix: In LogStream 2.3.2.
Workaround: Move the target Function out of the group, resave the Pipeline, and only then delete the Function.

2020-09-27 – v.2.3.1 – Enabling Boot Start as Different User Fails

Problem: When a root user tries to enable boot-start as a different user (e.g., using /opt/cribl/bin/cribl boot-start enable -u <some‑username>), they receive an error of this form:

error: found user=0 as owner for path=/opt/cribl, expected uid=NaN. 
Please make sure CRIBL_HOME and its contents are owned by the uid=NaN by running: 
[sudo] chown -R NaN:[$group] /opt/cribl 

Fix: In LogStream 2.3.2.
Workaround: Install LogStream 2.2.3 (which you can download here), then upgrade to 2.3.1.

2020-09-17 – v.2.3.0 – Worker Groups menu tab hidden after upgrade to LogStream 2.3.0

Problem: Upon upgrading an earlier, licensed LogStream installation to v. 2.3.0, the Worker Groups tab might be absent from the Master Node's top menu.
Fix: In LogStream 2.3.1.
Workaround: Click the Home > Worker Groups tile to access Worker Groups.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 on RHEL 6, RHEL 7

Problem: Upon upgrading to v. 2.3.0, LogStream might fail to start on RHEL 6 or 7, with an error message of the following form. This occurs when the user running LogStream doesn't match the LogStream binary's owner. LogStream 2.3.0 applies a restrictive permissions check using id -un <uid>, which does not work with the version of id that ships with these RHEL releases.

id: 0: No such user
ERROR: Cannot run command because user=root with uid=0 does not own executable 

Fix: In LogStream 2.3.1.
Workaround: Update your RHEL environment's id version, if possible.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 with OpenId Connect

Problem: Upon upgrading an earlier LogStream installation to v. 2.3.0, OIDC users might be unable to restart the LogStream server.
Fix: In LogStream 2.3.1.
Workaround: Edit $CRIBL_HOME/default/cribl/cribl.yml to add the following lines to its the auth section:

filter_type: email_whitelist
scope: openid profile email

2020-06-11 – v.2.1.x – Can't switch from Worker to Master Mode

Problem: In a Distributed deployment, attempting to switch Distributed Settings from Worker to Master Mode blocks with a spurious "Git not available...Please install and try again" error message.
Fix: In LogStream 2.3.0.
Workaround: To initialize git, switch first from Worker to Single mode, and then from Single to Master mode.

2020-05-19 – v.2.1.x – Login page blocks

Problem: Entering valid credentials on the login page (e.g., http://localhost:9000/login) yields only a spinner.
Fix: In LogStream 2.3.0.
Workaround: Trim /login from the URL.

2020-02-22 – v.2.1.x – Deleting resources in default/

Problem: In a Distributed deployment, deleting resources in default/ causes them to reappear on restart.
Workaround/Fix: In progress.

2019-10-22 – v. 2.0 – In-product upgrade issue on v2.0

Problem: Using in-product upgrade feature in v.1.7 (or earlier) fails to upgrade to v2.0, due to package-name convention change.
Workaround/Fix: Download the new version and upgrade per steps laid out here.

2019-08-27 – v.1.7 – In-product upgrade issue on v1.7

Problem: Using in-product upgrade feature in v1.6 (or earlier) fails to upgrade to v1.7 due to package name convention change.
Workaround/Fix: Download the new package and upgrade per steps laid out here.

2019-03-21 – v.1.4 – S3 stagePath issue on upgrade to v.1.4+

Problem: When upgrading from v1.2 with a S3 output configured, stagePath was allowed to be undefined. In v.1.4+, stagePath is a required field. This might causing schema violations when upgrading older configs.
Workaround/Fix: Reconfigure the output with a valid stagePath filesystem path.