Skip to main content
Version: 3.2

Known Issues

2021-11-24 – v3.2.0 – Data from Collectors and Collector-based Sources isn't reaching Routes

Problem: Routes are not recognizing data from Collectors and from the following Collector-based Sources: Prometheus Scraper, Office 365 Activity, Office 365 Services, and Office 365 Message Trace. This data will not flow through Routes, but will be sent to the default Destination(s).

Workarounds: 1. In Collectors' config modals, open the Result Routing tab, Disable the Send to Routes default, and directly specify a Pipeline and Destination. (This option is not available in Prometheus Scraper or in the Office 365 Sources.) 2. Skip v.3.2.0.

Fix: Planned for LogStream 3.2.1.

2021-11-17 – v3.2.0 – TLS certs that use passphrases won't decrypt private keys

Problem: Sources whose TLS config uses a (known good) passphrase will fail to decrypt private keys. You will see a connect error, or an error of the form: TLS validation error, is passphrase correct?

Workarounds: 1. Edit the Leader Node's inputs.yml file to to insert a plaintext TLS passphrase; then reload or restart the LogStream server; then commit and deploy the new config. 2. Use a cert and key that do not require a passphrase. 3. Skip v.3.2.0.

Fix: Planned for LogStream 3.2.1.

2021-11-17 – v3.2.0 – Only one Chain Function works per Pipeline

Problem: If you add more than one Chain Function to a Pipeline, only the first will take effect. Chain Functions lower in the stack will simply pass the data down to the next Function.

Workaround: Design your data flow to require at most one Chain Function per Pipeline.

Fix: Planned for LogStream 3.2.1.

2021-11-17 – v3.2.0 – Exporting a Pack to another Group requires a Leader restart

Problem: Exporting a Pack to a different Worker Group via the UI succeeds, but opening the Pack on the target Group fails with a Cannot read property...undefined error.

Workaround: To resolve the error and make the target Pack accessible, restart the Leader. To prevent the error, export and import the Pack as a file.

Fix: Planned for LogStream 3.2.1.

2021-11-17 – v3.2.0 – Re-enabling a Function group mistakenly re-enables all its Functions

Problem: When you change a Function group from disabled to enabled, all of its Functions are enabled, regardless of their individual enabled/disabled states when the group was disabled.

Workaround: Avoid disabling and re-enabling Functions as a group (e.g., for testing or stepwise debugging purposes).

2021-11-16 – v3.2.0 – QuickConnected Source goes to wrong Destination

Problem: Some QuickConnect connections send data to an unintended Destination. We've observed this in single-instance deployments that include the Cribl-supplied cribl_metrics_rollup Pipeline, or include other Pipelines/Packs with stateful Functions like Rollup Metrics or Aggregations. Data will either flow through Routes instead of your specified QuickConnect Destination, or will continue flowing to the original QuickConnect Destination after you drag the connection to a different Destination.

Workaround: Use the Data Routes interface to manage the Pipeline and stateful Functions indicated above. If your QuickConnect data doesn't oblige a changed QuickConnect Destination, restart LogStream. This will stop data flow to the unintended Destination, and redirect it to the intended Destination.

Fix: Planned for LogStream 3.2.1.

2021-10-06 – v3.1.2 – Monitoring page omits Collector Sources' data

Problem: With functioning Office 365 Activity and Office 365 Services Sources, the Job Inspector reports data being retrieved, and Monitoring > Data > Destinations reports data being sent out. However, Monitoring > Data > Sources falsely reports no data being received. The problem is isolated to this Sources page. It might also affect the Office 365 Message Trace and Prometheus Scraper Sources.

Workaround: Use the Source modal's Live Data tab, the Monitoring > System > Job Inspector page, and/or the Monitoring > Data > Destinations page to monitor throughput.

Fix: In LogStream 3.1.3.

2021-09-29 – v.3.0.2, 3.1.1 – Memory leak with multiple Collectors

Problem: Configuring multiple Collectors can lead to gradual but cumulative memory leaks. Due to a caching error, the memory can be recovered only by restarting LogStream.

Workaround: Restart LogStream on the affected Worker Nodes.

Fix: In LogStream 3.1.3.

2021-09-15 – v3.0.0–3.1.3 – High CPU usage with Google Cloud Pub/Sub Source

Problem: The Google Cloud Pub/Sub Source substantially increases CPU usage, which stays high even after data stops flowing. This causes throughput degradation, and more-frequent failed to acknowledge errors.

Workaround: Configure the Google Cloud Pub/Sub Source's Advanced Settings > Max backlog to 1000.

Fix: In 3.2.0, which defaults the Max backlog to 1000, and also relaxes the retry interval from 10 seconds to 30 seconds.

2021-09-14 – v.3.1.1 – Modifying Collector > Preview > Capture settings can break Capture elsewhere

Problem: Modifying the capture settings in a Collector's Run > Preview > Capture modal can improperly modify the Filter expression in Capture modals for other Collectors, Sources, and Routes/Pipelines.

Fix: In LogStream 3.1.2.

2021-09-10 – v.3.1.1 – Worker Group certificate name drop-down shows certificates from the Leader

Problem: When configuring certificates at Groups<group‑name> > Settings > API Server Settings > TLS, certificates configured on the Leader incorrectly appear on the Certificate name drop-down.

Fix: In LogStream 3.1.2.

2021-09-10 – v.3.1.1 – Git Collapsed Actions broken in 3.1.1

Problem: With Collapsed Actions enabled, clicking the Commit & Push button has no effect. (The Commit & Deploy button works properly.)

Workaround: Disable Collapsed Actions, to restore separate Commit and Git Push buttons.

Fix: In LogStream 3.1.2.

2021-09-08 – All versions through v.3.1.1 – UDP support is currently IPv4-only

Problem: Where Sources and Destinations connect over UDP, they currently support IPv4 only, not IPv6. This applies to Syslog, Metrics, and SNMP Trap Sources; and to Syslog, SNMP Trap, StatsD, StatsD Extended, and Graphite Destinations.

Workaround: Integrate via IPv4 if possible.

Fix: UDP Sources and Destinations gained IPv6 support in LogStream 3.1.2.

2021-09-02 – v.3.1.0 – Google Pub/Sub authentication via proxy environment variable fails

Problem: When LogStream's Google Cloud Pub/Sub Source and Destination attempt authentication through a proxy, using the https_proxy environment variable, they send an HTTP request to http://www.googleapis.com:443/oauth2/v4/token. This request fails with 504/502 errors. The root cause is a mismatch in dependency libraries, whose correction has been identified, but requires broader testing.

Workaround: Configure the proxy in transparent mode, to avoid relying on environment variables.

Fix: In 3.1.2.

2021-08-24 – v.3.1.0 – High CPU load with LogStream version 3.1.0

Problem: LogStream 3.1.0 added code-execution safeguards that inadvertently increased CPU load, and decreased throughput, with several Functions and most expressions.

Workaround: Downgrade to v.3.0.4.

Fix: Upgrade to v.3.1.1 or later.

2021-08-18 – v.3.1.0 – Clone Collector option broken

Problem: Clicking a 3.1.0 Collector modal's Clone Collector button simply closes the modal. (If you have unsaved changes, you'll first be challenged to confirm closing the parent modal – but the expected cloned modal won't open.)

Workaround: Click + Add New to re-create your original Collector's config from scratch, adding any desired modifications.

Fix: In LogStream 3.1.1.

2021-08-18 – All versions through 3.1.0 – Tabbed code blocks broken on in-app docs

Problem: When docs with tabbed code blocks are opened in the Help drawer, the default (leftmost) tab seizes focus. Other tabs will not display when clicked.

Workaround: Click the blue/linked page title atop the Help drawer to open the same page on docs.cribl.io, where all tabs can be selected.

Fix: In LogStream 3.1.1.

2021-08-14 – v.3.1.0 – Splunk Load Balanced Destination does not migrate auth type

Problem: In a Splunk Load Balanced Destination with Indexer discovery enabled and a corresponding Auth token defined, upgrading to LogStream 3.1.0 corrupts the Auth token field's value.

Workaround: Set the Authentication method to Manual and resave the token's value.

Fix: In 3.1.1.

2021-08-11 – v.3.1.0 – C.Secret() values are undefined in Collectors

Problem: Calling the C.Secret() internal method within a Collector field resolves incorrectly to an undefined substring. E.g., in URL fields, C.Secret() values will resolve to /undefined/ path substrings.

Workarounds: 1. Use C.vars and a Global Variable, instead of using this method. 2. Root cause is that C.Secret() in Collectors and Pipeline Functions has access only to secrets that were created before the last restart. Therefore, restart Worker Processes to refresh the method's access.

Fix: In 3.1.1.

2021-08-10 – v.3.1.0 – Pre-processing Pipelines break Flows display

Problem: Attaching a pre-processing Pipeline to a Source breaks the Monitoring > Flows (beta) page's display. Attempting to remove Sources/Destinations from that page's selectors throws a cryptic Sankey error.

Workaround: Temporarily detach pre-processing Pipelines if you want to check Flows.

Fix: Planned for LogStream 3.1.1, but couldn't reproduce the error.

2021-07-29 – v.3.0.2 through 3.0.4 – Upgrades via UI require broader permissions

Problem: Upgrading from v.3.0.x via the UI requires the the cribl user to be granted write permission on the parent directory above $CRIBL_HOME. The symptom is an error message of the form: Upgrade failed: EACCES: permission denied, mkdir '/opt/unpack.xxxxxxx.tmp'.

Workaround: Either adjust permissions, or upgrade via the filesystem. For complete instructions, see Upgrading.

Fix: Does not affect LogStream 3.1 or higher.

2021-07-26 – v.3.0.x–3.1.0 – Packs with orphaned lookups block access to Worker Groups

Problem: If a Pack references a lookup file that's missing from the Pack, pushing the Pack to a Worker Group will block access to the Group's UI. You will see an error message of the form: "The Config Helper service is not available because a configuration file doesn't exist... Please fix it and restart LogStream."

Workaround: On the Leader Node, review the config helper logs ($CRIBL_HOME/log/groups/<group>/*.log) to see which references are broken. (In a single-instance deployment, see $CRIBL_HOME/log/*.log.) Then manually resolve these references in the Pack's configuration.

Fix: Planned for LogStream 3.1.1, but couldn't reproduce the error.

2021-07-20 – v.3.0.3 – Can't add Functions to a Pipeline named config

Problem: You cannot add Functions to a Pipeline if the Pipeline is named config, because this name conflicts with the reserved route for the Create Pipeline dialog.

Workarounds: Don'tcha name your Pipelines config.

Fix: Version TBD.

2021-07-06 – All versions through 3.1.x – Duplicate Workers/​Worker GUIDs

Problem: Multiple Workers have identical GUIDs. This creates problems in Monitoring, upgrading and versioning, etc., because all Workers show up as one.

Cause: This is caused by configuring one Worker and then copying its cribl/ directory to other Workers, to quickly bootstrap a deployment.

Workaround: Don't do this! Instead, use the Bootstrap Workers from Leader endpoint.

Fix: Planned for LogStream 3.4.0.

2021-07-02 – v.3.0.2–3.0.3 – Sample file's last line not displayed upon upload

Problem: When uploading (attaching) a sample data file, the file's final line is not displayed in the Add Sample Data modal.

Workarounds: This is a UI bug only. LogStream correctly processes the complete sample data, which should show up when viewing the sample afterwards (e.g., within a Pipeline's preview pane).

Fix: In LogStream 3.1.0.

2021-07-02 – All versions – Date fields misleadingly preview with string symbol

Problem: In Preview or Capture, incoming events like _raw will be displayed in the right pane with an α symbol that indicates string data. However, calling new Date() and then C.Time.strptime() methods in an Eval Function will return null on the OUT tab.

Cause: Due to the nature of JSON serialization, the incoming event's Date field is misleadingly subsumed under the event's α string symbol. It's actually a structured type, not a string...yet.

Workaround: If you see unexpected null results, stringify the datetime field as you extract it, e.g.: new Date().toISOString(). Feeding the resulting field to Time methods should return datetime strings as expected.

2021-06-22 – v.3.0.0–3.0.2 – Internal C.Text.relativeEntropy() method – broken typeahead and preview

Problem: The C.Text.relativeEntropy() internal method is missing from JavaScript expressions' typeahead drop-downs. You can manually type or paste in the method, and save your Function and Pipeline, but LogStream's right Preview pane will (misleadingly) always show the method returning 0.

Workarounds: Use other means (such as the Live button) to preview and verify that the method is (in fact) returning valid results.

Fix: In LogStream 3.0.3.

2021-05-20 – 3.0.0 – Multiple Functions Break LogStream 3.0 Pipelines

Problem: After upgrade to LogStream 3.0.0, including any of the following Functions in a Pipeline can break the Pipeline: GeoIP, Redis, DNS Lookup, Reverse DNS, Tee. Symptom is an error of the form: Pipeline process timeout has occurred. Less seriously, including these Functions in a Pipeline can suppress Preview's display of fields/values.

Workarounds: If you use these Functions in your Pipelines, stay with (or restore) a pre-3.0 version until LogStream 3.0.1 is available.

Fix: In LogStream 3.0.1.

2021-05-19 – 3.0.0 – Leader's Changes fly-out stays open after Commit

Problem: In the Leader's left nav, the Changes fly-out remains stuck open after you commit pending changes.

Workarounds: Hover or click away. Then hover or click back to reopen the fly-out.

Fix: In LogStream 3.0.1.

2021-05-18 – 3.0.0 – Packs > Export in "Merge" mode omits schemas and custom Functions

Problem: Exporting a Pack with the export mode set to Merge omits schemas and custom Functions configured within the Pack's Knowledge > Schemas.

Workarounds: 1. Change the export mode to Merge safe, and export again. 2. If that doesn't preserve the schema and Functions, revert to Merge export mode; install the resulting Pack onto its target(s); and then manually copy/paste the schema(s) and Functions from the source Pack's UI to the target Pack's UI.

Fix: In LogStream 3.0.1.

2021-05-17 – v.3.0.0–3.2.0 – Can't Enable KMS on Worker Group after installing license

Problem: Enabling HashiCorp Vault or AWS KMS on a Worker Group, after installing a LogStream license on the same Group, fails with a spurious External KMS is prohibited by the current license error message.

Workaround: On the Leader, navigate to Settings > Worker Processes. Restart the affected Worker Group's CONFIG_HELPER process. Then return to that Worker Group's Security > KMS Settings, re-enter the same KMS configuration, and save.

2021-05-10 – 2.4.5 – Elasticsearch Destination, with Auto version discovery, doesn't send Authorization header

Problem: When the Elasticsearch Destination has Basic Authentication enabled, and its Elastic version field specifies Auto version discovery, LogStream fails to send the configured username and password credentials along with its API initial request. Elasticsearch responds with an HTTP 401 error.

Workaround: Explicitly set the Elastic version to either 7.x or 6.x (depending on your Elasticsearch cluster's version); then stop and restart LogStream to pick up this configuration change.

Fix: In LogStream 3.1.0.

2021-05-04 – 2.4.5 – Office 365 Message Trace Source skips events

Problem: The Event Breaker Rule provided for the Office 365 Message Trace Source mistakenly presets the Default timezone to ETC/GMT‑0. This setting causes LogStream to discover events but not collect them.

Workaround: Reset the Rule's Default timezone to UTC, then click OK and resave the Ruleset.

Fix: In 3.0.2.

2021-05-03 – v.2.4.4–3.01 – Rollup Function suppresses sourcetype metrics

Problem: sourcetype metrics can be suppressed when the Cribl Internal > CriblMetrics Source is enabled and the cribl_metrics_rollup pre-processing Pipeline is attached to a Source.

Workarounds: Disabling the pre-processing pipeline restores sourcetype and any other missing data. However, without the rollup, a much higher data volume will be sent to the indexing tier.

Fix: In LogStream 3.0.2.

2021-04-20 – v.2.4.3–2.4.5 – Orphaned S3 staging directories

Problem: Using the S3 Destination, defining a partitioning expression with high cardinality can proliferate a large number (up to millions) of empty directories. This is because LogStream cleans up staged files, but not staging directories.

Workaround: Programmatically or manually delete stale staging directories (e.g., those older than 30 days).

Fix: In LogStream 3.0.2.

2021-04-12 – 2.4.4 – Splunk Sources do not support multiple-metric events

Problem: LogStream's Splunk Sources do not support multiple-measurement metric data points. (LogStream's Splunk Load Balanced Destination does.)

Fix: In LogStream 3.0.1.

2021-04-07 – v.2.4.2–2.4.5 – Google Cloud Storage Destination fails to upload files > 5 MB

Problem: The Google Cloud Storage Destination might fail to put objects into GCS buckets. This happens with files larger than 5 MB, and causes the Google Cloud API to report a vague Invalid argument error.

Workaround: Set the Max file size (MB) to 5 MB. Also, reduce the Max file open time (sec) limit from its default 300 (5 minutes) to a shorter interval, to prevent files from growing to the 5 MB threshold. (Tune this limit based on your observed rate of traffic flow through the Destination.)

Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.4 – Local login option visible even when disabled

Problem: The Log in with local user option is displayed to users even when you have disabled Settings > Authentication > Allow local auth for an OpenID Connect identity provider.

Workaround: Advise users to ignore this button. Although visible, it will not function.

Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.0–2.4.4 – Splunk TCP and LB Destinations' Workers trigger OOM errors and restart

Problem: With a Splunk TCP or Splunk Load Balanced Destination created after upgrading to LogStream 2.4.x, Workers' memory consumption may grow without bound, leading to out-of-memory errors. The API Process will restart the Workers, but there might be temporary outages.

Workaround: Toggle the Destination's Advanced Settings > Minimize in‑flight data loss slider to No. This will preserve Processes killed by OOM conditions.

Fix: In LogStream 2.4.5.

2021-03-31 – v.2.4.4 – OpenID Connect authentication always shows local-auth fallback

Problem: Even if OpenID Connect external authentication is configured to disable Allow local auth, LogStream's login page displays a Log in with local user button.

Workaround: Do not click that button.

Fix: In LogStream 3.0.0.

2021-03-31 – v.2.4.4 – Authentication options mistakenly display Cribl Cloud

Problem: The Settings > Authentication > Type drop-down offers a Cribl Cloud option, which is not currently functional. Attempting to configure and save this option could lock the admin user out of LogStream.

Workaround: Do not select, configure, or save that option.

Fix: In LogStream 2.4.5.

2021-03-30 – v.2.4.4 – Can't disable some Sources from within their config modals

Problem: In configuration modals for the Azure Blob Storage and Office 365 Message Trace Sources, the Enabled slider cannot be toggled off, and its tooltip doesn't appear.

Workaround: Disable your configured Source (where required) from the Manage Blob Storage Sources or the Manage Message Trace Sources page.

Fix: In LogStream 2.4.5.

2021-03-29 – v.2.4.x – SpaceOut Destination is broken

Problem: Within the SpaceOut game, you cannot shoot, and your player is immortal.

Workaround: There are other video games. After we defeat COVID, you'll even be able to buy a PS5.

Fix: Restored in LogStream 2.4.5.

2021-03-24 – v.2.4.x – Cribl App for Splunk blocks admin password changes, configuration changes, and Splunk-based authentication

Problem: Attempting to change the admin password via the UI triggers a 403/Forbidden message. You can reset the password by editing users.json, but can't save configuration changes to Settings, Pipelines, etc., because RBAC Roles are not properly applied.

Workaround: Using a 2.3.x version of the App enables local authentication and enables changes to Cribl/LogStream passwords and configuration/settings.

Fix: In LogStream 2.4.4.

2021-03-22 – v.1.7 through 2.4.3 – Azure Event Hubs Destination: Compression must be manually disabled

Problem: LogStream's Azure Event Hubs Destination provides a Compression option that defaults to Gzip. However, compressed Kafka messages are not yet supported on Azure Event Hubs.

Workaround: Manually reset Compression to None, then resave Azure Event Hubs Destinations.

Fix: In LogStream 2.4.4.

2021-03-17 – v.2.4.2, 2.4.3 – Parser Function > List of Fields copy/paste fails

Problem: When copying/pasting List of Fields contents between Parser Functions via the Copy button, the paste operation inserts unintended metadata instead of the original field references.

Workaround: Manually re-enter the second Parser Function's List of Fields.

Fix: In LogStream 2.4.4.

2021-03-13 – v.2.4.3 – UI can't find valid TLS .key files, blocking Master restarts and Worker reconfiguration

Problem: After upgrading to v.2.4.3, the UI fails to recognize valid TLS .key files, displaying spurious error messages of the form: "File does not exist: $CRIBL_HOME/local/cribl/auth/certs/<keyname>key." An affected Master will not restart. Affected Workers will restart, but will not apply changes made through the UI.

Workaround: Ideally, specify an absolute path to each key file, rather than relying on environment variables. If you're locked out of the UI, you'll need to manually edit the referenced paths within these configuration files in LogStream subdirectories: local/cribl/cribl.yml (General > API Server TLS settings) and/or local/_system/instance.yml (Distributed > TLS settings). Contact Cribl Support if you need assistance. A more drastic workaround is to disable TLS for the affected connections.

Fix: In LogStream 2.4.4.

2021-03-12 – v.2.4.2 – Redis Function with specific username can't authenticate against Redis 6.x ACLs

Problem: The Redis Function, when used with a specific username and Redis 6.x's Access Control List feature, fails due to authentication problems.

Workaround: In the Function's Redis URL field, point to the Redis default account, either with a password (e.g., redis://default:Password1@192.168.1.20:6379) or with no password (redis://192.168.1.20:6379). Do not specify a user other than default.

Fix: In LogStream 3.0.

2021-03-09 – v.2.4.3 – Splunk Destinations' in-app docs mismatch UI's current field order

Problem: For the Splunk Single Instance and Splunk Load Balanced Destinations, the in-app documentation omits the UI's Advanced Settings section. Some fields are documented out-of-sequence, or are omitted.

Workaround: Refer to the UI's tooltips, to the corrected Splunk Single Instance and Splunk Load Balanced online docs, and/or to the corrected PDF.

Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – Enabling Git Collapse Actions breaks Commit & Deploy

Problem: After enabling Settings > Distributed Settings > Git Settings > General > Collapse Actions, selecting Commit & Deploy throws a 500 error.

Workaround: Disable the Collapse Actions setting, then commit and deploy separately.

Fix: In LogStream 2.4.4.

2021-03-08 – v.2.4.3 – S3 Collector lacks options to reuse HTTP connections and allow-self signed certs

Problem: As of v.2.4.3, LogStream's AWS-related Sources & Destinations provide options to reuse HTTP connections, and to establish TLS connections to servers with self-signed certificates. However, the S3 Collector does not yet provide these options.

Fix: In LogStream 2.4.4.

2021-03-04 – v.2.4.2 – Esc key closes both Event Breaker Ruleset modals

Problem: After adding a rule to a Knowledge > Event Breaker Ruleset, pressing Esc closes the parent Ruleset modal along with the child Rule modal.

Workaround: Close the Rule modal by clicking either its Cancel button or its close box.

Fix: In LogStream 2.4.3.

2021-03-04 – v.2.4.2 – Aggregations Function in post-processing Pipeline addresses wrong Destination

Problem: An Aggregations Function, when used in a post-processing Pipeline, sends data to LogStream's Default Destination rather than to the Pipeline's attached Destination.

Workaround: If applicable, use the Function in a processing or pre-processing Pipeline instead.

Fix: In LogStream 2.4.3.

2021-02-25 – v.2.4.2 – On Safari, Event Breaker shows no OUT events

Problem: When viewing an Event Breaker's results on Safari, no events are displayed on the Preview pane's OUT tab.

Workaround: Use another supported browser.

Fix: In LogStream 2.4.3.

2021-02-22 – v.2.4.3 – Collection jobs UI errors

Problem: Collection jobs are missing from the Monitoring > Sources page, even though they are returned by metric queries. Also, the Job Inspector > Live modal displays an empty, unintended Configure tab.

Workaround: Use the Job Inspector to access collection results. Ignore the Configure tab.

Fix: In LogStream 2.4.4.

2021-02-19 – v.2.4.2 – Upon upgrade, Git remote repo setting breaks, blocking Worker Groups

Problem: If a Git remote repo was previously configured, upgrading to LogStream v.2.4.2 throws errors of this form upon startup: Failed to initialize git repository. Config versioning will not be available...Invalid URL.... The Master cannot commit or deploy to any Worker Group.

Workarounds: 1. Downgrade back to v.2.4.1 (or your previous working version). 2. Switch from Basic authentication to SSH authentication against the repo, to remove the username from requests. (The apparent root cause is Basic/http auth using a valid URL and username, but missing a password.)

Fix: In LogStream 2.4.3.

2021-02-19 – v.2.4.0, 2.4.1, 2.4.2 – Splunk (S2S) Forwarder access control blocks upon upgrade to LogStream 2.4.x

Problem: If Splunk indexers have forwarder tokens enabled, and worked with LogStream 2.3.x before, upgrading to LogStream 2.4.x causes data to stop flowing.

Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.

Fix: In LogStream 2.4.3.

2021-02-10 – v.2.4.0, 2.4.1 – With Splunk HEC Source, JSON payloads containing embedded objects trigger high CPU usage

Problem: Splunk HEC JSON payloads containing nested objects trigger high CPU usage, due to a flaw in JSON parsing.

Workaround: If you encounter this problem, rolling back to your previously installed LogStream version (such as v.2.3.4) resolves it.

Fix: In LogStream 2.4.2.

2021-01-30 – v.2.4.0 – Worker Nodes cannot connect to Master

Problem: Worker Nodes cannot connect to the Master after the Master is upgraded to v.2.4.0.

Workaround: Disable compression on the Workers. You can do so through the Workers' UI at System Settings > Distributed Settings > Master Settings > Compression, or by commenting out this line in each Worker's cribl.yml config file:

compression: gzip

Fix: In LogStream 2.4.1.

2021-01-25 – v.2.4.0 – S3 collection stops working due to auth secret key issues.

Problem: S3 collection stops after upgrade to 2.4.0 due to secret key re-encryption.

Workaround: Re-configure S3, save and re-deploy.

Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Google Cloud Storage Destination Needs Extra Endpoint to Initialize

Problem: The Google Cloud Storage Destination fails to initialize, displaying an error of the form: Bucket does not exist!

Workaround: In the outputs.yml file, under your cribl-gcp-bucket key endpoint, add: https://storage.googleapis.com. (in a single-instance deployment, locate this file at $CRIBL_HOME/local/cribl/outputs.yml. In a distributed deployment, locate it at $CRIBL_HOME/groups/<group name>/local/cribl/outputs.yml.)

Fix: In LogStream 2.4.1.

2021-01-14 – v.2.4.0 – Worker Groups' Settings > Access Management Is Absent from UI

Problem: In this release, the Worker Groups > <group‑name> > System Settings UI did not display the expected Access Management, Authentication, and Local Users sections.

Workaround: Manually edit the users.json file.

Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Route Filters Aren't Copied to Capture Modal

Problem: On the Routes page, selecting Capture New in the right pane does not copy custom Filter expressions to the resulting Capture Sample Data modal. That modal's Filter Expression field always defaults to true.

Workarounds: 1. Bypass the Capture New button. Instead, from the Route's own ••• (Options) menu, select Capture. This initiates a capture with the Filter Expression correctly populated. 2. Copy/paste the expression into the Capture Sample Data modal's Filter Expression field. Or, if the expression is displayed in that field's history drop-down, retrieve it.

Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Destinations' Documentation Doesn't Render from UI

Problem: Clicking the Help

link in a Destination's configuration modal displays the error message: "Unable to load docs. Please check LogStream's online documentation instead."

Workarounds: 1. Go directly to the online Destinations docs, starting here. 2. Follow the UI link to the docs landing page, click through to open or download the current PDF, and scroll to its Destinations section.

Fix: In LogStream 2.4.1.

2021-01-13 – v.2.4.0 – Esc Key Doesn't Consistently Close Modals

Problem: Pressing Esc with focus on a modal's drop-down or slider doesn't close the modal as expected. (Pressing Esc with focus on a free-text field, combo box, or nothing does close the modal – displaying a confirmation dialog first, if you have unsaved changes.)

Workarounds: Click the X close box at upper right, or click Cancel at lower right.

Fix: In LogStream 2.4.1.

2020-12-17 – v.2.3.0+ – Free-License Expiration Notice, Blocked Inputs

Problem: LogStream reports an expired Free license, and blocks inputs, even though Free licenses in v.2.3.0 do not expire.

Workaround: This is caused by time-limited Free license key originally entered in a LogStream version prior to 2.3.0. Go to Settings > Licensing, click to select and expand your expired Free license, and click Delete license. LogStream will recognize the new, permanent Free license, and will restore throughput.

Fix: In LogStream 2.4.1.

2020-11-14 – v.2.3.3 – Null Fields Redacted in Preview, but Still Forwarded

Problem: Where event fields have null values, LogStream (by default) displays them as struck-out in the right Preview pane. The preview is misleading, because the events are still sent to the output.

Workaround: If you do want to prevent fields with null values from reaching the output, use an Eval Function, with an appropriate Filter expression, to remove them.

Fix: Preview corrected in LogStream 2.3.4.

2020-10-27 – v.2.3.2 – Cannot Name or Save New Event Breaker Rule

Problem: After clicking Add Rule in a new or existing Event Breaker Ruleset, the Event Breaker Rule modal's Rule Name field is disabled. Because Rule Name is mandatory field, this also disables saving the Rule via the OK button.

Fix: In LogStream 2.3.3.

2020-10-12 – v.2.3.1 – Deleting One Function Deletes Others in Same Group

Problem: After inserting a new Function into a group and saving the Pipeline, deleting the Function also deletes other Functions lower down in the same group.

Fix: In LogStream 2.3.2.

Workaround: Move the target Function out of the group, resave the Pipeline, and only then delete the Function.

2020-09-27 – v.2.3.1 – Enabling Boot Start as Different User Fails

Problem: When a root user tries to enable boot-start as a different user (e.g., using /opt/cribl/bin/cribl boot-start enable -u <some‑username>), they receive an error of this form:

error: found user=0 as owner for path=/opt/cribl, expected uid=NaN. 
Please make sure CRIBL_HOME and its contents are owned by the uid=NaN by running:
[sudo] chown -R NaN:[$group] /opt/cribl

Fix: In LogStream 2.3.2.

Workaround: Install LogStream 2.2.3 (which you can download here), then upgrade to 2.3.1.

2020-09-17 – v.2.3.0 – Worker Groups menu tab hidden after upgrade to LogStream 2.3.0

Problem: Upon upgrading an earlier, licensed LogStream installation to v.2.3.0, the Worker Groups tab might be absent from the Master Node's top menu.

Fix: In LogStream 2.3.1.

Workaround: Click the Home > Worker Groups tile to access Worker Groups.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 on RHEL 6, RHEL 7

Problem: Upon upgrading to v.2.3.0, LogStream might fail to start on RHEL 6 or 7, with an error message of the following form. This occurs when the user running LogStream doesn't match the LogStream binary's owner. LogStream 2.3.0 applies a restrictive permissions check using id -un <uid>, which does not work with the version of id that ships with these RHEL releases.

id: 0: No such user
ERROR: Cannot run command because user=root with uid=0 does not own executable

Fix: In LogStream 2.3.1.

Workaround: Update your RHEL environment's id version, if possible.

2020-09-17 – v.2.3.0 – Cannot Start LogStream 2.3.0 with OpenId Connect

Problem: Upon upgrading an earlier LogStream installation to v.2.3.0, OIDC users might be unable to restart the LogStream server.

Fix: In LogStream 2.3.1.

Workaround: Edit $CRIBL_HOME/default/cribl/cribl.yml to add the following lines to its the auth section:

filter_type: email_whitelist
scope: openid profile email

2020-06-11 – v.2.1.x – Can't switch from Worker to Master Mode

Problem: In a Distributed deployment, attempting to switch Distributed Settings from Worker to Master Mode blocks with a spurious "Git not available...Please install and try again" error message.

Fix: In LogStream 2.3.0.

Workaround: To initialize git, switch first from Worker to Single mode, and then from Single to Master mode.

2020-05-19 – v.2.1.x – Login page blocks

Problem: Entering valid credentials on the login page (e.g., http://localhost:9000/login) yields only a spinner.

Fix: In LogStream 2.3.0.

Workaround: Trim /login from the URL.

2020-02-22 – v.2.1.x – Deleting resources in default/

Problem: In a Distributed deployment, deleting resources in default/ causes them to reappear on restart.

Workaround/Fix: In progress.

2019-10-22 – v. 2.0 – In-product upgrade issue on v2.0

Problem: Using in-product upgrade feature in v.1.7 (or earlier) fails to upgrade to v2.0, due to package-name convention change.

Workaround/Fix: Download the new version and upgrade per steps laid out here.

2019-08-27 – v.1.7 – In-product upgrade issue on v1.7

Problem: Using in-product upgrade feature in v1.6 (or earlier) fails to upgrade to v1.7 due to package name convention change.

Workaround/Fix: Download the new package and upgrade per steps laid out here.

2019-03-21 – v.1.4 – S3 stagePath issue on upgrade to v.1.4+

Problem: When upgrading from v1.2 with a S3 output configured, stagePath was allowed to be undefined. In v.1.4+, stagePath is a required field. This might causing schema violations when upgrading older configs.

Workaround/Fix: Reconfigure the output with a valid stagePath filesystem path.