Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF – v.3.1.2

Cribl LogStream 3.0

5 months ago by Michael Katz

2021-05-18 – Cribl LogStream 3.0 – GA Release

New Features

Ian the Goat is dancing because this is a major release.

Contextual Side + Top Navigation

We realized we were going to need a bigger boat. So we moved several anchored controls from LogStream's top nav to a slim left nav, which expands upon hover. This cleared room for a slimmer, contextual top nav.

This helped us throw the Data drop-down overboard, and promote Sources, Collectors, and Destinations to readily accessible, independent top links. When lots of controls accumulate on narrow browsers, LogStream will responsively move the rightmost controls onto a new ••• overflow menu.

Left nav, top nav, overflow menuLeft nav, top nav, overflow menu

Left nav, top nav, overflow menu

Packs Make Configuration Portable

A group of goats is called a "tribe" (or, for certain California goats, a "trip"); a group of LogStream configuration objects is now called a Pack. Highlighted above, this new feature enables administrators and developers to package and readily share extended configs (Pipelines, sample files, Knowledge objects, and more) across Worker Groups, and across organizations.

Packs are usable now – we've included a starter Pack in the 3.0 build – and we'll be adding further capabilities, too.

Renamed Modes

Together with the new slim side nav and flexible top nav, you'll find new naming (and shortcuts) to indicate what LogStream mode you're operating in:

  • In a single-instance deployment, you'll see (S)ingle Mode at left, and global controls anchored to the top nav.

  • In a distributed deployment without an Enterprise license (single Worker Group), a Configure link on the left nav replaces the former Default Group top link. The Workers, Mappings, and Monitoring links have also moved here.

  • In an Enterprise distributed deployment with multiple Worker Groups, click the new Groups link at left to select a Group and display its controls on the top nav.

  • The (L)eader Mode indicator at left shows the status of the hub/controller instance formerly known as the Master Node. You'll also now see this Leader naming throughout the UI and docs, along with its followers allowlist and blocklist. We'll be updating nomenclature within the codebase gradually, to avoid breaking changes (which we've found vastly overrated).

  • To display LogStream's Home page, click the new LogStream logo at the top left.

  • Right below that is a Find link where you can access global keyword search across LogStream objects (Routes, Pipelines, Sources, Destinations, Collectors, Event Breaker, and more). You can now search on wildcards, and the Ctrl+K (all platforms) and Cmd+K (MacOS) keyboard shortcuts still work.

Expanded Security and Secrets-Management Options

A new Settings > Security submenu gathers existing settings (Encryption Keys and Certificates), plus new KMS and Secrets options. See our Securing topic for the new options: LogStream's new built-in Key Management Service enables all users to use stored secrets to authenticate on AWS-based and Google Cloud–based integrations. With an Enterprise license, you can instead use HashiCorp Vault as an external KMS.

Google Pub/Sub Source and Destination

Speaking of Google Cloud, LogStream now supports the Google Cloud Pub/Sub messaging service as a native Source and Destination.

Upgrade with Backup and Automatic Rollback (Beta)

Managed upgrade of Worker Nodes through LogStream's UI has been enhanced with new features: configurable backup settings, automatic rollback on failed upgrade, and a more-streamlined UI. For details, see our Upgrading topic. UI-based upgrade is still in beta (with accompanying warnings about backing up your production data), but now it's an even mo' bettah beta.

Data Flow Visualization (Beta)

On the Monitoring page, a new Data Flow (beta) link offers a configurable, graphic visualization of data traffic and volume through your whole LogStream deployment.

Monitoring > Data Flow (beta feature)Monitoring > Data Flow (beta feature)

Monitoring > Data Flow (beta feature)

Other New Features and Improvements

  • The Office 365 Activity Source now provides an optional Publisher identifier field (CRIBL-4705).
  • Lookups now indicate any Pipelines that reference them.
  • Logs now show a combined view from all Workers in a Group.
  • The most-recently-updated Groups are foregrounded on the new Groups fly-out.
  • In-product documentation can now be anchored in a right drawer.
  • The Reducing Windows XML Events use-case doc now includes alternative approach to parsing XML (CRIBL‑4999).

Known Issue

❗️

Certain Functions break LogStream 3.0.0 Pipelines

Including any of the following Functions can break a 3.0.0 Pipeline: GeoIP, Redis, DNS Lookup, Reverse DNS, Tee. The symptom is an error of the form: Pipeline process timeout has occurred. Less seriously, including these Functions in a Pipeline can suppress Preview's display of fields/values.

If you use any of these Functions in your Pipelines, skip LogStream 3.0.0. Version 3.0.1 fixes these incompatibilities.

Deprecated Function

🚧

The Prometheus Publisher Function is now deprecated.

LogStream's native Prometheus Destination takes its place (CRIBL-4713).

Corrections

  • With OpenID Connect identity providers, LogStream now hides the Log in with local user option from users after administrators disable Settings > Authentication > Allow local auth (CRIBL-4855).

  • In OpenID Connect login flows, LogStream no longer attempts to parse tokens (CRIBL-5073).

  • When authenticating on AWS using IAM roles, Vault KMS payloads now properly include credentials (CRIBL-5242).

  • Corrected Google Cloud Storage Destination's failure to send files larger than 5 MB (CRIBL-4903).

  • For the Office 365 Message Trace Source, corrected the Event Breaker Rule's default time zone to UTC, to prevent skipped events (CRIBL-5116).

  • The Office 365 Message Trace Source now provides an optional Timeout (secs) field. Also, the REST Collector now provides an optional Request Timeout (secs) field (CRIBL-4920).

  • The global Settings (lower left) > System > Logging > Levels > New Channel modal now allows colons (:) in channel names (CRIBL-5132).

  • Saving a sample data file with an existing name now allows renaming without data loss (CRIBL-5015).

  • Corrected datagen creation from a sample data associated with a Pipeline (CRIBL-5011).

  • Resolved failure to clean up consumed Persistent Queue files (CRIBL-2092).

  • The Redis Function now supports Redis 6.x's Access Control List feature with specific usernames. (CRIBL-4675).

  • On distributed deployments, version upgrades now log failures to download packages (CRIBL‑4836).