Cribl LogStream 3.2

November 16, 2021 · 6 min read
Michael Katz
Staff Technical Writer

2021-11-16 – Cribl LogStream 3.2 – GA Release

Read our announcement and download LogStream.

New Features

Cribl is pleased to announce a whole Turducken of new features in this holiday-season LogStream release.

QuickConnect

With v.3.2, we introduce a whole new way to configure LogStream. QuickConnect is a rapid-development UI, in which you can visually connect LogStream Sources to Destinations by simply dragging and dropping lines.

You can insert Pipelines or Packs in the connections, or – as a new option – you can omit these processing stages entirely, and send incoming data directly to Destinations. If you’ve been waiting for a simple, quick way to prototype, test, and even deploy LogStream, this is for you.

QuickConnect UI
QuickConnect UI

In v.3.2, when you display the home page of a LogStream single instance or Worker Group, you’ll see prominent tiles prompting you to choose between the QuickConnect versus Route configuration UI. Here, your choice is a bit like Tinder, except in reverse: swipe right to stay with LogStream’s longstanding, familiar Routing interface. (The FarmersOnly.com choice, if you like.) Or swipe left to try out something new!

QuickConnect versus Routing UIs
QuickConnect versus Routing UIs

New Home Page

Speaking of that home page, it looks a little different, right? We had white space rent-free, so now we’ve filled it with a basic dashboard that gives you an instant overview of Worker Groups (in distributed deployments), Monitoring status, and recent actions.

New Navigation

Now that we offer Routing versus QuickConnect parallel worlds, we’ve reorganized the LogStream UI’s top navigation. Again. Here’s what you’ll see immediately:

  • Sources (including Collector Sources) and Destinations are now under a new Data submenu at the upper left. (You can use this menu to navigate to Source and Destination configs even when you’re in the QuickConnect UI).
  • The new Routing menu (second from the left) enables you to teleport between the Data Routes (traditional configuration UI) versus QuickConnect worlds.
  • The new Processing menu (third from the left) now contains the links to access Pipelines, Packs, and Knowledge objects.

Also, within global ⚙️ Settings (lower left), we’ve promoted Git Settings parallel to Distributed Settings, to make all the Git stuff easier to find.

GitOps

While simplifying LogStream deployment by adding the QuickConnect option, we’ve also accelerated more-complex use cases. With an Enterprise license, you can use LogStream’s new GitOps features to separate development of your LogStream configuration from deployment. Use your favorite Git-based version-control platform’s pull request, approve/reject, and CI/CD workflows to push production-ready changes from a development or staging branch into a main/production branch or release.

New controls include a Workflow: Push API option in Git Settings, where you can also designate a Git Branch as your source of truth; a new Environment field on all integrations, to independently specify an active branch; and a new Enable expression option on Routes to programmatically define each Route’s Destination.

New Sources and Destinations

LogStream 3.2 introduces several new native integrations:

  • OpenTelemetry (OTel) Source and Destination
  • Confluent Source & Destination
  • CrowdStrike Source
  • And, wait for it, a…LogStream Source!

LogStream now supports the OpenTelemetry protocol (OTLP) for metrics and traces; we plan to add logs support as soon as this solidifies in the OpenTelemetry project.

LogStream can now ingest traces and metrics directly from an application, or via the OTel agent or collector, using the OTLP protocol. Within LogStream, you can enrich and redact this data, and can also parse traces into metrics to feed downstream analytics dashboards.

And wasn’t it time LogStream added a Source named “LogStream”? This new Source appears only in distributed deployments, for a specific purpose: In a hybrid Cloud deployment, you can use it to route data between Workers, without being double-billed for any data flow between LogStream’s original ingress and final egress.

New Functions

LogStream 3.2 introduces two new Functions:

  • Event Breaker now enables you to bring LogStream’s longstanding event-breaking logic into a processing Pipeline.
  • Chain enables you to send a Pipeline’s output to another Pipeline or a Pack, and…yes, this Function includes some scope restrictions and guardrails against creating circular references. (See Known Issues below.)

Expanded Security and Secrets-Management Options

  • AWS KMS is now supported as an external Key Management Service, joining our original HashiCorp Vault option.
  • The Kafka Source and Destination now support Kerberos authentication.
  • The Redis Function now supports secure, credential-based authentication.

New Pack Export Options

In distributed deployments, you can now export (distribute) one or multiple Packs to multiple Worker Groups, in one operation.

Enhanced Notifications

Backpressure Notifications can now be triggered by Drop as well as Block conditions. Each Notification will identify the triggering condition. Also, you can now define Notifications for Persistent Queue usage above a (configurable) threshold percentage of buffer capacity.

Upgrades Automation and Sequence

Upgrading Worker Nodes to LogStream’s newest stable version can now be automated. This option defaults to enabled for LogStream Cloud deployments, and disabled for on-prem/customer-managed deployments. You can toggle this behavior at global ⚙️ Settings (lower left) > Upgrade > Disable automatic upgrades.

Upgrading to LogStream 3.2 (or higher) requires committing and deploying the Leader’s upgrade before upgrading Workers. This is especially important if you enable the above automatic upgrades option on-prem. See this note.

Other New Features and Improvements

  • The Elasticsearch API Source provides a new Enable proxy mode option. This enables LogStream to proxy a downstream Elasticsearch server, which can handle non–Bulk API requests requests that LogStream does not natively support.
  • These docs are on a new platform, where we have more control over versioning and formatting. Expect them to look a little different at first…and then, over the next few weeks, they’ll look even more different.

Known Issues

Fixes for all of these are planned for v.3.2.1:

  • CRIBL-7013, CRIBL-7047 QuickConnect through stateful Functions sends data to unintended Destination.
  • CRIBL-7049 TLS certs that use passphrases with private keys throw decrypt errors.
  • CRIBL-7044 Only one Chain Function works per Pipeline.
  • CRIBL-7043 Exporting a Pack to another Group requires a Leader restart
  • CRIBL-6961 When the GitOps Push Workflow has placed the UI in read-only mode, the commit UI displays a Revert button, even though reverting changes is not currently supported. Pressing the button will simply trigger an error message.

(See Known Issues for a cumulative list of these and other known issues, with workarounds and resolution status.)

Corrections

  • CRIBL-6692 Authentication Settings UI now behaves normally after manually editing configuration files.
  • CRIBL-6182 Corrected Google Pub/Sub Source and Destination’s high CPU usage and failed to acknowledge errors.
  • CRIBL-6379 Elasticsearch Destination now sends events using the create action (replacing the index action, which caused dropped events).
  • CRIBL-6281 REST Collector and Office 365 Message Trace Source now have an Advanced Settings > Disable time filter option to prevent timestamp conflicts that prevented events from being collected.
  • CRIBL-6225 The Job Inspector now correctly shows names of configured Event Breakers and Destinations.
  • CRIBL-4153 Corrected Maximum call stack size exceeded error when collecting high-cardinality metrics.
  • CRIBL-6663 Diag bundles now exclude all .crt, .pem, .cer, and .key files located anywhere within the $CRIBL_HOME path.
  • CRIBL-6975 The API Reference now properly documents the /system/lookups, /lib/parsers, and /lib/regex endpoints.