Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF – v.3.1.2

Cribl LogStream 2.3

about a year ago by Michael Katz

2020-09-16 - Cribl LogStream 2.3 – GA Release

New Features

Scheduled and Enhanced Data Collection

  • You can now set recurring schedules (cron jobs) to run distributed collection of data from multiple sources.
  • You can configure whether to skip, or defer and retry, scheduled collection jobs blocked by resource constraints.
  • You can limit concurrent running instances of all collection jobs, of all scheduled collection jobs, and of individual scheduled jobs.
  • You can now use relative time-range offsets to configure ad hoc and scheduled collection jobs.
  • Ad hoc (not scheduled) collection jobs can now automatically resume upon LogStream restart. Toggle this option per Collector at Advanced Settings > Resume job on boot.

REST Data Collector

  • Added ad hoc and scheduled collection of data from REST endpoints.
  • The new REST Collector offers four different discovery types, for progressively more schema-agnostic and dynamic data retrieval.

New Functions

  • Grok: This Function extracts fields from unstructured log data, using modular regex patterns. This makes it easy to load in and reuse pre-existing work.
  • Rename: This Function enables renaming of individual fields, or bulk renaming.
  • Metrics Rollup: This Function helps you aggregate frequently generated metrics into wider time buckets.

New Sources

  • Kinesis Firehose: Receive streaming data via Kinesis' HTTP(S) endpoint.
  • Raw HTTP: Receive all HTTP requests on a specified port, creating corresponding events that can be pushed to Event Breakers.
  • Office 365 Services: Receive batch data from the Office 365 Service Communications API, covering service incidents on multiple Microsoft cloud services.
  • Office 365 Activity: Receive batch data from the Office 365 Management Activity API, covering actions and events on Azure Active Directory, Exchange, SharePoint, and other Microsoft servers.

New Destinations

  • Wavefront: Added support for sending events out to Wavefront analytics.
  • SignalFx: Added support for sending objects out to SignalFx monitoring.

New Datagens

  • LogStream now ships with more datagen sample files, which quick-start configuration and testing on several common scenarios.

Improved Monitoring/Visualizations

  • Dense data views are now condensed to compact sparklines, to quickly show overall trends.
  • Richer details are available on click/zoom.
  • Bytes in/out, and events in/out, can be independently displayed.
  • Average and instantaneous (previous-minute) throughputs are now displayed.
  • A new Monitoring > Jobs page displays in-flight, queued, and failed collection jobs/tasks, per Worker Node.
  • Collection jobs now have improved error logging, along with clearer metrics and error reporting in the Job Inspector modal.

Simplified Preview Pane

  • Cleaner UI for capturing, attaching, pasting, reloading, and reviewing sample data.
  • Expand All toggle added to collapsed events.
  • Expanded display of existing sample and datagen files.

In-Product Documentation

  • Documentation on Sources, Destinations, and Functions is now available directly in LogStream, even without an internet connection (such as airgapped installations).
  • Sources' documentation now prominently indicates Event Breaker support.

Permanent Free License

  • A permanent Free license (with certain restrictions) is now included in the download.
  • LogStream Free licenses do not expire, so they no longer require quarterly renewal. (LogStream One licenses do expire, and require annual renewal.)
  • Existing LogStream Free and LogStream One configurations and workflows might require some updating.

Working With Data: New Features and Improvements

  • Destinations now support System Fields that (optionally) identify the LogStream Source, Pipeline, Node, Worker Process, Destination, etc., that processed each event.
  • The Parser Function now provides a Clean Fields option, to replace non-alphanumeric characters with _.
  • The Parser Function now also supports extraction of delimiter-based values, with arbitrary Delimiter, Quote, and Escape characters.

UX/UI: New Features and improvements

  • First run of ./cribl start now displays the default username and password in the CLI.
  • Preview pane can now be fully collapsed, allowing wider display of Route and Pipeline details.
  • Monitoring > Stats page now provides Maximize buttons on individual panels/graphs.
  • Saving a stored TLS certificate to a Source or Destination now supports a passphrase.
  • Settings > Certificates UI now shows certificates' expiration dates.
  • Settings > Certificates UI now shows disk paths to certificates and keys.
  • Licenses are now automatically propagated from the Master to Worker Groups.
  • License issues/failures for all Worker Groups are now displayed upon login to the Master Node.
  • Syslog Source now provides a Default Timezone selector.
  • Splunk HEC Source now provides a Description field to clearly identify individual Auth tokens.
  • CLI now warns when a command is run as a different user.

Other Improvements or Changes

  • A large number of general UX improvements.
  • ./cribl keys CLI command now supports a -g <group> switch, to manage keys in distributed deployments.
  • Settings > General Settings now include a Proxy Settings option to ignore HTTP/S_PROXY environment variables.
  • All AWS Sources and Destinations now support Assume Role.
  • Splunk Load Balanced Destination now supports nested field serialization.
  • Splunk HEC Source now supports /services/collector/raw endpoint and associated metadata.
  • Splunk HEC Destination now supports compressed payloads.
  • LogStream now supports Splunk bucket readers.
  • All regex inputs now support the dotAll modifier flag.
  • Knowledge > Regex Library now supports field extractions (named capture groups) for common data sources.
  • Knowledge > Event Breaker Rules Library now includes a Do Not Break ruleset.
  • LogStream now closes Sources upon Worker Process shutdown.
  • args.json file is now encrypted before disk write, and decrypted upon read.
  • For clarity, renamed input conditioning Pipelines to pre-processing Pipelines.
  • For clarity, renamed output conditioning Pipelines to post-processing Pipelines.
  • Task reapers are now created lazily – when needed – to improve resource efficiency.
  • Task reapers now process successful tasks before unsuccessful tasks.
  • Shipped a new license that expires on: Never.

Mac OS Builds Discontinued

As of version 2.3.0, Cribl no longer provides darwin builds of LogStream for Mac OS. LogStream now incorporates native Linux components, to support features like the new Grok Function on LogStream's production OS.