Cribl LogStream ‚Äď Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF ‚Äď v.3.1.2

Cribl.Cloud Deployment

As an alternative to downloading and deploying the LogStream binary to a physical or virtual machine, you can register a hosted LogStream instance on our Cribl.Cloud portal. This launches a SaaS version of LogStream.

About LogStream (and This Page)

If you're new to LogStream, please see our Basic Concepts page and Getting Started Guide for orientation. The current page focuses on a Cloud deployment's differences from other deployment options ‚Äď referred to below as "LogStream binaries" or "customer-managed deployments."

Why Use Cloud Deployment?

LogStream Cloud is designed to simplify deployment, and to provide certain advantages over using your own infrastructure, in exchange for some current restrictions.

LogStream Cloud Advantages

  • Tap LogStream‚Äôs power, with no responsibility to install or manage software. LogStream¬†Cloud is fully hosted and managed by Cribl. so you can launch a configured instance within minutes.
  • Continuous delivery of upgrades and new features.
  • Free, up to 1 TB/day of data throughput (data ingress + egress) for all new accounts through Jan.¬†1, 2022.
  • Quickly expand your LogStream Cloud deployment by purchasing metered billing. Pay only for what you use.

LogStream Cloud Restrictions

LogStream Cloud provides (by design) a simplified deployment and a simplified user interface.

LogStream Cloud's simplified left navLogStream Cloud's simplified left nav

LogStream Cloud's simplified left nav

These are the current differences ‚Äď detailed below ‚Äď from the LogStream binary deployments that are described in the remainder of this documentation:

  • No Worker Groups, and no Worker Mappings controls.
  • No Filesystem Source, Collector, or Destination. A Cloud deployment has no local filesystem to read from or write to.
  • No Scripts or Script Collector.
  • Simplified administration. Cribl manages backup and restore of all configurations, high availability, and upgrade and maintenance, all automatically on your behalf.
  • No KMS secrets stores (available in LogStream binaries, v.3.0 and above).
  • TLS is provided on certain Sources for encryption only, with predefined certificates. LogStream¬†Cloud does not currently support importing your own certificates for TLS mutual authentication.
  • Hosted initially on AWS' US West Region, with more options to follow.

Cloud Deployment Quick Start

Ready to take the red pill? This section explains how to register and manage a LogStream Cloud instance.

Registering a Cribl.Cloud Portal

To get started:

  1. Start at: https://cribl.cloud/signup/
  2. Select the New User? Free signup option, and register.
  3. Follow Cribl's email link to confirm your registration and sign in.
  4. Bookmark your Cribl.Cloud portal page, for all that follows.

Select Organization Page

When you own or are a member of multiple Cribl.Cloud Organizations, the Select¬†Organization splash page ‚Äď displayed after you sign in ‚Äď enables you to select which Organization you want to work with.

Select Organization interstitial pageSelect Organization interstitial page

Select Organization interstitial page

Click any tile's \/ accordion to reveal a detailed description, if provided. Click the appropriate tile (or its open accordion's Dashboard button) to configure that Organization.

Organization tile's details and controlsOrganization tile's details and controls

Organization tile's details and controls

You¬†can click Leave if you want to remove yourself as a member of another owner's Organization. This option requires confirmation ‚Ästproceed only if you're sure! (You won't see this button on Organizations that you own.)

Exploring the Cribl.Cloud Portal

Now that you're here ‚Äď explore the furniture. The Cribl.Cloud portal's left sidebar allows you to navigate among the following pages/links:

Workspaces > LogStream

When you log into the Cribl.Cloud portal, you'll land on this page's Overview tab. This is where you'll launch your LogStream instance, and where you'll connect to it on subsequent logins.

Workspaces > LogStream ‚Äď Overview¬†tabWorkspaces > LogStream ‚Äď Overview¬†tab

Workspaces > LogStream ‚Äď Overview¬†tab

Overview Tab

The big Connect¬†to your LogStream button is the main event here ‚Äď click it to launch your LogStream instance. However, the surrounding pane displays the following useful information:

Location: Fully-qualified URL at which you access the associated LogStream instance.

Egress Address: The instance's current public IP address. This address is dynamic; Cribl will occasionally update it when we need to rescale core infrastructure.

Last Updated: Date on which Cribl last pushed an infrastructure change (notably including changes to the above Egress Address).

Version: Your deployed LogStream version. Latest indicates that you're in sync with the most-recent downloadable LogStream binary.

Region: The AWS Region where your LogStream instance is running.

Data Sources Tab

The same page's Data Sources tab lists ports, and data ingestion inputs, that are open and available to use. Return to this tab to copy Ingest Addresses (endpoints) as needed.

Organization

This left tab is displayed only to an Organization's owner. It offers the following tabs along its top.

Details Tab

The Organization Details tab offers these controls to make your Cribl.Cloud deployment more recognizable than its randomly generated Organization ID:

Alias: Optionally, enter a "friendly" name for your Organization. Upon signing in, members will see this alias above the Organization ID on the Select Organization page.

Description: Optionally, use this field to add further details about your Organization. On the Select Organization page, members can view these details by expanding the Organization's tile.

Click Save to immediately apply your changes.

There is also a Delete Organization button at the lower left. Don't click that. (Not unless you mean it.)

Organization Details tabOrganization Details tab

Organization Details tab

Members Tab

The Organization > Members tab provides access to inviting and managing other users.

Learning

The Learning page links to everything you need to learn about LogStream, to goat forth and do great things:

  • Sandboxes (free, interactive tutorials on fully hosted integrations).
  • Documentation.
  • LogStream versions comparison.
  • Concept/demo videos.

Software

If you prefer to take the blue pill, this page offers download links for Cribl's LogStream and AppScope software. You can download either binaries or Docker containers (hosting Ubuntu 20.04) to install and manage on your own hardware or virtual machines.

Account

This tab offers a self-explanatory Sign Out link, and a link to the Select Organization page, where you can traverse to other Organizations.

Account tabAccount tab

Account tab

Managing LogStream Cloud

Once you've registered on the portal, here's how to access LogStream Cloud:

  1. Sign in to your Cribl.Cloud portal page.
  2. Select the Organization to work with.
  3. From the protal's Overview tab, select Connect to your LogStream.
  4. The LogStream Cloud UI will open in a new tab or window ‚Äď ready to goat!

Note the Tenant ID link at the LogStream Cloud home page's upper left, under the Welcome! message. You can click this link to reopen the Cribl.Cloud portal page, to access Data Sources configurations. To return to this home page from anywhere else in LogStream Cloud's UI, click the LogStream logo in the upper-left corner.

Inviting and Managing Other Users

From the Organization > Members tab, an Organization's owner can invite new users to join the Organization, assign access roles to new and existing members, and remove pending invites and/or existing members.

Organization > Members tab: Managing Invites and MembersOrganization > Members tab: Managing Invites and Members

Organization > Members tab: Managing Invites and Members

Inviting Members

Click + Invite Member to open the modal shown below. Enter the Email address of the new user you want to invite, assign them a Role (explained just below), and then click Invite to send the invitation.

Invite User modalInvite User modal

Invite User modal

Member Roles

Each Role that you can assign to members confers a default Role within the Organization's LogStream instance. Here are the Roles, their corresponding permissions, and who can assign each:

Member RoleLogStream RoleOptions/Restrictions
AdminadminAny Organization owner can assign
Editoreditor_allAssignable only with Enterprise plan
Read-Onlyreader_allAssignable only with Enterprise plan
OwneradminCan't be assigned, but can manage Organization details

Note that:

  • Full role-based access control is available only with an Enterprise plan. (For¬†all available Enterprise features, see LogStream Pricing.)

  • Owners of non-Enterprise Cribl.Cloud Organizations will be able to assign only the Admin Role in the Invite¬†User modal shown above.

  • The¬†one Member Role that you cannot assign or transfer is your own Owner Role. A¬†user can acquire this superuser Role only by signing up as the owner of their own Cribl.Cloud Organization.

  • Only an Organization's Owner can manage the Organization's¬†details.

ūüďė

When you assign a Cribl.Cloud Member Role, it is mapped to a LogStream Role as described above. However, these users will not be visible as local users within the LogStream UI.

Responding to Invites

At the address you entered, the new member receives an email with an Accept Invitation link to either sign into their existing Cribl.Cloud account, or else sign up to create an account and its credentials.

After signing in, they'll have access to your Organization and LogStream instance at the Role level you've specified.

Managing Invites

While an invite is pending, the Organization > Members tab offers you these options to deal with commonly encountered issues:

  • Reinvite: If your invited member didn't receive your invitation email, you can click this button to resend it.

  • Copy Link: If emails aren't getting through at all, click this button to copy and share a URL that will take the invitee directly to the signup page. This target page encapsulates the same identity, Organization, and Role you specified in the original email invite.

  • Remove: This is for scenarios where you need to revoke a pending invite. (You sent someone a duplicate invite, your invitee is spending too much time in space to be a productive collaborator, etc.) After clicking this button, you'll see a confirmation dialog.

After 7 days, if an invite has been neither accepted nor revoked, it expires. In this case, it is removed from the Members tab.

Managing InvitesManaging Invites

Managing Invites

Managing Members

Once a user has accepted an invite, the Organization > Members tab offers you these options to modify their membership in your Organization:

  • Edit: Switch this member to a different Role. (The Edit option is displayed only if you have an Enterprise plan.)

  • Remove: Remove this member from your Organization. After clicking this button, you'll see a confirmation dialog. (Proceeding will not affect this user's access to any other Cribl.Cloud Organizations they might own or be members of.)

LogStream Cloud Pricing

Beyond the free tier, an optional paid LogStream Cloud account offers support from 8am‚Äď5pm, plus the ability to expand to 5 TB/day of throughput. In the Cribl.Cloud portal, select Request¬†Pricing to talk with Cribl about upgrading your free account.

You'll pay only for what you use ‚Äď the data you send to LogStream, and the data sent to external destinations. However, data sent to your AWS S3 storage is always free.

Data DirectionMonthly ChargeAnnual Charge
Price per GB sent in to LogStream$0.15/GB$0.125/GB
Price per GB sent out to external destinations$0.15/GB$0.125/GB

Example Pricing Scenario

Assume that you want to send 1,000 GB/day to LogStream. You reduce that data by 40% (a standard reduction that we see every day for customers). And you send the remaining 600 GB to an external destination:

(1,000 GB/day in + 600 GB/day out) x $0.15 (monthly rate per GB) = $240/day

Note that you can send a copy of all of your data to an S3 location of your choice, for no added data-egress charge.

Differences from LogStream Binaries

LogStream Cloud differs from a deployed LogStream binary in the following ways. Keep all these differences in mind as you navigate LogStream's current UI, in-app help (including tooltips), and documentation.

Simplified Administration

Compared to a LogStream binary that you deploy on your own infrastructure, LogStream Cloud's left nav, and particularly its Settings page, are much simpler.

LogStream Cloud's Settings left nav

Here are the key options streamlined out of the Cloud version.

Simplified Distributed Architecture

The Settings > Worker Processes and Settings > Distributed Settings links are omitted, and the left nav contains no Worker Groups or Mappings links. LogStream Cloud is configured like a distributed deployment with a single Worker Group. All Workers will share the same configuration.

Git Preconfigured

The Settings > Distributed Settings > Git Settings section is omitted. A local git client is preconfigured in your Cribl.Cloud portal. On LogStream Cloud's left nav, use the Changes link to commit/push changes to git. Select Deploy at the UI's top right to deploy your committed changes. LogStream Cloud does not support Git remote repos.

Automatic Restarts and Upgrades

The Settings > Controls and Settings > Upgrade links are omitted. Cribl handles restarts and version upgrades automatically on your behalf.

Simplified Access Management and Security

The Settings > Access Management section is omitted. All users of a given LogStream Cloud instance share a single admin login.

The Settings > Security section is omitted. Certificates are predefined for you on the Cribl.Cloud portal's Data Sources tab (see Available Ports and TLS Configurations below). LogStream Cloud does not support KMS secrets stores.

Other Simplified Settings

The Settings > Licensing link is omitted. Your license is managed by your parent Cribl.Cloud portal.

The Settings > Scripts link is omitted. LogStream Cloud does not support configuring or running shell scripts.

Support Options

The Settings > Diagnostics link is omitted. For help with any troubleshooting needs:

  • Click the Intercom link at LogStream's lower right.
  • Join Cribl's Community¬†Slack #logstream-cloud channel.
  • If you have a paid account, contact Cribl Support.

Available Ports and TLS Configurations

To get data into LogStream Cloud, your Cribl.Cloud portal provides several data sources and ports already enabled for you, plus 10 additional ports (20000-20010) that you can use to add and configure more LogStream Sources.

The¬†Cribl.Cloud portal's Data¬†Sources tab displays the pre‚ÄĎenabled Sources, their endpoints, reserved and available ports, and protocol details. For the existing Sources listed here, Cribl recommends using the preconfigured endpoint and port to send data into LogStream.

TLS encryption is enabled for you on several Sources, also indicated on the Cribl.Cloud portal's Data Sources tab. All TLS is terminated by the Network Load Balancer (NLB) sitting in front of the Workers.

Currently, LogStream Cloud does not enable you to import your own certificates for mutual TLS authentication. LogStream Cloud uses TLS to provide encryption in the wire,¬†but leaves authentication at the protocol layer ‚Äď e.g., Splunk HEC or S2S tokens, Kafka authorization, etc.

Available ports and TLS certificatesAvailable ports and TLS certificates

Available ports and TLS certificates

Simplified Source, Collector, and Destination Configuration

LogStream Cloud provides no Filesystem Source, Filesystem Collector, or Filesystem Destination. (A Cloud deployment has no local filesystem to read from or write to.)

Several commonly used Sources are preconfigured for you, within LogStream Cloud's UI, and ready to use.

ūüöß

In a preconfigured Source's configuration, never change the Address field, even though the UI shows an editable field. If you change these fields' value, the Source will not work as expected.

After you create a Source and deploy the changes, it can take a few minutes for the Source to become available in LogStream Cloud's load balancer. However, LogStream will open the port and be able to receive data immediately.

Do Not Enable TLS Within Sources

Several LogStream Cloud Sources' configuration modals include a TLS Settings (Server Side) tab, inherited from the LogStream binary's UI. Do not set this tab's Enabled slider to Yes, nor configure any of the resulting fields. Any settings that you configure will conflict with LogStream Cloud Sources' predefined TLS configurations.

Cloud FAQ

Here are some common questions and answers about LogStream Cloud's current state and anticipated evolution.

Are there egress costs to send data to another AWS service in the same Region as LogStream Cloud?

As of August/September 2021, there is a charge, because the data must pass through a public internet interface. Cribl is exploring ways to mitigate this using virtual networks.

Do Worker Nodes provide elastic growth as ingest and egress increase?

As of August/September 2021, it is important to size for anticipated usage. When you anticipate higher throughput, you must communicate this to Cribl, so we can provision additional Workers. Cribl is exploring automated provisioning.

Will LogStream Cloud support a hybrid model with both on-prem and cloud-based Worker Groups?

Later in 2021, Cribl anticipates separating the control plane from the data plane. You will then be able to manage all your LogStream Worker Groups (cloud and/or on-prem) centrally through the Cloud interface.

How can I send Amazon Kinesis Data Firehose (KDF) data to LogStream Cloud?

  1. If you use an AWS load balancer, use only a Classic Load Balancer. LogStream Cloud currently defaults to using AWS Network Load Balancers (NLBs) in front of each Cloud stack, and KDF is not compatible with NLBs or Application Load Balancers. (For details, see Amazon's Troubleshooting Amazon Kinesis Data Firehose documentation.)

  2. Also, enable duration-based sticky sessions with cookie expiration disabled. (For details, see Amazon's Duration-Based Session Stickiness documentation.)

As an alternative KDF, you can write to an S3 bucket and use LogStream Cloud's native SQS-based S3 Source. You can also use a Lambda function to define the source data, connecting it to a LogStream Cloud REST Collector with HTTP discovery.

Updated 6 days ago

Cribl.Cloud Deployment


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.