Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up here)
Download entire manual as PDF - v2.4.4

Kinesis

Cribl LogStream supports receiving data records from Amazon Kinesis Streams.

📘

Type: Pull | TLS Support: YES (secure API) | Event Breaker Support: No

Configuring Cribl LogStream to Receive Data from Kinesis Streams

Select Data > Sources, then select Kinesis from the Data Sources page's tiles or left menu. Click Add New to open the Kinesis > New Source modal, which provides the following fields.

General Settings

Input ID: Enter a unique name to identify this Kinesis Stream Source definition.

Stream name: Kinesis stream name (not ARN) to read data from.

Shard iterator start: Location at which to start reading a shard for the first time. Defaults to Earliest Record.

Record data format: Format of data inside the Kinesis Stream records. Gzip compression is automatically detected. Options include:

  • Cribl (the default): Use this option if LogStream wrote data to Kinesis in this format. This is a type of NDJSON.
  • Newline JSON: Use if the records contain newline-delimited JSON (NDJSON) events – e.g., Kubernetes logs ingested through Kinesis. This is a good choice if you don't know the records' format.
  • CloudWatch Logs: Use if you've configured CloudWatch to send logs to Kinesis.
  • Event per line: NDJSON can use this format when it fails to parse lines as valid JSON.

Region: Region where the Kinesis stream is located. Required.

Authentication

Use the Authentication Method buttons to select an AWS authentication method:

  • Auto: This default option uses the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or the attached IAM role. Works only when running on AWS.

  • Manual: You must select this option when not running on AWS.

Auto Authentication

When using an IAM role to authenticate with Kinesis Streams, the IAM policy statements must include the following Actions:

  • kinesis:GetRecords
  • kinesis:GetShardIterator
  • kinesis:ListShards

For details, see AWS' Actions, Resources, and Condition Keys for Amazon Kinesis documentation.

Manual Authentication

The Manual option exposes these additional fields:

Access key: Enter your AWS access key. If not present, will fall back to env.AWS_ACCESS_KEY_ID, or to the metadata endpoint for IAM role credentials.

Secret key: Enter your AWS secret key. If not present, will fall back to env.AWS_SECRET_ACCESS_KEY, or to the metadata endpoint for IAM credentials.

Assume Role

Enable for Kinesis Streams: Whether to use Assume Role credentials to access Kinesis Streams. Defaults to No.

AssumeRole ARN: Enter the Amazon Resource Name (ARN) of the role to assume.

External ID: Enter the External ID to use when assuming role.

Processing Settings

Fields (Metadata)

In this section, you can add fields/metadata to each event, using Eval-like functionality.

  • Name: Field name.
  • Value: JavaScript expression to compute field's value (can be a constant).

Pre-Processing

In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.

Advanced Settings

Shard selection expression: A JavaScript expression to be called with each shardId for the stream. The shard will be processed if the expression evaluates to a truthy value. Defaults to true.

Service Period: Time interval (in minutes) between consecutive service calls. Defaults to 1 minute.

Endpoint: Kinesis stream service endpoint. If empty, the endpoint will be automatically constructed from the region.

Signature version: Signature version to use for signing Kinesis Stream requests. Defaults to v4.

Verify KPL checksums: Enable this setting to verify Kinesis Producer Library (KPL) event checksums.

Reuse connections: Whether to reuse connections between requests. The default setting (Yes) can improve performance.

Reject unauthorized certificates: Whether to accept certificates that cannot be verified against a valid Certificate Authority (e.g., self-signed certificates). Defaults to Yes.

Internal Fields

Cribl LogStream uses a set of internal fields to assist in handling of data. These "meta" fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

Field for this Source:

  • __inputId

Updated 2 days ago

Kinesis


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.