Cribl LogStream supports receiving data records from Amazon Kinesis Streams.
Type: Pull | TLS Support: YES (secure API) | Event Breaker Support: No
Select Data > Sources, then select Kinesis from the Data Sources page's tiles or left menu. Click Add New to open the Kinesis > New Source modal, which provides the following fields.
Input ID: Enter a unique name to identify this Kinesis Stream Source definition.
Stream name: Kinesis stream name (not ARN) to read data from.
Shard iterator start: Location at which to start reading a shard for the first time. Defaults to
Record data format: Format of data inside the Kinesis Stream records. Gzip compression is automatically detected. Options include:
- Cribl (the default): Use this option if LogStream wrote data to Kinesis in this format. This is a type of NDJSON.
- Newline JSON: Use if the records contain newline-delimited JSON (NDJSON) events – e.g., Kubernetes logs ingested through Kinesis. This is a good choice if you don't know the records' format.
- CloudWatch Logs: Use if you've configured CloudWatch to send logs to Kinesis.
- Event per line: NDJSON can use this format when it fails to parse lines as valid JSON.
Region: Region where the Kinesis stream is located. Required.
Use the Authentication Method buttons to select an AWS authentication method:
Auto: This default option uses the environment variables
AWS_SECRET_ACCESS_KEY, or the attached IAM role. Works only when running on AWS.
Manual: You must select this option when not running on AWS.
When using an IAM role to authenticate with Kinesis Streams, the IAM policy statements must include the following Actions:
For details, see AWS' Actions, Resources, and Condition Keys for Amazon Kinesis documentation.
The Manual option exposes these additional fields:
Access key: Enter your AWS access key. If not present, will fall back to
env.AWS_ACCESS_KEY_ID, or to the metadata endpoint for IAM role credentials.
Secret key: Enter your AWS secret key. If not present, will fall back to
env.AWS_SECRET_ACCESS_KEY, or to the metadata endpoint for IAM credentials.
Enable for Kinesis Streams: Whether to use Assume Role credentials to access Kinesis Streams. Defaults to
AssumeRole ARN: Enter the Amazon Resource Name (ARN) of the role to assume.
External ID: Enter the External ID to use when assuming role.
In this section, you can add fields/metadata to each event, using Eval-like functionality.
- Name: Field name.
In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.
shardId for the stream. The shard will be processed if the expression evaluates to a truthy value. Defaults to
Service Period: Time interval (in minutes) between consecutive service calls. Defaults to
Endpoint: Kinesis stream service endpoint. If empty, the endpoint will be automatically constructed from the region.
Signature version: Signature version to use for signing Kinesis Stream requests. Defaults to
Verify KPL checksums: Enable this setting to verify Kinesis Producer Library (KPL) event checksums.
Reuse connections: Whether to reuse connections between requests. The default setting (
Yes) can improve performance.
Reject unauthorized certificates: Whether to accept certificates that cannot be verified against a valid Certificate Authority (e.g., self-signed certificates). Defaults to
Cribl LogStream uses a set of internal fields to assist in handling of data. These "meta" fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.
Field for this Source:
Updated 2 days ago