Skip to main content
Version: 3.2

Amazon SQS

Cribl LogStream supports receiving events from Amazon Simple Queuing Service.

Type: Pull | TLS Support: YES (secure API) | Event Breaker Support: No

Configuring Cribl LogStream to Receive Data from Amazon SQS

In the QuickConnect UI: Click + New Source, or click + Add beside Sources. From the resulting drawer's tiles, select [Pull >] Amazon > SQS. Next, click either + Add New or (if displayed) Select Existing. The drawer will now provide the following options and fields.

Or, in the Data Routes UI: From the top nav of a LogStream instance or Group, select Data > Sources. From the resulting page's tiles or the Sources left nav, select [Pull >] Amazon > SQS. Next, click + Add New to open a New Source modal that provides the following options and fields.

General Settings

Input ID: Enter a unique name to identify this SQS Source definition.

Queue: The name, URL, or ARN of the SQS queue to read events from. This value must be a JavaScript expression (which can evaluate to a constant), enclosed in single quotes, double quotes, or backticks. To specify a non-AWS URL, use the format: '{url}/<queueName>'. (E.g., ':port/<myQueueName>'.)

Queue type: The queue type used (or created). Defaults to Standard. FIFO (First In, First Out) is the other option.

Create queue: If toggled to Yes, LogStream will create the queue if it does not exist.

Region: AWS Region where the SQS queue is located. Required, unless the Queue entry is a URL or ARN that includes a Region.

Authentication

Use the Authentication Method buttons to select an AWS authentication method.

Auto

This default option uses the AWS instance's metadata service to automatically obtain short-lived credentials from the IAM role attached to an EC2 instance. The attached IAM role grants LogStream Workers access to authorized AWS resources. Can also use the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Works only when running on AWS.

Manual

If not running on AWS, you can select this option to enter a static set of user-associated IAM credentials (your access key and secret key) directly or by reference. This is useful for Workers not in an AWS VPC, e.g., those running a private cloud. The Manual option exposes these corresponding additional fields:

  • Access key: Enter your AWS access key. If not present, will fall back to the env.AWS_ACCESS_KEY_ID environment variable, or to the metadata endpoint for IAM role credentials.

  • Secret key: Enter your AWS secret key. If not present, will fall back to the env.AWS_SECRET_ACCESS_KEY environment variable, or to the metadata endpoint for IAM credentials.

Secret

If not running on AWS, you can select this option to supply a stored secret that references an AWS access key and secret key. The Secret option exposes this additional field:

  • Secret key pair: Use the drop-down to select a secret key pair that you've configured in LogStream's internal secrets manager or (if enabled) an external KMS. Follow the Create link if you need to configure a key pair.

Assume Role

Enable for SQS: Whether to use Assume Role credentials to access SQS. Defaults to No.

AWS account ID: SQS queue owner's AWS account ID. Leave empty if SQS queue is in same AWS account.

AssumeRole ARN: Enter the Amazon Resource Name (ARN) of the role to assume.

External ID: Enter the external ID to use when assuming role.

Processing Settings

Fields (Metadata)

In this section, you can add fields/metadata to each event, using Eval-like functionality.

Name: Field name.

Value: JavaScript expression to compute field's value (can be a constant).

Pre-Processing

In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.

Advanced Settings

Endpoint: SQS service endpoint. If empty, the endpoint will be automatically constructed from the AWS Region.

Signature version: Signature version to use for signing SQS requests. Defaults to v4.

Max messages: The maximum number of messages that SQS should return in a poll request. Amazon SQS never returns more messages than this value. (However, fewer messages might be returned.) Acceptable values: 1 to 10. Defaults to 10.

Visibility timeout seconds: The duration (in seconds) that the received messages are hidden from subsequent retrieve requests, after they're retrieved by a ReceiveMessage request. Defaults to 600.

Num receivers: The number of receiver processes to run. The higher the number, the better the throughput, at the expense of CPU overhead. Defaults to 3.

Reuse connections: Whether to reuse connections between requests. The default setting (Yes) can improve performance.

Reject unauthorized certificates: Whether to accept certificates that cannot be verified against a valid Certificate Authority (e.g., self-signed certificates). Defaults to Yes.

Environment: If you're using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Connected Destinations

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source's data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

Internal Fields

Cribl LogStream uses a set of internal fields to assist in handling of data. These "meta" fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

Fields for this Source:

  • __inputId
  • __sqsSysAttrs

The _sqsSysAttrs field can take on the following properties, which are reported to LogStream from SQS:

  • __sqsSysAttrs.ApproximateFirstReceiveTimestamp: Returns the time (epoch time in milliseconds) the message was first received from the queue.
  • __sqsSysAttrs.ApproximateReceiveCount: Returns the number of times a message has been received from the queue without being deleted.
  • __sqsSysAttrs.SenderId: For an IAM user, returns the IAM user ID (e.g.: ABCDEFGHI1JKLMNOPQ23R). For an IAM role, returns the IAM role ID (e.g.: ABCDE1F2GH3I4JK5LMNOP:i-a123b456).
  • __sqsSysAttrs.SentTimestamp: Returns the time (epoch time in milliseconds) the message was sent to the queue.
  • __sqsSysAttrs.MessageDeduplicationId: Returns the value provided by the producer that calls the SendMessage action.
  • __sqsSysAttrs.MessageGroupId: Returns the value provided by the producer that calls the SendMessage action – messages with the same MessageGroupId are returned in sequence.
  • __sqsSysAttrs.SequenceNumber: Returns the sequence-number value provided by Amazon SQS.
  • __sqsSysAttrs.AWSTraceHeader: Returns the AWS X‑Ray trace header string.

For background on these message properties, see AWS' ReceiveMessage > Request Parameters documentation.

SQS Permissions

The following permissions are needed on the SQS queue:

  • sqs:ReceiveMessage
  • sqs:DeleteMessage
  • sqs:GetQueueAttributes
  • sqs:GetQueueUrl
  • sqs:CreateQueue (optional, if and only if you want LogStream to create the queue)

Troubleshooting Notes

VPC endpoints for SQS might need to be set up in your account. Check with your administrator for details.

How LogStream Pulls Data

Workers poll messages from SQS. The call will return a message if one is available, or will time out after 1 second if no messages are available.

Each Worker gets its share of the load from SQS, and it receives a notification of a file newly added to an S3 bucket. By default, SQS returns a maximum of 10 messages in a single poll request.