Cribl LogStream supports receiving events from Amazon Simple Queuing Service.
Type: Pull | TLS Support: YES (secure API) | Event Breaker Support: No
Select Data > Sources, then select SQS from the Data Sources page's tiles or left menu. Click Add New to open the SQS > New Source modal, which provides the following fields.
Input ID: Enter a unique name to identify this SQS Source definition.
Queue type: The queue type used (or created). Defaults to
FIFO (First In, First Out) is the other option.
Create queue: If toggled to
Yes, LogStream will create the queue if it does not exist.
Region: AWS Region where the SQS queue is located. Required, unless the Queue entry is a URL or ARN that includes a Region.
Authentication Method: Select an AWS authentication method.
Auto: This default option uses the environment variables
AWS_SECRET_ACCESS_KEY, or the attached IAM role. Works only when running on AWS.
Manual: You must select this option when not running on AWS.
The Manual option exposes these corresponding additional fields:
Access key: Enter your AWS access key. If not present, will fall back to
env.AWS_ACCESS_KEY_ID, or to the metadata endpoint for IAM role credentials.re
Secret key: Enter your AWS secret key. If not present, will fall back to
env.AWS_SECRET_ACCESS_KEY, or to the metadata endpoint for IAM credentials.
Enable for SQS: Whether to use Assume Role credentials to access SQS. Defaults to
AWS account ID: SQS queue owner's AWS account ID. Leave empty if SQS queue is in same AWS account.
AssumeRole ARN: Enter the Amazon Resource Name (ARN) of the role to assume.
External ID: Enter the external ID to use when assuming role.
In this section, you can add fields/metadata to each event, using Eval-like functionality.
Name: Field name.
In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.
Endpoint: SQS service endpoint. If empty, the endpoint will be automatically constructed from the AWS Region.
Signature version: Signature version to use for signing SQS requests. Defaults to
Max messages: The maximum number of messages that SQS should return in a poll request. Amazon SQS never returns more messages than this value. (However, fewer messages might be returned.) Acceptable values: 1 to 10. Defaults to
Visibility timeout seconds: The duration (in seconds) that the received messages are hidden from subsequent retrieve requests, after they're retrieved by a
ReceiveMessage request. Defaults to
Num receivers: The number of receiver processes to run. The higher the number, the better the throughput, at the expense of CPU overhead. Defaults to
Reuse connections: Whether to reuse connections between requests. The default setting (
Yes) can improve performance.
Reject unauthorized certificates: Whether to accept certificates that cannot be verified against a valid Certificate Authority (e.g., self-signed certificates). Defaults to
Cribl LogStream uses a set of internal fields to assist in handling of data. These "meta" fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.
Fields for this Source:
_sqsSysAttrs field can take on the following properties, which are reported to LogStream from SQS:
__sqsSysAttrs.ApproximateFirstReceiveTimestamp: Returns the time (epoch time in milliseconds) the message was first received from the queue.
__sqsSysAttrs.ApproximateReceiveCount: Returns the number of times a message has been received from the queue without being deleted.
__sqsSysAttrs.SenderId: For an IAM user, returns the IAM user ID (e.g.:
ABCDEFGHI1JKLMNOPQ23R). For an IAM role, returns the IAM role ID (e.g.:
__sqsSysAttrs.SentTimestamp: Returns the time (epoch time in milliseconds) the message was sent to the queue.
__sqsSysAttrs.MessageDeduplicationId: Returns the value provided by the producer that calls the
__sqsSysAttrs.MessageGroupId: Returns the value provided by the producer that calls the
SendMessageaction – messages with the same
MessageGroupIdare returned in sequence.
__sqsSysAttrs.SequenceNumber: Returns the sequence-number value provided by Amazon SQS.
__sqsSysAttrs.AWSTraceHeader: Returns the AWS X‑Ray trace header string.
For background on these message properties, see AWS' ReceiveMessage > Request Parameters documentation.
The following permissions are needed on the SQS queue:
sqs:CreateQueue(optional, if and only if you want LogStream to create the queue)
Updated 2 days ago