Cribl Stream can send metric events directly to Amazon Managed Service for Prometheus (AMP) using the Prometheus remote write protocol, authenticated with AWS Signature Version 4 (SigV4).

This destination is a dedicated tile for AMP, distinct from the general Prometheus destination. It enforces HTTPS delivery to the AMP endpoint, automatically injects the required X-Prometheus-Remote-Write-Version header, and manages all SigV4 request-signing headers. The region you specify is validated against the URL you provide.

Type: Streaming | TLS Support: Yes (required) | PQ Support: Yes

IAM Permissions ​

The IAM identity used by Cribl Stream (whether an EC2 instance role, ECS task role, or a static credential pair) must have the aps:RemoteWrite permission on the target AMP workspace.

If you use Assume Role, the IAM role being assumed must also have aps:RemoteWrite, and the Cribl Stream identity must have sts:AssumeRole permission on that role.

Configure Cribl Stream to Output to Amazon AMSP ​

1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
   • To configure via QuickConnect, navigate to Routing > QuickConnect. Select Add Destination, select Amazon AMSP from the list, and choose either Select Existing or Add New.
   • To configure via the Routes, select Data > Destinations. Under the Amazon group, select Amazon AMSP, then select Add Destination.
2. In the New Destination modal, configure the following under General Settings:
   • Output ID: Enter a unique name to identify this Amazon AMSP output definition. If you clone this Destination, Cribl Stream will add -CLONE to the original Output ID.
   • Description: Optionally, enter a description.
   • Remote Write URL: The AMP workspace remote write endpoint. Copy this URL from the AWS console on the AMP workspace page. The URL must use HTTPS and follow the form https://aps-workspaces.<REGION>.amazonaws.com/workspaces/<WORKSPACE_ID>/api/v1/remote_write. FIPS endpoints (aps-workspaces-fips.<REGION>.amazonaws.com) and dual-stack endpoints (aps-workspaces.<REGION>.api.aws) are also accepted. The region in the URL must match the Region field below.
   • Region: The AWS region where your AMP workspace is located, for example us-east-1. This value must match the region embedded in the Remote Write URL.
3. Next, you can configure the following Optional Settings:
   • Backpressure behavior: Whether to block, drop, or queue events when all receivers are exerting backpressure.
   • Tags: Optionally, add tags that you can use to filter and group Destinations on the Destinations page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.
4. Optionally, you can adjust the Authentication, Assume Role, Persistent Queue, Processing, Retries, and Advanced settings outlined in the sections below.
5. Select Save, then Commit & Deploy.

Authentication ​

Use the Authentication method drop-down to select how Cribl Stream obtains AWS credentials for SigV4 signing.

Auto: This default option uses the AWS SDK for JavaScript to automatically obtain credentials in the following order of attempts:

1. Environment Variables: Loaded from environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
2. IAM Identity Center (SSO): Loaded from AWS IAM Identity Center configuration.
3. Shared Credentials File: Loaded from the shared credentials file (~/.aws/credentials).
4. IAM Roles for Amazon EC2/ECS: Loaded from AWS Identity and Access Management (IAM) roles attached to an EC2 instance or ECS container.
5. JSON File on Disk: Loaded from a JSON file on disk.
6. Other Credential-Provider Classes: Other credential-provider classes provided by the AWS SDK for JavaScript.

The Auto method works both when running on AWS and in other environments where the necessary credentials are available through one of the above methods.

SSO Providers ​

When using the auto authentication method, you can leverage SSO providers like SAML and Okta to issue temporary credentials. These credentials should be set in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK will then use these environment variables to authenticate.

Secret Key pair: Select this option to supply a stored secret that references an AWS access key and secret key. The Secret Key pair option exposes this field:

• Secret key pair: Use the drop-down to select an API key/secret key pair that you’ve configured in Cribl Stream’s secrets manager. A Create link is available to store a new, reusable secret.

Assume Role ​

When using Assume Role to access resources in a different Region than Cribl Stream, you can target the AWS Security Token Service (STS) endpoint specific to that Region by using the CRIBL_AWS_STS_REGION environment variable on your Worker Node. To specify a custom STS endpoint, set the CRIBL_AWS_STS_ENDPOINT environment variable. If both are set, CRIBL_AWS_STS_ENDPOINT takes precedence. Setting an invalid Region or endpoint results in a fallback to the global STS endpoint.

Enable: Toggle on to use Assume Role credentials to access the AMP workspace.

AssumeRole ARN: Enter the Amazon Resource Name (ARN) of the role to assume.

External ID: Enter the External ID to use when assuming role.

Duration (seconds): Duration of the Assumed Role’s session, in seconds. Minimum is 900 (15 minutes). Maximum is 43200 (12 hours). Defaults to 3600 (1 hour).

Persistent Queue Settings ​

Mode: Use this menu to select when Cribl Stream engages the persistent queue in response to backpressure events from this Destination. The options are:

Backpressure duration limit: Displays when Mode is set to Backpressure. Controls how long a Destination must experience backpressure before data is queued to persistent storage. Defaults to 30 seconds. Lowering this limit strengthens data loss prevention; raising it prevents unnecessary queue activation in environments with frequent, short network interruptions.

Processing Settings ​

Post-Processing ​

Pipeline: Pipeline or Pack to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output. By default, includes cribl_host (Cribl Stream Node that processed the event) and cribl_wp (Cribl Stream Worker Process that processed the event). Supports wildcards. Other options include:

• cribl_input - Cribl Stream Source that processed the event.
• cribl_output - Cribl Stream Destination that processed the event.
• cribl_pipe - Cribl Stream Pipeline that processed the event.
• cribl_route - Cribl Stream Route (or QuickConnect) that processed the event.
• cribl_group - Cribl Stream Worker Group of the node that processed the event.
• cribl_mode - Cribl Stream mode of the node that processed the event.

Retries ​

Honor Retry-After header: Toggle on to honor a Retry-After header, provided that the header specifies a delay no longer than 180 seconds. Cribl Stream/Edge limits the delay to 180 seconds even if the Retry-After header specifies a longer delay. Any Retry-After header received takes precedence over all other options configured in the Retries section. Toggle off to ignore all Retry-After headers.

Default Retry Behavior: By default, Cribl Stream/Edge applies predefined retry strategies for certain failed requests to the Destination using an exponential backoff algorithm. You can override these settings by manually configuring the retry behavior for a particular error code.

Destinations created before the 4.16.0 release do not support the default 4xx series retry behavior. If your Destination was created prior to this release and you’d like it to support the behavior, reconfigure the Destination.

To retry requests, Cribl Stream/Edge applies the following rules by default:

By default, any 429 (Too Many Requests) response that is not listed in the retry settings is treated as a non-retryable 4xx error and dropped. If you want 429 responses retried, ensure 429 is included in the Settings for failed HTTP requests.

Settings for failed HTTP requests: You can also manually configure Cribl Stream/Edge to automatically retry requests that receive particular HTTP response status codes.

Upon receiving a response code that’s on the list, Cribl Stream/Edge first waits for a set time interval called the Pre-backoff interval and then begins retrying the request. Time between retries increases based on an exponential backoff algorithm whose base is the Backoff multiplier, until the backoff multiplier reaches the Backoff limit (ms). At that point, Cribl Stream/Edge continues retrying the request without increasing the time between retries any further.

If the sender (which manages the connection to the Destination) is at capacity, it will not accept any incoming events. These incoming events originate internally from a previous stage of the data flow when Destinations send outbound requests to their respective external services, and they include retry requests and new requests. Any events that were already in transit when the sender reached capacity will continue to be processed downstream.

Sender capacity is freed up when an outgoing request succeeds or encounters a non-retryable error. When the sender has available capacity again, it will resume accepting incoming events. This capacity management is influenced by the number of active connections and configured limits, such as concurrency and buffer sizes. If a Pipeline sends events faster than the Destination can process, the buffers may fill up, leading to backpressure and Sender at capacity warnings. This backpressure prevents the sender from accepting additional requests until capacity is restored.

For each response code you want to add to the list, select Add Setting and configure the following settings:

• HTTP status code: A response code that indicates a failed request, for example 429 (Too Many Requests) or 503 (Service Unavailable).
• Pre-backoff interval (ms): The amount of time to wait before beginning retries, in milliseconds. Defaults to 1000 (one second).
• Backoff multiplier: The base for the exponential backoff algorithm. A value of 2 (the default) means that Cribl Stream/Edge will retry after 2 seconds, then 4 seconds, then 8 seconds, and so on.
• Backoff limit (ms): The maximum backoff interval Cribl Stream/Edge should apply for its final retry, in milliseconds. Default (and minimum) is 10,000 (10 seconds); maximum is 180,000 (180 seconds, or 3 minutes).

Retry timed-out HTTP requests: Toggle on to automatically retry requests that have timed out and display the following settings for configuring retry behavior:

• Pre-backoff interval (ms): The amount of time to wait before beginning retries, in milliseconds. Defaults to 1000 (one second).
• Backoff multiplier: The base for the exponential backoff algorithm. A value of 2 (the default) means that Cribl Stream/Edge will retry after 2 seconds, then 4 seconds, then 8 seconds, and so on.
• Backoff limit (ms): The maximum backoff interval Cribl Stream/Edge should apply for its final retry, in milliseconds. Default (and minimum) is 10,000 (10 seconds); maximum is 180,000 (180 seconds, or 3 minutes).

Advanced Settings ​

Validate server certs: Reject certificates that are not authorized by a trusted CA (for example, the system’s CA). Defaults to toggled on.

Round-robin DNS: Toggle on to enable round-robin DNS lookup across multiple IP addresses, IPv4 and IPv6. When a DNS server resolves a Fully Qualified Domain Name (FQDN) to multiple IP addresses, Cribl Stream will sequentially use each address in the order they are returned by the DNS server for subsequent connection attempts.

Request timeout: Amount of time (in seconds) to wait for a request to complete before aborting it. Defaults to 30.

Request concurrency: Maximum number of concurrent requests per Worker Process. When Cribl Stream hits this limit, it begins throttling traffic to the downstream service. Defaults to 5. Minimum: 1. Maximum: 32.

Body size limit (KB): Maximum size of the request body before compression. Defaults to 1024 KB, and cannot exceed 1024 KB. AMP rejects remote write requests larger than 1 MB with a non-retryable 413 error.

Events-per-request limit: Maximum number of events to include in the request body. The 0 default allows unlimited events.

Flush period (sec): Maximum time between requests. Low values could cause the payload size to be smaller than its configured maximum. Defaults to 1.

Extra HTTP headers: Select Add Header to add Name/Value pairs to pass as additional HTTP headers. Values will be sent encrypted. Note that SigV4 signing headers (Authorization, Host, X-Amz-Date, X-Amz-Content-Sha256, X-Amz-Security-Token) and X-Prometheus-Remote-Write-Version are managed automatically; do not set these here.

Metric renaming expression: A JavaScript expression that can be used to rename metrics. The default expression - name.replace(/[^a-zA-Z0-9_]/g, '_') - replaces all characters that are not letters, digits, or underscores with the Prometheus-supported _ character. Use the name global variable to access the metric’s name. You can access event fields’ values via __e.<fieldName>.

Use _bucket suffix for histogram buckets: Toggle on to serialize histogram bucket series as <metric>_bucket, matching the Prometheus histogram naming convention. This setting affects only the bucket series; Cribl Stream always appends the _sum and _count suffixes to histogram and summary series regardless of this setting. Defaults to toggled on for new Destinations.

Send metadata: Whether to generate and send metrics’ metadata (type and metricFamilyName) along with the metrics. Defaults to toggled on. Toggling off hides this additional field:

• Metadata flush period (sec): How frequently metrics metadata is sent out. Value must at least equal the base Flush period (sec). Defaults to 60 seconds.

Failed request logging mode: Use this drop-down to determine which data should be logged when a request fails. Select among None (the default), Payload, or Payload + Headers. With this last option, Cribl Stream will redact all headers, except non-sensitive headers that you declare below in Safe headers.

Safe headers: Add headers to declare them as safe to log in plaintext. (Sensitive headers such as authorization will always be redacted, even if listed here.) Use a tab or hard return to separate header names.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Internal Fields ​

Cribl Stream uses a set of internal fields to assist in forwarding data to a Destination.

If an event contains the internal field __criblMetrics, Cribl Stream will send it to the AMP endpoint as a metric event. Otherwise, Cribl Stream will drop the event.

Notes on HTTP-Based Outputs ​

• To proxy outbound HTTP/S requests, see System Proxy Configuration.

• The AMSP destination always uses HTTPS and enforces that the Remote Write URL is an aps-workspaces endpoint.

• The SigV4 headers (Authorization, Host, X-Amz-Date, X-Amz-Content-Sha256, X-Amz-Security-Token) and the X-Prometheus-Remote-Write-Version header are managed automatically. Do not set these manually in Extra HTTP headers or they will conflict with the signing process.

• Like the Prometheus destination, all payloads are protobuf-encoded and snappy-compressed per the Prometheus remote_write specification. There is no separate compression setting.

• Cribl Stream uses HTTP keepalives to reuse a connection for multiple requests. If the remote side closes the connection or a request fails, Cribl Stream establishes a new connection for subsequent requests.

Troubleshooting ​

The Destination’s configuration modal has helpful tabs for troubleshooting:

Live Data: Try capturing live data to see real-time events as they flow through the Destination. On the Live Data tab, click Start Capture to begin viewing real-time data.

Logs: Review and search the logs that provide detailed information about the delivery process, including any errors or warnings that may have occurred.

Test: Ensures that the Destination is correctly set up and reachable. Verify that sample events are sent correctly by clicking Run Test.

You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify delivery issues. Analyze the graphs showing events and bytes in/out over time.