Add an AWS API Dataset Provider

​

A Dataset Provider tells Cribl Search where to query and contains access credentials. Here, you will add an AWS API Dataset Provider.

To add a new Dataset Provider, select Data, then Dataset Providers, then Add Provider.

If you see a drop-down showing your Stream Worker Groups and Data Lake Amazon S3 Destinations, ignore them and select Create to add a new provider.

Set the following configurations in the New Dataset Provider modal:

1. ID is a unique identifier for the Dataset Provider. This is how you’ll reference it when assigning Datasets to it. Start the ID with a letter; the rest of the ID can use letters, numbers, and underscores (for example, my_dataset_provider_1).
2. Description is optional.
3. Set Dataset Provider Type to AWS API.
4. Authentication method provides two options, Assume Role and AWS keys. See how to grant access to AWS for details on each option.
5. Select Add Configuration to specify your AWS account. The configuration depends on the Authentication method selected and you can use only one method for all configurations. In the Account Configurations table:
   • Assume Role requires the IAM role’s ARN (AssumeRole ARN) and has options to define an External ID and Duration.
     • The External ID on the Dataset Provider must match the external ID defined in the IAM Role Trust Policy.
     • Duration defines the Assumed Role’s session length of time, in seconds. Minimum is 900 (15 minutes), default is 3600 (1 hour), and maximum is 43200 (12 hours).
   • AWS keys requires the IAM user’s account Name, Access key, and Secret key.
6. Select Save when finished.

For details on obtaining your AWS credentials, see Grant Access to AWS.

Permission Requirements for AWS API

​

Accessing specific AWS endpoints requires the following permissions:

Add an AWS API Dataset

​

Now you’ll add a Dataset that tells Cribl Search what data to search from the Dataset Provider.

To add a new Dataset, select Data, then Datasets, then Add Dataset.

Set the following configurations in the New Dataset modal:

1. ID is an identifier unique for both Cribl Search and Cribl Lake. You’ll use this to specify the Dataset in a query’s scope, telling Cribl Search to search the Dataset. Start the ID with a letter; the rest of the ID can use letters, numbers, and underscores (for example, my_dataset_1).
2. Description is optional.
3. Set Dataset Provider to the ID of an AWS Dataset Provider.
4. Under Enabled endpoint, select Add Endpoints to select the endpoints for your Dataset. Select an endpoint from the drop-down menu. Your options are:
   • ec2_instances
   • ec2_volumes
   • ec2_security_groups
   • lambda_functions
   • iam_users
   • iam_roles
   • iam_groups
   • iam_policies
   • iam_mfa_devices
   • cloudformation_stacks
   • cloudformation_stacksets
   • cloudformation_exports
   • dynamodb_backups
   • rds_exports
   • rds_backups
   • rds_clusters
   • rds_cluster_endpoints
   • rds_instances
   • rds_security_groups
   • rds_certificates
   • cloudtrail_events
   • vpc_subnets
   • vpc_network_interfaces
   • efs_file_systems
5. Under AWS Regions, select Add Regions to specify the AWS regions to query for the endpoint(s).
6. In Processing, you can apply rules for breaking data into discrete events. For more information, see Datatypes.
7. In Snapshots, you can set up API Snapshots.
8. Select Save when finished.

Search AWS API

​

Now that you have a Dataset Provider and Dataset, you’re ready to start searching.

Search results can start showing up within a second or two, but when the search completes depends on how much data there is in the account.