On This Page

Home / Search/ Troubleshoot/Troubleshoot Federated Search

Troubleshoot Federated Search ​

Diagnose and resolve common issues when you search federated Datasets.

This page covers federated search, where Cribl Search queries data in place, in your own storage, through a Dataset Provider. It lists issues you might encounter, and recommends how to diagnose and resolve them.

Federated Search Troubleshooting Tree ​

If you get unexpected results from a search against an object store or data lake, start with this troubleshooting path:

  1. Are you retrieving objects? (Check this using the .show objects command.)
  2. If you’ve retrieved no objects, check bucket path(s), Search Permissions to access the Dataset and Dataset Provider, and access permissions on the object store.
  3. If you are retrieving objects, search for raw events with no time filtering.
  4. If you find no events, check the data format and permissions.
  5. If you find events, check your original time filtering and time parsing - is time recognized accurately? Also check v1 Datatypes that contain Event Breakers.

Amazon Connectivity Issues ​

This section covers issues specific to Amazon Datasets.

S3 Returns No Elements ​

You might find no elements returned in response to your query.

Possible Cause ​

Check your Search Permission to access the Dataset and Dataset Provider. Also check Datatypes and Event Breakers.

Recommendations ​

Check contents (ListBuckets in S3) to see, and adjust, the Cribl user’s or role’s permissions on the bucket.

KMS Authorization Fails on S3 ​

If you are using a custom KMS (key management system), you might find that Cribl Search cannot authorize on S3.

Possible Cause ​

Cribl Search cannot access your keys.

Recommendations ​

Include AssumeRole permissions in the KMS console and in the trust policy, to allow Cribl Search to access the keys.

KMS Decrypt Errors on S3 ​

Retrieving results from objects in an S3 bucket fail, and internal logs show errors of the form:
User is not authorized to perform kms:Decrypt on resource.

Possible Cause ​

Your IAM role is missing permissions required to access S3 objects that are encrypted with a KMS key.

Recommendations ​

Add this sub-element to the applicable IAM role:

 "Action": [
     "kms:Decrypt",
     "kms:Describe*",
     "kms:Get*",
     "kms:List*",
     "kms:GenerateDataKey"
 ],

Existing AssumeRole Fails to Authorize on S3 ​

You might find that your existing AssumeRole from Cribl Stream/Edge fails to authorize Cribl Search on S3.

Possible Cause ​

Cribl Search requires a different Principal element.

Recommendations ​

Add the required Principal to your trust policy. (Follow the link to see how to copy this Principal out of your Dataset Provider modal.)

Can’t Add Buckets to Amazon Security Lake Dataset ​

You might find that you cannot add new buckets to your Dataset.

Possible Cause ​

Insufficient permissions on Amazon Security Lake.

Recommendations ​

Check whether you have added the securitylake:ListDataLakes permission to your AssumeRole.

Performance Fixes ​

This section covers workarounds for slow, inefficient, or failing searches.

Scheduled Search Jobs Canceled Due to Queue Length Limit ​

You might find that some scheduled search jobs were canceled because the search queue reached its configured limit.

Possible Cause ​

By default, the search queue can hold five times the maximum number of concurrent searches. When the queue is full, Cribl Search rejects additional jobs.

Recommendations ​

To check the state of the queue, enter .show queue stats in the query box.

Contact Cribl Support or your Cribl account team to adjust the queue limit for your workload.

Feature Workarounds ​

This section covers substitutes for features that Cribl Search doesn’t provide out-of-the-box.

Get the Length of an Array Field ​

Cribl Search currently has no native function to return the length of an array’s field, without stringifying the value.

Recommendations ​

  1. Strip everything from the array except for commas; apply the strlen() function; and then add 1 to the result.

  2. Try a query of the form: print <arrayName> = dynamic([1,2,3]) | extend len = <arrayName>.length