Diagnose and resolve common errors in working with Cribl Search.

This page lists errors and issues that you might commonly encounter when working with Cribl Search, and provides recommendations for diagnosing and resolving the problems.

General Troubleshooting Tree ​

If you are getting unexpected results from a search against an object store or data lake, start by following this general troubleshooting path:

1. Are you retrieving objects? (Check this using the .show objects command.)
2. If you’ve retrieved no objects, check bucket path(s), Search Permissions to access the Dataset and Dataset Provider, and access permissions on the object store.
3. If you are retrieving objects, search for raw events with no time filtering.
4. If you find no events, check the data format and permissions.
5. If you find events, check your original time filtering and time parsing - is time recognized accurately? Also check v1 Datatypes that contain Event Breakers.

Amazon Connectivity Issues ​

This section covers issues specific to Amazon Datasets.

S3 Returns No Elements ​

You might find no elements returned in response to your query.

Possible Cause ​

Check your Search Permission to access the Dataset and Dataset Provider. Also check Datatypes and Event Breakers.

Recommendations ​

Check contents (ListBuckets in S3) to see, and adjust, the Cribl user’s or role’s permissions on the bucket.

KMS Authorization Fails on S3 ​

If you are using a custom KMS (key management system), you might find that Cribl Search cannot authorize on S3.

Possible Cause ​

Cribl Search cannot access your keys.

Recommendations ​

Include AssumeRole permissions in the KMS console and in the trust policy, to allow Cribl Search to access the keys.

KMS Decrypt Errors on S3 ​

Retrieving results from objects in an S3 bucket fail, and internal logs show errors of the form:
User is not authorized to perform kms:Decrypt on resource.

Possible Cause ​

Your IAM role is missing permissions required to access S3 objects that are encrypted with a KMS key.

Recommendations ​

Add this sub-element to the applicable IAM role:

 "Action": [
     "kms:Decrypt",
     "kms:Describe*",
     "kms:Get*",
     "kms:List*",
     "kms:GenerateDataKey"
 ],

Existing AssumeRole Fails to Authorize on S3 ​

You might find that your existing AssumeRole from Cribl Stream/Edge fails to authorize Cribl Search on S3.

Possible Cause ​

Cribl Search requires a different Principal element.

Recommendations ​

Add the required Principal to your trust policy. (Follow the link to see how to copy this Principal out of your Dataset Provider modal.)

Can’t Add Buckets to Amazon Security Lake Dataset ​

You might find that you cannot add new buckets to your Dataset.

Possible Cause ​

Insufficient permissions on Amazon Security Lake.

Recommendations ​

Check whether you have added the securitylake:ListDataLakes permission to your AssumeRole.

Performance Fixes ​

This section covers workarounds for slow, inefficient, or failing searches.

Scheduled Search Jobs Canceled Due to Queue Length Limit ​

You might find that some scheduled search jobs were canceled because the search queue reached its configured limit.

Possible Cause ​

By default, the search queue can hold five times the maximum number of concurrent searches. When the queue is full, Cribl Search rejects additional jobs.

Recommendations ​

To check the state of the queue, enter .show queue stats in the query box.

Contact Cribl Support or your Cribl account team to adjust the queue limit for your workload.

Feature Workarounds ​

This section covers substitutes for features that Cribl Search doesn’t provide out-of-the-box.

Get the Length of an Array Field ​

Cribl Search currently has no native function to return the length of an array’s field, without stringifying the value.

Recommendations ​

1. Strip everything from the array except for commas; apply the strlen() function; and then add 1 to the result.

2. Try a query of the form: print <arrayName> = dynamic([1,2,3]) | extend len = <arrayName>.length

Metrics and Monitors Issues ​

This section covers issues specific to Metrics and Monitors.

Cribl Search Asks You to Create a Metrics Dataset ​

You open Metrics, and Cribl Search shows Create a Metrics Dataset and invites you to add one, even though your Organization already ingests metrics.

Possible Cause ​

No metrics Dataset has been shared with you. Cribl Search lists only the metrics Datasets you can access, so an empty list looks the same as no Dataset existing at all. Adding a Dataset needs the Editor or Admin Permission, so the suggested action might also be unavailable to you.

Recommendations ​

Ask a Member with the Maintainer Permission on the metrics Dataset to share it with you, instead of adding a new Dataset. Cribl Search auto-provisions one metrics Dataset per lakehouse engine. See Metrics Permissions.

If you have the Admin Permission and still see this state, open the Engines tab to confirm that a lakehouse engine exists and its status is Ready.

The Metrics Explorer Is Empty with No Message ​

The Metrics Explorer opens but lists no metrics Datasets and gives no explanation.

Possible Cause ​

Cribl Search could not load the Dataset or lakehouse engine list. It suppresses its guidance message when that request fails, which leaves an empty Explorer.

Recommendations ​

Refresh the page. If the Explorer is still empty, check the status of your lakehouse engines, then contact Cribl Support.

A Monitor Is Missing from the Monitors List ​

A Monitor that a colleague can see doesn’t appear in your Monitors list.

Possible Cause ​

The Monitors list shows only the Monitors whose queries all read metrics Datasets you can access. A Monitor on a Dataset you can’t access is hidden rather than shown as locked.

Recommendations ​

Ask a Member with the Maintainer Permission to share every metrics Dataset the Monitor evaluates. A Monitor with more than one query can read more than one Dataset, and you need access to all of them. See Monitor Permissions.

Notification Target Fields Look Blank ​

You open a Notification target and its connection and credential fields are empty.

Possible Cause ​

Cribl Search omits those fields for Members who can’t create Notification targets. It leaves them out of the response rather than masking their values, so the form renders them as empty even though the target is fully configured.

Recommendations ​

Check whether you have the Admin Permission on Cribl Search before treating the Target as misconfigured. See Credential Fields on Targets.

Email Notifications Issues ​

The section details the troubleshooting steps you can take if an email Notification fails to reach its intended recipient.

Test the Email Notification Target ​

To prevent email Notifications from failing in production, you can check the target’s configuration as follows:

1. Open the target (in Settings > Search > Notification Targets).
2. Select Test Target.
3. Add one or more email addresses to the Test Target modal. (Include at least one address where you can check incoming messages.)
4. Select Send Test Email.
5. If the test succeeds, check the specified inbox(es) to verify that the test message actually arrived. If not, re-examine the target’s configuration.

The preconfigured system_email target does not enable displaying this Test Target option or its parent modal.

Check Notification Logs ​

A failed email Notification leaves a log entry. You can search Cribl internal logs for SMTP errors using a query of this form:

dataset="cribl_internal_logs" level=error channel="output:*" ioName=smtp

Within any returned log events, unwrap the reason field to find details about the root cause of the outbound email failure.