On This Page

Home / Edge/ Secure Your Deployment/ Centralize and Reuse Global Secrets/Back Up, Audit, and Troubleshoot Global Secrets

Back Up, Audit, and Troubleshoot Global Secrets

Global secrets are supported only in Cribl.Cloud.

On the Leader, Cribl persists global secrets configuration under $CRIBL_HOME/local/cribl/. The following table describes the files that hold the global secrets configuration.

FilePurpose
secret-stores.ymlSecret store declarations, including store type, connection URL, authentication settings, TLS configuration, and health check settings.
secret-folders.ymlSecret folder declarations, including the backing secretStoreId, description, and sharing configuration (groupIds, fleetIds).
secret-folders/<folder-id>/global-secrets.ymlPer-secret-folder secret metadata, including secret type, description, tags, and (for folders in externally backed secret stores) secret references with paths and response fields. Does not contain values for sensitive fields.
secret-folders/<folder-id>/values.jsonEncrypted sensitive values for global secrets in Cribl-backed secret folders. Not present for secret folders backed by external secret stores. Not tracked in Git. Cribl-managed: do not edit or delete.

Cribl updates these files when you save changes to secret stores, secret folders, or global secrets. Do not manually edit these files. Incorrect or missing files can prevent the Leader from correctly loading global secrets.

Git tracks the YAML metadata but excludes the values.json file for each secret folder. Without values.json, global secrets in the built-in cribl store cannot be recovered from Git alone.

Backup and Restore in Cribl.Cloud

In Cribl.Cloud, for the built-in cribl secret store, Cribl backs up global secret configuration and encrypted values. You do not need to back up Leader filesystem paths manually. To restore global secrets from a backup, contact Cribl Support.

For secret stores that are backed by an external provider, Cribl caches secret values in memory only and never persists them on disk. When the Leader restarts, it reinitializes each secret store client and re-fetches global secrets from the external provider. You do not need to restore the global secrets in Cribl because the source of truth is the external provider. If the external secret store is healthy, Cribl automatically re-resolves the global secrets.

Audit and Review Change History

Cribl provides two complementary views of global secrets activity:

  • Git-tracked metadata: Cribl stores definitions for secret stores, secret folders, and global secrets in Leader files under $CRIBL_HOME/local/cribl/. Secret values are not stored in Git.

    • When you perform a global commit, those files participate in commits like the rest of Leader configuration. Diffs show changes to metadata and sharing rules, not plaintext secret values. Git commit history records who committed the changes. See Marker Files for Git Review for more information.
  • Leader audit.log: Create, update, delete, reload, and related operations on secret stores, secret folders, and global secrets emit JSON lines to the Leader’s audit.log.

    • Each line includes details about the operation like the time, action, resource type (global-secrets, secret-folders, or secret-stores), authenticated user or API key context, resource identifiers, outcome, and a reason in case of failure. Cribl never writes global secret values to audit.log.

    • Audit events are written only on the Leader, not per node. To read and export audit.log, see Internal Logs.


    The following example demonstrates an audit.log line for deleting a global secret:

    {"time":"2026-06-14T08:43:01.945Z","action":"delete","type":"global-secrets","user":"admin","id":"myid4","outcome":"success","folderId":"payments-secrets1","secretType":"text","requestId":"123a4567-8910-bcde-1f2g-34a5bcd678ef","service":"secrets"}

Marker Files for Git Review

To give each Worker Group and Edge Fleet a safe audit trail of global secret changes in Git, Cribl writes a marker file, global-secrets-marker.yml, under the local/cribl tree (for example, groups/<groupId>/local/cribl/global-secrets-marker.yml in a Distributed deployment). The file holds fingerprints of credential fields, not the global secret values themselves, so that diffs show which fields changed without listing secret values in plain text. Deploy then follows the normal commit-and-deploy path with an updated configuration bundle.

Git cannot show a readable diff for global secret values because Cribl stores them only as encrypted data on the Leader or resolves them from an external secret store. Cribl uses fingerprints to record changes to credential material for the Worker Group or Edge Fleet while omitting secrets from the diff, and refreshes global-secrets-marker.yml when those fingerprints change. The global-secrets-marker.yml file might be absent until the first refresh, or contain only an empty map after entries are cleared.

Cribl refreshes fingerprints when:

  • Sharing rules change for a secret folder (so different Worker Groups and Edge Fleets gain or lose access to secrets in that folder).

  • A secret folder reloads from an externally backed secret store (so resolved values from the external provider might have changed even when Cribl metadata paths did not).

  • Global secrets in a secret folder that is shared with a Worker Group or Edge Fleet are added, updated, or removed (including field-level edits such as rotated passwords).

Marker File Structure

In global-secrets-marker.yml, each top-level key identifies one global secret using the same composite form as references. For example:

'secretFolderId/globalSecretId':
  description: 'My global secret'
  secrets:
    username: a1b2c3d4e5f67890
    password: 90f8e7d6c5b4a321

The values mapped to each key include:

  • secrets (required): A map of field names to fingerprint strings. For every populated credential field on the global secret, Cribl stores the first 16 hexadecimal characters of the SHA-256 digest of the field’s value to keep the diff short and readable. This applies for both sensitive fields (like password, secretKey, and value) and non-sensitive fields (like username and apiKey).
  • description (optional): If the global secret has a description, Cribl includes it in plain text to provide non-sensitive context for the fingerprints.

Cribl writes keys in sorted order for stable diffs.

Git Commits and Marker Files

When you use the Git workflow to commit configuration for a Worker Group or Edge Fleet, the commit always includes global-secrets-marker.yml. You cannot deselect global-secrets-marker.yml in the commit file list. This ensures that global secret changes are visible to reviewers and deploy picks up a new configuration bundle when fingerprints change. Do not remove or omit global-secrets-marker.yml from commits made outside the UI, such as manual Git edits.

Marker File Limitations

Fingerprints reflect what Cribl last wrote into the Worker Group or Edge Fleet configuration for review and deploy. They do not prove which credential value a node is using at the moment. If behavior does not match the latest marker entries, reload the affected secret folder or commit and deploy the Worker Group or Edge Fleet again.

Marker file entries are digests and optional description text. They facilitate audits and make diffs more readable, but they are not a substitute for access control, external provider audit logs, or runtime monitoring.

Treat truncated digests as change indicators: a new fingerprint means that Cribl recorded an updated value for the field when it refreshed the marker file. Identical fingerprints for two different secrets are unlikely in normal use but are not a cryptographic guarantee of equality across unrelated secrets.

Troubleshooting

This troubleshooting section addresses issues with global secrets during deploy or Pack clone. For HTTP status codes for secret store, secret folder, and global secret API operations, see the Cribl API reference.

Secret Store Health Check Failures

Cribl periodically checks connectivity to each secret store that is backed by an external provider. If a health check fails, Cribl sets the secret store’s status to unhealthy, displays a system bulletin message, and blocks reload for the store’s secret folders. Cribl blocks deploy when the Leader must fetch global secrets from the unhealthy secret store and the secrets are not already cached in memory, such as during Leader startup before it successfully resolves secrets from the external provider. You can still commit configuration changes. If the Leader already cached the global secrets, deploy can proceed using the cached values even while the secret store is unhealthy.

The system bulletin message includes an error message to explain the reason for the failed health check. Take the appropriate action based on the error message to resolve the issue in the external provider.

After the issue is fixed, the next periodic health check will automatically detect recovery. When the next health check succeeds, Cribl clears the bulletin message and unblocks reload for affected secret folders. If deploy failed because the Leader could not resolve global secrets while the secret store was unhealthy, deploy will succeed again when the Leader can fetch secrets from the recovered store.

Health Check Failure Messages for HashiCorp Vault-Backed Secret Stores

Cribl displays the following system bulletin messages when a health check fails for a secret store that is backed by HashiCorp Vault:

FailureError message
Connection was refused or Vault is unreachableCould not validate the provided Vault configuration: ECONNREFUSED
Vault is sealedVault is sealed and cannot serve requests
Token is invalid or expiredVault rejected the configured credentials (token invalid or expired)
Authentication credentials configured for the secret store do not have the correct permissions to read the Vault health check endpoint.Vault access denied (HTTP 403). Verify that Cribl's token or policy has not been revoked.
Health endpoint returned a non-200 responseHTTP status code and response body for diagnosis

Deploy Failures

While the Leader assembles a deployment bundle for a Worker Group or Edge Fleet, it must resolve global secrets in secret folders shared with the Group or Fleet. If the Leader cannot resolve global secrets, deploy will fail before the Leader produces a valid bundle with the error Failed to deploy bundle: Could not fetch global secrets. Please try again!.

To resolve the issue:

  1. Verify that the Workspace Leader is healthy and that Global Settings > Security > Secret Stores does not show a non-healthy status for the relevant store. If the Leader restarted recently, wait a few minutes and retry the deploy.

  2. Verify that every global secret in folders shared with the Worker Group or Edge Fleet resolves successfully. A misconfigured or unresolvable secret can block deploy even when no configuration references it.

  3. Retry the commit and deploy.

Cloning Packs and Sharing Global Secrets

When you clone a Pack into a Worker Group or Edge Fleet, Cribl checks whether referenced global secrets are shared with the target Group or Fleet. If the necessary secret folders are not shared, clone returns an error like Global secrets "<folder>/<secret>" are not shared with destination "<group>". Referenced global secrets will be inaccessible. If several global secrets are unshared, Cribl lists each reference in the error message, separated by commas.

To resolve the issue:

  1. Share the secret folder that contains the global secret with the target Worker Group or Edge Fleet or change the Pack configuration to remove or replace the global secret references.

  2. Retry the Pack clone operation.