Conceptual Walkthrough of AppScope

​

This two-minute video provides a conceptual walkthrough of Cribl open-source AppScope project.

Download AppScope

​

Cribl Stream must download the AppScope package from the Cribl CDN the first time it performs an operation on this Source. This will also update the AppScope package’s version, if a newer one is available in the CDN.

If Cribl Stream cannot access the CDN (for example, if you are working on an airgapped instance), you should manually download the binary and place it in $CRIBL_HOME/state/download/scope.

Configure Cribl Stream to Receive AppScope Data

​

1. On the top bar, select Products, and then select Cribl Stream. Under Worker Groups, select a Worker Group. Next, you have two options:
   • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Source and select the Source you want from the list, choosing either Select Existing or Add New.
   • To configure via the Routes, select Data > Sources (Stream) or More > Sources (Edge). Select the Source you want. Next, select Add Source.
2. In the New Source modal, configure the following under General Settings:
   • Input ID: Enter a unique name to identify this AppScope Source definition. If you clone this Source, Cribl Stream will add -CLONE to the original Input ID.
   • Description: Optionally, enter a description.
   • UNIX domain socket: For details, see Unix Domain Socket Fields below.
3. Under Authentication, select an Authentication method from the dropdown:
   • Manual: Use this default option to enter the shared secret clients must provide in the authToken header field. Select Generate if you need a new auth token. If empty, unauthorized access will be permitted.
   • Secret: This option exposes an Auth token (text secret) drop-down, in which you can select a stored secret that references the auth token described above. The secret can reside in Cribl Stream’s internal secrets manager or (if enabled) in an external KMS. Click Create if you need to configure a new secret.
4. Next, you can configure the following Optional Settings:
   • UNIX socket permissions: Permissions to set for socket. For the preconfigured in_appscope source, defaults to 777. When creating a new AppScope Source, you should set this to 777. If empty, falls back to the runtime user’s default permissions.
   • Tags: Optionally, add tags that you can use to filter and group Sources in Cribl Stream’s UI. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.
5. Optionally, configure any TLS settings, AppScope Rules, Persistent Queue, Processing, Advanced settings and Connected Destinations outlined in the sections below.
6. Select Save, then Commit & Deploy.

Unix Domain Socket Fields

​

When toggled on, the Unix doman socket field exposes the following two fields to specify a file-backed UNIX domain socket connection to listen on.

• UNIX socket path: Path to the UNIX domain socket. Defaults to $CRIBL_HOME/state/appscope.sock.
• UNIX socket permissions: Permissions to set for this socket, for example, 777. If empty, Cribl Stream will use the runtime user’s default permissions. (Appears under Optional Settings)

When UNIX domain socket is toggled off, you instead see the following two fields to specify a network host and port.

• Address: Enter the hostname/IP on which to listen for AppScope data. (For example, localhost.) Defaults to 0.0.0.0, meaning all addresses.
• Port: Enter the port number to listen on.

By default:

• In Cribl Stream, UNIX domain socket is toggled off, with default network connections (address and port) of 0.0.0.0:10090 for TCP, and 0.0.0.0:10091 for TLS, respectively.
• In Cribl Edge, UNIX domain socket is toggled on, with a UNIX socket path of $CRIBL_HOME/state/appscope.sock.

TLS Settings (Server Side)

​

This left tab is displayed only when the Optional Settings toggle UNIX domain socket is off. It provides the following options.

Enabled: Defaults to toggled off. When toggled on:

Certificate name: Name of the predefined certificate.

Private key path: Server path containing the private key (in PEM format) to use. Path can reference $ENV_VARS.

Passphrase: Passphrase to use to decrypt private key.

Certificate path: Server path containing certificates (in PEM format) to use. Path can reference $ENV_VARS.

CA certificate path: Server path containing CA certificates (in PEM format) to use. Path can reference $ENV_VARS.

Authenticate client (mutual auth): Require clients to present their certificates. Used to perform mutual authentication using SSL certs. Default is toggled off. When toggled on:

• Validate client certificates: Reject certificates that are not authorized by a CA in the CA certificate path, or by another trusted CA (for example, the system’s CA). Default is toggled on.

• Common name: Regex that a peer certificate’s subject attribute must match in order to connect. Defaults to .*. Matches on the substring after CN=. As needed, escape regex tokens to match literal characters. (For example, to match the subject CN=worker.cribl.local, you would enter: worker\.cribl\.local.) If the subject attribute contains Subject Alternative Name (SAN) entries, the Source will check the regex against all of those but ignore the Common Name (CN) entry (if any). If the certificate has no SAN extension, the Source will check the regex against the single name in the CN.

Minimum TLS version: Optionally, select the minimum TLS version to accept from connections.

Maximum TLS version: Optionally, select the maximum TLS version to accept from connections.

AppScope Rules

​

The AppScope Rules settings are available:

• In Cribl Stream single-instance – but not Cribl.Cloud – deployments.
• In Cribl Edge, both single-instance and Cloud.

Rules: Click Add Rule to include processes to scope, and to link to an AppScope config. Once you have saved the configuration, and committed and deployed your changes, AppScope will instrument any process that matches a Rule, on any Edge Node in the Fleet.

(When no Rules are defined, you can still scope by PID in the Edge Processes page. Scope by PID only instruments a single process running in one Edge Node.)

• Process name: Matches if the string value you enter corresponds to the basename of the scoped process.

• Process argument: Matches if the string value you enter appears as a substring anywhere in the scoped process’ full command line (including options and arguments).

• AppScope config: Select an AppScope config.

Transport override: Enter a URL to override aspects of the transport configuration, such as the hostname, port, or TLS settings. For details, see Transport Override Details.

Transport Override Details

​

In scenarios like the following, use the Transport override option to extend the defaults in AppScope’s transport configuration:

• When this Source is set to TCP mode, it typically listens on the default address 0.0.0.0. Scoped clients will need a specific IP or hostname to connect. In these cases, set Transport override URL to a specific IP/hostname (example format: tcp://my.host.name). This Source will parse the URL and look for the hostname and port, then use those values to override what would otherwise be sent to the scope start command.

• When this Source is set to UNIX domain socket, it listens by default on $CRIBL_HOME/state/appscope.sock. The socket is often created on a mounted volume in a container. The path to that socket might be different outside the container, or when mounted into another container. In these cases, set the Transport override URL to specify an alternative path (example format: unix:///some/other/volume/appscope.sock).

Persistent Queue Settings

​

In the Persistent Queue Settings tab, you can optionally specify persistent queue storage, using the following controls. Persistent queue buffers and preserves incoming events when a downstream Destination has an outage or experiences backpressure.

Before enabling persistent queue, learn more about persistent queue behavior and how to optimize it with your system:

• About Persistent Queues
• Optimize Source Persistent Queues (sPQ)

On Cribl-managed Cloud Workers (with an Enterprise plan), this tab exposes only the Enable Persistent Queue toggle. If enabled, PQ is automatically configured in Always On mode, with a maximum queue size of 1 GB disk space allocated per PQ‑enabled Source, per Worker Process.

The 1 GB limit is on uncompressed inbound data, and the queue does not perform any compression. This limit is not configurable. For configurable queue size, compression, mode, and other options below, use a hybrid Group.

Enable persistent queue: Default is toggled off. When toggled on:

Mode: Select a condition for engaging persistent queues.

• Always On: This default option will always write events to the persistent queue, before forwarding them to the Cribl Stream data processing engine.
• Smart: This option will engage PQ only when the Source detects backpressure from the Cribl Stream data processing engine.

Smart mode only engages when necessary, such as when a downstream Destination becomes blocked and the Buffer size limit reaches its limit. When persistent queue is set to Smart mode, Cribl attempts to flush the queue when every new event arrives. The only time events stay in the buffer is when a downstream Destination becomes blocked.

Buffer size limit: The maximum number of events to hold in memory before reporting backpressure to the sender and writing the queue to disk. Defaults to 1000. This buffer is for all connections, not just per Worker Process. For that reason, this can dramatically expand memory usage. Connections share this limit, which may result in slightly lower throughput for higher numbers of connections. For higher numbers of connections, consider increasing the limit.

Commit frequency: The number of events to send downstream before committing that Stream has read them. Defaults to 42.

File size limit: The maximum data volume to store in each queue file before closing it and (optionally) applying the configured Compression. Enter a numeral with units of KB, MB, and so forth. If not specified, Cribl Stream applies the default 1 MB.

Queue size limit: The maximum amount of disk space that the queue is allowed to consume on each Worker Process. Once this limit is reached, this Source will stop queueing data and block incoming data. Required, and defaults to 5 GB. Accepts positive numbers with units of KB, MB, GB, and so forth. Can be set as high as 1 TB, unless you’ve configured a different Worker Process PQ size limit in Group or Fleet settings.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. To this field’s specified path, Cribl Stream will append /<worker-id>/inputs/<input-id>.

Compression: Optional codec to compress the persisted data after a file closes. Defaults to None; Gzip is also available.

In Cribl Stream 4.1 and later, the Source persistent queue default Mode is Always on, to best ensure events’ delivery. For details on optimizing this selection, see Optimize Source Persistent Queues (sPQ).

You can optimize Workers’ startup connections and CPU load at Group/Fleet settings > Worker Processes.

Processing Settings

​

Event Breakers

​

Event Breaker rulesets: A list of event breaking rulesets that will be applied to the input data stream before the data is sent through the Routes. Defaults to System Default Rule.

Event Breaker buffer timeout: How long (in milliseconds) the Event Breaker will wait for new data to be sent to a specific channel, before flushing out the data stream, as-is, to the Routes. Minimum 10 ms, default 10000 (10 sec), maxiumum 43200000 (12 hours).

Fields

​

In this section, you can define new fields or modify existing ones using JavaScript expressions, similar to the Eval function.

• The Name specifies the field name, which can either be a new field (unique within the event) or an existing field name to modify its value.
• The Value is a JavaScript expression (enclosed in quotes or backticks) to compute the field’s value (can be a constant). Select this field’s advanced mode icon (far right) if you’d like to open a modal where you can work with sample data and iterate on results.

This flexibility means you can:

• Add new fields to enrich the event.
• Modify existing fields by overwriting their values.
• Compute logic or transformations using JavaScript expressions.

Pre-Processing

​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline or Pack to process data from this input before the data is sent through the Routes.

Disk Spooling

​

For Cribl Search to access the data that arrives at an AppScope Source, Disk Spooling must be enabled.

Enable disk persistence: Whether to save metrics to disk. When toggled on, exposes this section’s remaining fields.

Bucket time span: The amount of time that data is held in each bucket before it’s written to disk. The default is 10 minutes (10m).

Max data size: Maximum disk space the persistent metrics can consume. Once reached, Cribl Stream will delete older data. Example values: 420 MB, 4 GB. Default value: 100 MB.

Max data age: How long to retain data. Once reached, Cribl Stream will delete older data. Example values: 2h, 4d. Default value: 24h (24 hours).

Compression: Optionally compress the data before sending. Defaults to gzip compression. Select none to send uncompressed data.

Path location: Path to write metrics to. Default value is $CRIBL_HOME/state/appscope.sock.

Advanced Settings

​

Enable proxy protocol: Toggle on if the connection is proxied by a device that supports Proxy Protocol v1 or v2.

IP allowlist regex: Regex matching IP addresses that are allowed to establish a connection.

Active connection limit: Maximum number of active connections allowed per Worker Process. Defaults to 1000; enter 0 to allow unlimited connections.

Socket idle timeout (seconds): The duration that Cribl Stream will wait for activity on an idle TCP socket before closing the connection. Disabled when set to 0, the default.

Forced socket termination timeout (seconds): The extra time the server waits before forcibly closing a socket that has been idle (TCP socket idle timeout) or exceeded its maximum lifespan (TCP socket max lifespan) but has not yet properly closed. This prevents resource leaks caused by unresponsive clients or network issues. Configure based on network latency and client behavior. Default: 30 seconds. Set to 0 to disable.

Socket max lifespan (seconds): The duration that a socket is allowed to remain open, regardless of activity. This setting prevents resource exhaustion (such as TCP pinning) by limiting the lifespan of connections. Configure based on expected connection durations and resource availability. Disabled when set to 0, the default.

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Connected Destinations

​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

AppScope with Edge on Kubernetes

​

When Cribl Edge detects that a scope’d process is running inside a Kubernetes container, it reports the Kubernetes metadata as kube_** properties. It adds these to the incoming events and metrics, and you can view the combined events and metrics on the AppScope Source’s Live Data tab.

For Cribl Edge to detect that it’s running in a Kubernetes Pod, you must first set the CRIBL_K8S_POD environment variable.

Internal Fields

​

Cribl Stream uses a set of internal fields to assist in handling of data. These “meta” fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

Fields for this Source:

• __inputId
• __srcIpPort

Examples

​

The following examples work only in Cribl Stream. You can vary the scoped commands (ps -ef and curl) as desired.

Cribl.Cloud – TLS

​

An in_appscope_tls TLS Source is preconfigured for you on Cribl.Cloud, using port 10090. You can send it AppScope data using this command:

./scope run -c <Your-Ingress-Address>:10090 -- ps -ef

Cribl Cloud – TCP

​

An in_appscope_tcp TCP Source is preconfigured for you on Cribl.Cloud, using port 10091. You can send it AppScope data using this command:

./scope run -c tcp://<Your-Ingress-Address>:10091 -- curl -so /dev/null \
https://wttr.in/94105

Periodic Logging

​

Cribl Stream logs metrics about incoming requests and ingested events once per minute.

These logs are stored in the metrics.log file. To view them in the UI, open the Source’s Logs tab and choose Worker Process X Metrics from the drop-down, where X is the desired Worker process.

This kind of periodic logging helps you determine whether a Source is in fact still healthy even when no data is coming in.