On This Page

Home / Edge/ Knowledge Libraries/AppScope Configs

AppScope Configs

You can obtain high-fidelity application data from any Linux command or application using AppScope, an open‑source tool from Cribl. This includes data that’s difficult or impossible to obtain any other way. (Note that AppScope is no longer being actively developed by Cribl.)

AppScope is included in your Cribl Edge installation. One or two AppScope Sources are enabled right out-of-the-box; they are configured to send data to Cribl Edge or Cribl Stream on the same host.

AppScope ships with a set of config files that offer a variety of options for what events and metrics you want AppScope to capture, and whether to enable payload capture. These are: sample_config, default_metrics_events, verbose_metrics_events, and verbose_metrics_events_payloads.

To understand precisely what one of these files does, select it to open its AppScope Configs modal and select Manage as JSON. For explanations and definitions, see the Config File and Schema Reference sections in the AppScope docs.

You can customize the config files, or edit them via Manage as JSON. When editing, you can revert to the original state by selecting Restore Default. The safer option is to select Clone AppScope config, then edit and save your new version with a new file name. The sample configs, and any variants you have created and saved, constitute a reusable set of AppScope configurations in the AppScope Config Library.

When you have assigned an AppScope config to a Rule in an AppScope Source, and you change it in the Knowledge UI, the changes will not take effect until you do the following:

  • In the AppScope Source AppScope Rules settings, re-assign the config to the Rule. (That is, unselect the config file, then select it again.)
  • Save and commit.

Creating AppScope Configs
​

To access Appscope Configs:

  1. In the sidebar, select Fleets, and choose a Fleet.
  2. Then, on the Fleets submenu, select More, then Knowledge, then Appscope Configs.

Select Add AppScope config to open a modal to begin creating an AppScope config, using the options below.

Once you’ve created AppScope configs, you can edit, delete, search, and tag them as desired.

General Settings
​

ID: Enter a unique name to identify this AppScope Config.

Description: Optionally, enter a description that will remind you what the config is for when you see it listed in the library.

Tags: Optionally, enter any desired tags.

Transports
​

AppScope offers several options for routing data. See Data Routing in the AppScope docs.

Use AppScope Source’s IP/port or UNIX domain socket?: Toggle on to apply the transport options defined in the associated AppScope Source. Toggle off to expose the following fields in this section.

Cribl authentication token: Optionally, set a value to add to the config-event header (as a top-level authToken property) when AppScope establishes a connection.

Send metrics and events over the same connection?: Defaults to toggled on, which locks some settings into preconfigured values. Toggle off if you prefer to route metrics and events independently.

Connection type: From the drop-down, choose one of the connection types described below, then enter values in whatever fields the UI exposes.

  • Toggle Send metrics and events over the same connection? on to configure a single connection by choosing TCP, Edge (the default), or Unix.
  • When routing metrics and events independently, you configure one connection for metrics and another for events; the drop-down offers two additional connection types: UDP and File.

Transports and Connection Types
​

At the heart of any AppScope config are the transports you define. In AppScope, a transport specifies something to send, how to send it, and where to send it. The things to send can be metrics, events, or logs.

Depending on the choices you make in the Transport settings, the UI will make some or all of the following types of connections available.

UDP: The destination is the network server whose hostname or IP address you specify in the Host field, along with a UDP Port to listen on.

TCP: The destination is the network server whose hostname or IP address you specify in the Host field, along with a TCP Port to listen on. The Enable TLS toggle defaults to off.

Unix: The destination is a Unix domain server listening on the Unix domain Socket path you specify.

File: The destination is the file whose directory location you specify in the File path field, along with a Buffering method.

Edge: The destination is Cribl Edge, listening on a preconfigured Unix domain socket.

TLS Settings (TCP Only)
​

When the Connection type is TCP, the UI exposes the Enable TLS toggle, which defaults to off.

If you toggle Enable TLS on:

  • Toggle Validate server certs on if you want the connection to fail when the TLS server certificate cannot be validated. Defaults to off.

  • Leave CA certificate path blank if you toggle Validate server certs on and are using the local OS-provided trusted CA certificates to validate the server’s certificate. Otherwise, specify the server path containing CA certificates, which must be in PEM format.

Logs Settings
​

Configure where and how AppScope should send its own logs.

Connection type: From the drop-down, choose one of the connection types described above, then enter values in whatever fields the UI exposes.

Log level: Set logging verbosity. When you toggle Send metrics and events over the same connection? on, Log level is locked at warning.

File path: Specify a directory in which to store logs. If not specified, defaults to /tmp/scope.log.

Buffering: Set this method toLine if there’s a chance that multiple scoped processes will be writing to the same file. This prevents the log file from getting scrambled, with interleaved lines. However, in single-writer scenarios, selecting Full often improves performance.

Metrics Settings
​

The Schema Reference in the AppScope docs lists and fully describes every metric that AppScope can emit.

Process (proc.*) metrics periodically report information about resource usage at a point in time. All other metrics report on actions taken by a scoped process. For examples, see Metrics and Events in AppScope in the AppScope documentation.

Should AppScope emit metrics?: Defaults to toggled on, which exposes the following options.

Format: AppScope can emit metrics in either of two formats: NDJSON (the default), or StatsD. If you choose StatsD, the UI exposes the following fields:

  • StatsD prefix: Optionally, enter a prefix for AppScope to add to metrics.
  • StatsD max length: Optionally, enter a maximum length, in bytes, for StatsD-formatted metrics. AppScope will truncate any metric which exceeds the maximum length. (Defaults to 512.)

Verbosity: Set a value between 0 and 9. (Defaults to 4.) Verbosity controls both the cardinality of metrics that AppScope emits, and which kinds of metrics AppScope aggregates. For details, see Metrics and Events in AppScope in the AppScope documentation. But the general principle is:

  • At low verbosity, metrics summarize multiple actions over a reporting period.
  • At higher verbosity, metrics provide details about single actions in real time.

Reporting period: Set the metric summary interval in seconds. Defaults to 10.

Watch list: Select categories of metrics to emit. Defaults to the following categories: StatsD, File System, Network, HTTP, DNS, Process.

Events Settings
​

The Schema Reference in the AppScope docs lists and fully describes every event that AppScope can emit.

Should AppsScope emit events?: Defaults to toggled on, which exposes the following controls in this section.

Event-rate limiter (events per second): AppScope discards events generated in excess of the limiter setting, which defaults to 10000. To disable the limiter, set it to 0.

Enhance filesystem event data: If toggled on (default), AppScope enhances fs.open events by adding the uid, gid, and mode fields.

Watch list: Specify categories of events to emit, and configure their content and structure. Defaults to the following categories: Console, File, Filesystem, Network, DNS, and HTTP. For details, see Categories.

Events Categories
​

The Watch list is a table providing a row for each enabled category of events. The Type column contains the category names. The other columns (Name, Field, and Value) contain regular expressions.

At the right edge of the Name, Field, and Value fields is a Copy button, a Flag button to append Regex flags, and an Expand button to open a validation modal.

AppScope emits an event when its values match all the regexes in the corresponding category’s row. For HTTP, request and response events must also match the headers listed in the Header column.

For Console events, toggle Allow binary on (default) if you want Console events to include binary data along with text data. If you prefer to redact binary data, toggle Allow binary off.

The Type column enumerates any or all of the following event category names. Switch the toggle in the Enabled column to include a category.

  • Console: Includes writes to standard out and error. Use this to monitor console output, especially in containerized environments where logging to files isn’t commonly done.
  • File: Includes writes to files. Use this to monitor log files and/or any other files of interest.
  • Filesystem: Includes filesystem operations like open, close, and delete.
  • Network: Includes open and close events on network connections.
  • DNS: Includes DNS request and response events.
  • HTTP: Includes HTTP request and response events.
  • Granular: Seldom used, and not emitted by default. Can be useful when tracking down a problem. When this type is toggled on, AppScope sends more events than it otherwise would, giving you a more granular view of the activity of the scoped process.

Some usage examples:

  • For fs.* events, AppScope normally emits fs.open and fs.close events. With Granular enabled, every time a read or write occurs, AppScope emits fs.read or fs.write, respectively.
  • For net.* events, AppScope normally emits net.open, net.close, and net.app.With Granular enabled, every time a read or write is done on the socket, AppScope emits net.rx or net.tx, respectively.

Payloads
​

AppScope never sends encrypted payloads to disk, to Cribl Stream, or to Cribl Edge.

Capture all payloads: Toggle on to enable capturing all payloads. Defaults is off. Enable this setting with caution: when scoping I/O-intensive programs, it can produce large amounts of data.

Protocol detection: Select Add Protocol to specify a protocol (for example, StatsD, HTTP, Redis) for AppScope to detect in network payloads, as well as what AppScope should do upon detecting that protocol. AppScope checks the first packet on a channel against the regular expression for each protocol until it finds a match. Then AppScope treats the matching protocol as the detected protocol for its channel, and ignores any others listed. See Sample Protocols for regexes that detect common protocols.

Name: Enter a short, descriptive name to define the protocol-detect event. AppScope inserts this value into the header for payloads sent to Cribl. Since AppScope sends this as JSON, you cannot include double quotes or backslashes.

Regex: Enter a regular expression to detect the desired protocol.

Generate protocol-detect-events: Toggle on to generate protocol-detect events. Default is toggled off.

Capture payload: Toggle on to send payloads from matching streams to the configured destinations. Default is toggled off.

Convert to hex: Toggle on (default) if you want AppScope to convert part of the payload to a hex-string before applying the regex. Toggle off if you want AppScope to apply the regex to the binary payload.

Bytes to convert: The number of bytes to convert to hex before applying the regex. Defaults to 256.

Payload directory: Specify the directory where AppScope should store payload files. Consider using a performant filesystem to reduce I/O performance impacts. Defaults to tmp. When General Settings > Send metrics and events over the same connection? is toggled off, AppScope ignores this field and sends payloads to the default destination.

Advanced Settings
​

Send process-start message?: Toggle on (default) to enable the process-start message. This message is the first one AppScope sends upon establishing a connection. It describes AppScope’s configuration for that session, as well as the application being scoped.

Command directory: If you want to dynamically change AppScope configuration while scoping a running process, save a scope.<pid> file to a command directory of your choice, where <pid> is the PID of the scoped process, and the file contains lines of the form <ENV_VAR>=<value>. Once per reporting period, AppScope looks for a scope.<pid> file in the command directory, which defaults to /tmp. You can configure reporting period in the Metrics tab.

Metadata: If you want to add a field to events and metrics that AppScope emits, select Add Metadata and define it as a key-value pair. Key/value definitions cannot include environment variables.

Per-process configs: Optionally, define a config that, given a process that fits a particular pattern, will partly or completely override the config you have defined already.

Select Add config and configure the Filters relevant to the process(es) you want to match:

  • Process name: Enter a string to match the basename of the scoped process.
  • Process argument: Enter an expression to match the scoped process’s full command line including options and arguments.
  • Hostname: Enter a string to match the hostname of the machine where the scoped process is running.
  • Username: Enter a string to match the username for the scoped process’s UID.
  • Environment: To match when an environment variable is set, enter its name alone (for example, FOO). To match when the variable is set with a particular value, enter both (for example, FOO=bar).
  • Ancestor: Enter a string to match the basename of the scoped process’s parent, parent’s parent, and so on.

Then, in the Config column, select Edit. This takes you through the settings, beginning with General Settings, but this time you’ll be specifying what in the previously-defined settings to override when a process matches your Filters.

Protocol Detection Examples for Payloads
​

Here are some example definitions for detecting protocols in payloads.

Redis
​

This regex detects the plain-text Redis serialization protocol (RESP).

Name: Redis

Regex: "^[*]\\d+|^[+]\\w+|^[$]\\d+"

Mongo
​

This regex detects the MongoDB wire protocol. Since it’s binary, we convert to hex.

Name: Mongo

Regex:^240100000000000000000000d407

Convert to hex: Toggle on.

Convert length: 14

HTTP
​

By default, AppScope uses an internally-defined protocol detector for HTTP similar to this example.

Name: HTTP

Regex:"HTTP\\/1\\.[0-2]|PRI \\* HTTP\\/2\\.0\r\n\r\nSM\r\n\r\n"

StatsD
​

By default, AppScope uses an internally-defined protocol detector for StatsD similar to this example.

Name: STATSD

Regex: "^([^:]+):([\\d.]+)\\|(c|g|ms|s|h)"

TLS
​

By default, AppScope uses an internally-defined protocol detector for TLS/SSL similar to this example.

Name: TLS

Regex: "^(?:(?:16030[0-3].{4})|(?:8[0-9a-fA-F]{3}01))"

Convert to hex: Toggle on.

Convert length: 5