Skip to main content
Version: 3.2

TCP JSON

Cribl LogStream supports receiving of data over TCP in JSON format (see the protocol below).

Type: Push | TLS Support: YES | Event Breaker Support: No

Configuring Cribl LogStream to Receive TCP JSON Data

In the QuickConnect UI: Click + New Source, or click + Add beside Sources. From the resulting drawer's tiles, select [Push >] TCP JSON. Next, click either + Add New or (if displayed) Select Existing. The drawer will now provide the following options and fields.

Or, in the Data Routes UI: From the top nav of a LogStream instance or Group, select Data > Sources. From the resulting page's tiles or the Sources left nav, select [Push >] TCP JSON. Next, click + Add New to open a New Source modal that provides the following options and fields.

LogStream ships with a TCP JSON Source preconfigured to listen on Port 10070. You can clone or directly modify this Source to further configure it, and then enable it.

General Settings

Input ID: Enter a unique name to identify this TCP JSON Source definition.

Address: Enter hostname/IP to listen for TCP JSON data. E.g., localhost or 0.0.0.0.

Port: Enter the port number to listen on.

IP allowlist regex: Regex matching IP addresses that are allowed to establish a connection. Defaults to .* (i.e., all IPs).

Authentication Settings

Use the Authentication method buttons to select one of these options:

  • Manual: Use this default option to enter the shared secret that clients must provide in the authToken header field. Exposes an Auth¬†token field for this purpose. (If left blank, unauthenticated access will be permitted.) A Generate link is available if you need a new secret.

  • Secret: This option exposes an Auth¬†token (text secret) drop-down, in which you can select a stored secret that references the authToken header field value described above. The secret can reside in LogStream's internal secrets manager or (if enabled) in an external KMS. A Create link is available if you need a new secret.

TLS Settings (Server Side)

Enabled defaults to No. When toggled to Yes:

Certificate name: Name of the predefined certificate.

Private key path: Server path containing the private key (in PEM format) to use. Path can reference $ENV_VARS.

Passphrase: Passphrase to use to decrypt private key.

Certificate path: Server path containing certificates (in PEM format) to use. Path can reference $ENV_VARS.

CA certificate path: Server path containing CA certificates (in PEM format) to use. Path can reference $ENV_VARS.

Authenticate client (mutual auth): Require clients to present their certificates. Used to perform mutual authentication using SSL certs. Defaults to No. When toggled to Yes:

  • Validate client certs: Reject certificates that are not authorized by a CA in the CA¬†certificate path, or by another trusted CA (e.g., the system's CA). Defaults to¬†No.

  • Common name: Regex matching subject common names in peer certificates allowed to connect. Defaults¬†to¬†.*. Matches on the substring after CN=. As¬†needed, escape regex tokens to match literal characters. E.g., to match the subject CN=worker.cribl.local, you would enter: worker\.cribl\.local.

Minimum TLS version: Optionally, select the minimum TLS version to accept from connections.

Maximum TLS version: Optionally, select the maximum TLS version to accept from connections.

In a LogStream Cloud deployment, do not set the TLS Settings (Server Side) tab's Enabled slider to Yes, nor configure any of the tab's resulting TLS fields. Any settings that you configure here would conflict with the LogStream Cloud Source's predefined TLS configuration.

Processing Settings

Fields (Metadata)

In this section, you can add fields/metadata to each event, using Eval-like functionality.

Name: Field name.

Value: JavaScript expression to compute field's value (can be a constant).

Pre-Processing

In this section's Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.

Advanced Settings

Enable Proxy Protocol: Enable if the connection is proxied by a device that supports Proxy Protocol v1 or v2.

Environment: If you're using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Connected Destinations

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source's data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

Internal Fields

Cribl LogStream uses a set of internal fields to assist in handling of data. These "meta" fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

Field for this Source:

  • __inputId
  • __srcIpPort

Format

LogStream expects TCP JSON events in newline-delimited JSON format:

  1. A header line. Can be empty ‚Äď e.g., {}. If authToken is enabled (see above) it should be included here as a field called authToken. When authToken is not set, the header line is optional. In this case, the first line will be treated as an event if does not look like a header record.

    In addition, if events need to contain common fields, they can be included here under fields. In the example below, region and AZ will be automatically added to all events.

  2. A JSON event/record per line.

    Sample TCP JSON Events
    {"authToken":"myToken42", "fields": {"region": "us-east-1", "AZ":"az1"}}

    {"_raw":"this is a sample event ", "host":"myHost", "source":"mySource", "fieldA":"valueA", "fieldB":"valueB"}
    {"host":"myOtherHost", "source":"myOtherSource", "_raw": "{\"message\":\"Something informative happened\", \"severity\":\"INFO\"}"}

TCP JSON Field Mapping to Splunk

If a TCP JSON Source is routed to a Splunk destination, fields within the JSON payload are mapped to Splunk fields. Fields that do not have corresponding (native) Splunk fields become index-time fields. For example, let's assume we have a TCP JSON event as below:

{"_time":1541280341, "host":"myHost", "source":"mySource", "_raw":"this is a sample event ", "fieldA":"valueA"}

Here, _time, host, and source become their corresponding fields in Splunk. The value of _raw becomes the actual body of the event, and fieldA becomes an index-time field (fieldA::`valueA``).

Examples

Testing TCP JSON In

This first example simply tests that data is flowing in through the Source:

  1. Configure Cribl LogStream to listen on port 10001 for TCP JSON. Set authToken to myToken42.
  2. Create a file called test.json with the payload above.
  3. Send it over to your Cribl LogStream host: cat test.json | nc <myCriblHost> 10001

LogStream to LogStream Cloud

This second example demonstrates using TCP JSON to send data from one LogStream instance to a downstream LogStream Cloud instance. We assume that the downstream Cloud instance uses LogStream Cloud's default TCP JSON Source configuration.

So all the configuration happens on the upstream instance's TCP¬†JSON Destination. Replace the <tenant‚ÄĎID> placeholder with the tenant¬†ID from your Cribl¬†Cloud portal.

TCP JSON Destination Configuration

On the upstream LogStream instance's Destination, set the following field values to match the target Cloud instance's defaults:

General Settings

Address: in.logstream.<tenant‚ÄĎID>.cribl.cloud ‚Äď you can simply copy/paste your Cribl¬†Cloud portal's Ingest¬†Endpoint here

Port: 10070

TLS Settings (Client Side)

Enabled: Yes

Validate server certs: Yes