Cribl LogStream – Docs

Cribl LogStream Documentation

Questions? We'd love to help you! Meet us in #Cribl Community Slack (sign up)
Download entire manual as PDF - v2.3.2

Single-Instance Deployment

Getting started with Cribl LogStream on a single instance

For small-volume or light processing environments – or for test or evaluation use cases – a single instance of Cribl LogStream might be sufficient to serve all inputs, event processing, and outputs. This page outlines how to implement a single-instance deployment.

Architecture

Requirements

  • OS:
    • Linux: Red Hat, CentOS, Ubuntu, Amazon Linux (64bit)

⚠️

Mac OS is no longer supported as of v. 2.3, due to LogStream's incorporation of Linux-native components.

  • System:

We assume that 1 physical core is equivalent to 2 virtual/hyperthreaded CPUs (vCPUs). All quantities listed above are minimum requirements. To fulfill the above requirements using cloud-based virtual machines, see Recommended AWS, Azure, and GCP Instance Types.

  • Browser Support: Firefox 65+, Chrome 70+, Safari 12+, Microsoft Edge

Network Ports

By default, LogStream listens on the following ports:

Component

Default Port

UI

9000

HTTP In

10080

Splunk to Cribl LogStream data port

localhost:10000 (Cribl App for Splunk)

| criblstream Splunk search command to Cribl LogStream

localhost:10420 (Cribl App for Splunk)

User options

  • Other data ports as required.

Overriding Default Ports

The above ports can be overridden in the following configuration files:

  • Cribl UI port (9000): Default definitions for host, port, and other settings are set in $CRIBL_HOME/default/cribl/cribl.yml, and can be overridden by defining alternatives in $CRIBL_HOME/local/cribl/cribl.yml.

  • Data Ports: HTTP In (10080), TCPJSON in (10420) Splunk to Cribl (10000) : Default definitions for host, port and other settings are set in $CRIBL_HOME/default/cribl/inputs.yml, and can be overridden by defining alternatives in $CRIBL_HOME/local/cribl/inputs.yml.

Installing on Linux

  • Install the package on your instance of choice. Download it here.
  • Ensure that required ports are available (see Network Ports).
  • Un-tar in a directory of choice, e.g., /opt/:
    • tar xvzf cribl-<version>-<build>-<arch>.tgz

Running

Go to the $CRIBL_HOME/bin directory, where the package was extracted (e.g.: /opt/cribl/bin). Here, you can use ./cribl to:

  • Start: ./cribl start
  • Stop: ./cribl stop
  • Reload: ./cribl reload
  • Restart: ./cribl restart
  • Get status: ./cribl status
  • Switch a distributed deployment to single-instance mode: ./cribl mode-single (uses the default address:port 0.0.0.0:9000)

👍

For other available commands, see CLI Reference.

Next, go to http://<hostname>:9000 and log in with default credentials (admin:admin). You can now start configuring Cribl LogStream with Sources and Destinations, or start creating Routes and Pipelines.

📘

In the case of an API port conflict, the process will retry binding for 10 minutes before exiting.

Enabling Start on Boot

Cribl LogStream ships with a CLI utility that can update your system's configuration to start LogStream at system boot time. The basic format to invoke this utility is:

[sudo] $CRIBL_HOME/bin/cribl boot-start [enable|disable] [options] [args]

📘

Boot-start is supported only on Linux. For options and arguments, see the CLI Reference.

Newer systems use systemd to start processes at boot, while older ones use initd.

Using systemd

To enable Cribl LogStream to start at boot time with systemd, you need to run the boot-start command. If the user that you want to run LogStreams does not exist, create it prior to executing. E.g., running LogStream as user charlize on boot:

sudo $CRIBL_HOME/bin/cribl boot-start enable -m systemd -u charlize

This will install a unit file (as below) and start Cribl LogStream at boot time as user charlize. A ‑configDir option can be used to specify where to install the unit file. If not specified, this location defaults to /etc/systemd/system.

If necessary, change ownership for the Cribl LogStream installation:

[sudo] chown -R charlize $CRIBL_HOME

Next, use the enable command to ensure that the service starts on system boot:

[sudo] systemctl enable cribl

To disable starting at boot time, run the following command:

sudo $CRIBL_HOME/bin/cribl boot-start disable

[Unit]
Description=Systemd service file for Cribl LogStream.
After=network.target

[Service]
Type=forking
User=charlize
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
PIDFile=/install/path/to/cribl/pid/cribl.pid
ExecStart=/install/path/to/cribl/bin/cribl start
ExecStop=/install/path/to/cribl/bin/cribl stop
ExecStopPost='/bin/rm -f /install/path/to/cribl/pid/cribl.pid'
ExecReload=/install/path/to/cribl/bin/cribl reload
TimeoutSec=60

[Install]
WantedBy=multi-user.target

Using initd

To enable Cribl LogStream to start at boot time with initd, you need to run the boot-start command. If the user that you want to run LogStreams does not exist, create it prior to executing. E.g., running LogStream as user charlize on boot:

sudo $CRIBL_HOME/bin/cribl boot-start enable -m initd -u charlize

This will install an init.d script in /etc/init.d/cribl.init.d, and start Cribl LogStream at boot time as user charlize. A ‑configDir option can be used to specify where to install the script. If not specified, this location defaults to /etc/init.d.

If necessary, change ownership for the Cribl LogStream installation:

[sudo] chown -R charlize $CRIBL_HOME

To disable starting at boot time, run the following command:

sudo $CRIBL_HOME/bin/cribl boot-start disable

🚧

Do NOT Run LogStream as Root!

If LogStream is required to listen on ports 1–1024, it will need privileged access. On a Linux system with POSIX capabilities, you can achieve this by adding the CAP_NET_BIND_SERVICE capability. For example: # setcap cap_net_bind_service=+ep $CRIBL_HOME/bin/cribl

On some OS versions (such as CentOS), you must add an -i switch to the setcap command. For example: # setcap -i cap_net_bind_service=+ep $CRIBL_HOME/bin/cribl

Upon starting the LogStream server, a Port xxx is already in use error might indicate that setcap did not successfully execute.

System Proxy Configuration

You can direct all outbound HTTP/S requests to go through proxy servers. You do so by setting a few environment variables before starting LogStream, as follows:

Configure the HTTP_PROXY and HTTPS_PROXY environment variables either with your proxy's IP address, or with a DNS name that resolves to that IP address. Optionally, follow either convention with a colon and the port number to which you want to send queries.

HTTP_PROXY examples:

$ export HTTP_PROXY=http://10.15.20.25:1234
$ export HTTP_PROXY=http://proxy.example.com:1234

HTTPS_PROXY examples:

$ export HTTPS_PROXY=http://10.15.20.25:5678
$ export HTTPS_PROXY=http://proxy.example.com:5678

📘

Case Conflicts

The environment variables' names can be either uppercase or lowercase. However, if you set duplicate versions of the same name, the lowercase version takes precedence. E.g., if you've set both HTTPS_PROXY and https_proxy, the IP address specified in https_proxy will take effect.

Proxy Confguration with systemd

If you are proxying outbound traffic with systemd, list your proxy environment variables in the systemd unit file's [Service] section by adding statements of this form:

[Service]
...
Environment=https_proxy=<yourproxy>
Environment=https_proxy=http://proxy.example.com:1234
Environment=https_proxy=http://10.10.1.1:8080

This will prevent LogStream from throwing "failed to send anonymized telemetry metadata" errors.

Authenticating on Proxies

You can use HTTP Basic authentication on HTTP or HTTPS proxies. Specify the user name and password in the proxy URL. For example:

$ export HTTP_PROXY=http://username:[email protected]:1234
$ export HTTPS_PROXY=http://username:[email protected]:5678

Bypassing Proxies with NO_PROXY

If you've set the above environment variables, you can negate them for specified (or all) hosts. Set the NO_PROXY environment variable to identify URLs that should bypass the proxy server, and instead be sent as direct requests. Use the following format:

$ export NO_PROXY="<list of hosts/domains>"

Usage notes:

  • Within the list, separate the host/domain names with commas or spaces.

  • Optionally, each host/domain entry can be followed by a port. If specified, the port must match. If not specified, the protocol's default port is assumed.

  • If specified, subdomain names must match. E.g., NO_PROXY=foo.example.com will send requests directly to https://foo.example.com, but https://bar.example.com requests will go through the proxy.

  • You can use leading wildcards like NO_PROXY="*.us, .org".

  • NO_PROXY="*" disables all proxies.

  • NO_PROXY with an empty list disables no proxies.

Where Proxies Apply

Proxy configuration is relevant to the following LogStream components that make outbound HTTP/S requests:

Destinations

Sources

Collectors

Scaling Up

A single-instance installation can be configured to scale up and utilize as many resources on the host as required. See Sizing and Scaling for details.

Updated 3 days ago

Single-Instance Deployment


Getting started with Cribl LogStream on a single instance

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.