Cribl - Docs

Getting started with Cribl LogStream

Questions? We'd love to help you! Meet us in #cribl (sign up)

Changelog    Guides

Splunk Load Balanced

Splunk is a streaming destination type. When Cribl is installed/co-located in a Splunk heavy forwarder, with Splunk Load Balanced output you can load balance data out to multiple Splunk receivers.

How does load balancing work


Cribl will attempt to load balance outbound data as fairly as possibly across all receivers. Data is sent to all receivers simultaneously and the amount sent to each depends on these parameters:

  1. Respective destination weight
  2. Respective destination historical data

By default, historical data is tracked for 300s and it is used to influence the traffic sent to each destination so as to ensure that differences decay over time and total ratios converge towards configured weights.

Example:
Suppose we have two receivers, A and B each with weight of 1 i.e. they are configured to receive equal amount of data. Suppose further that the load balance stats period is set at default 300s and, to make things easy, for each period there are 200 events of equal size (Bytes) that need to be balanced.

Interval
Time Range
Events to be dispensed

1

time=0s ---> time=300s

200

Both A and B start this interval with 0 historical stats each
Let's assume that due to various circumstances 200 events are "balanced" as follows:
A = 120 events and B = 80 events a difference of 40 events and a ratio of 1.5:1

Interval
Time Range
Events to be dispensed

2

time=300s ---> time=600s

200

At the beginning of interval 2, the load balancing algorithm will look back to the previous interval stats and carry half of the receiving stats forward. I.e. A will start the interval with 60 and B with 40. To determine how many events A and B receive during this interval, Cribl will use their weights and their stats as follows:

Total number events: events to be dispensed + stats carried forward = 200 + 60 + 40 = 300
Number of events per each destination (weighed): 300/2 = 150 (they're equal due to equal weight)
Number of events to send to each destination A: 150 - 60 = 90 and B: 150 - 40 = 110

End of interval 2 totals: A=120+90=210, B=80+110=190, a difference of 20 events and a ratio of 1.1:1.

Over the subsequent intervals, the difference becomes exponentially less pronounced and insignificant and thus the load gets balanced fairly.

Configuring Cribl to output to load balance to multiple Splunk destinations


While on Destinations screen, select Splunk Load Balanced from the vertical menu, then click Add New:

  • Output Id: Enter a unique name to identify this Splunk LB destination definition.
  • DNS Resolution Period (s): Re-resolve any hostnames every this many seconds and pick up destinations from A records. Defaults to 60s.
  • Destinations: Set of Splunk receivers where to load balance data to.
    • Host: Hostname of the Splunk receiver.
    • Port: Port number to send data to.
    • TLS: Whether to inherit TLS configs from group setting or disable TLS. Defaults Inherit.
    • TLS Servername: Servername to use if establishing a TLS connection. If not specified defaults to connection host (iff not an IP), otherwise the global TLS settings.
    • Weight: The weight to use for load balancing purposes.
  • Load Balance Stats Period (s): Lookback traffic history period. Defaults at 300s.
  • Exclude Current Host IPs: Exclude all IPs of the current host from the list of any resolved hostnames. Defaults to Yes.

TLS Settings (client side)

  • Disabled defaults to Yes. When toggled to No:
    • Validate Server Certs: Require client to reject connections to servers whose certs are not signed by one of the supplied CAs. Defaults to No.
    • Server Name (SNI): Server Name Indication.
    • CA Certificate Path : Path on client where to find CA certificates to use to verify the server's cert in PEM format. Path can reference $ENV_VARS.
    • Private Key Path (mutual auth): Path on client where to find the private key to use in PEM format. Path can reference $ENV_VARS. Use only if mutual auth is required.
    • Certificate Path (mutual auth) : Path on client where to find certificates to use in PEM format. Path can reference $ENV_VARS. Use only if mutual auth is required.

Then, click Save.

Note on DNS A Records: If multiple receivers are behind a hostname (i.e. multiple A records) all resolved IPs will inherit the weight the host, unless each IP is specified separately. In Cribl load balancing, IP settings take priority over those from hostnames.

SSL Configuration for Splunk Cloud - Special Note


To connect to Splunk Cloud you'll need to extract the private and public key from the Splunk provided Splunk Cloud Certificate (typically bundled in an app)

Step 1: Test connectivity to Splunk Cloud using the Root CA certificate
openssl s_client -CApath path_to_ca.pem -connect hostnameToSplunkCloud:9997

Step 2: Extract the Private key from Splunk Cloud Certificate. At the prompt you will need the sslPassword value in outputs.conf bundled with the Splunk Cloud app.
openssl ec -in path_to_server_cert.pem -out private.pem

Step 3: Extract the Public Key for Server Certificate
openssl x509 -in path_to_server_cert.pem -out server.pem

Step 4: In Cribl, in the destination TLS section enter the following:

  • CA Certificate Path: Path to CA Certificate
  • Private Key Path (mutual auth): Path to private.pem (above)
  • Certificate Path (mutual auth): Path to server.pem (above)