On This Page

Home / Stream/ Setting Up Cribl Stream/ Deployment Planning/OS Tuning for Large Deployments

OS Tuning for Large Deployments

Large on-prem deployments that handle high connection volumes or data rates may require operating system (OS) tuning to maintain optimal performance. To prevent performance issues on high-load Worker Nodes, you might need to tune your OS.

Review and apply the following OS-level configurations to address common bottlenecks like connection tracking table exhaustion and buffer overruns, ensuring stable operation for your Cribl environment.

The parameters described in this section are Linux kernel and operating system settings, not Cribl Stream configuration options. They directly affect how the OS handles network connections and system resources for high-throughput Cribl Stream workloads.

Connection Tracking (Netfilter)
​

Applies to: Worker Nodes (handling high volumes of inbound network connections.)

In deployments where you expect a high number of inbound connections, Linux netfilter connection tracking (conntrack) could become a bottleneck. Connection tracking is enabled by default on most Linux distributions and maintains a table of active network flows. When this table fills up, the kernel drops packets, which can cause connection failures and data loss.

When to Tune
​

Monitor your system logs (/var/log/messages or journalctl \-k) for messages indicating conntrack table exhaustion:

nf_conntrack: table full, dropping packet

If you observe these messages, you should increase the conntrack table size and potentially adjust related parameters.

Key Conntrack Parameters
​

These sysctl parameters govern connection tracking behavior:

  • net.netfilter.nf_conntrack_max: Maximum number of flows tracked concurrently. By default, it derives from the buckets value: nf_conntrack_max= nf_conntrack_buckets × 4.

  • net.netfilter.nf_conntrack_buckets: Hash buckets for the conntrack table. If not set at module load, the kernel computes a default from total system memory (roughly total memory in bytes / 16384) with floor and ceiling limits applied.

For the full list of netfilter conntrack sysctls and their defaults, see Netfilter Conntrack Sysfs variables.

Check Current Values and Live Load
​

Use the following commands to inspect your current conntrack configuration and active connection count:

# Current max entries
sysctl net.netfilter.nf_conntrack_max

# Current active tracked flows (count)
sysctl net.netfilter.nf_conntrack_count

# Alternatively via procfs
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

Compare the count against the max. If the count approaches or equals the max during normal operation, you need to increase capacity.

Method 1 (Recommended): Increase the conntrack Table Size
​

This is the standard and generally recommended approach to eliminate conntrack bottlenecks by simply giving the table more capacity.

To ensure stability, set the maximum number of tracked flows high enough to maintain adequate margin above your peak observed load. For very large deployments handling hundreds of thousands of concurrent connections, you may need values of 4 million or higher.

# For example, to set approximately 2 million tracked connections (adjust per sizing)
echo "net.netfilter.nf_conntrack_max=2097152" >> /etc/sysctl.conf
sysctl -p

Adjust the value based on your deployment’s actual connection load.

If you substantially increase nf_conntrack_max, review nf_conntrack_buckets for balance. By default, the kernel derives buckets from available memory, and max defaults to approximately 4 times the bucket count. Maintaining this ratio helps ensure efficient hash table performance.

Method 2 (Extreme Case): Disable Connection Tracking
​

In extremely high-throughput, purpose-built environments where Cribl Stream is the only service and network overhead is a top concern, disabling conntrack entirely can be considered an aggressive measure. While this is often the simplest and most effective measure for some on-prem deployments, approach it with caution due to its side effects.

How to Disable conntrack (Persistent)
​

To prevent the module from loading at boot:

echo "install nf_conntrack /bin/true" >> /etc/modprobe.d/disable-conntrack.conf

Disabling conntrack is NOT generally recommended and carries significant consequences:

  • Breaks stateful firewalls: Most modern firewall rules (iptables, nftables) rely on conntrack for stateful filtering. Disabling it will break stateful firewalls, requiring a complete rewrite of your rules to be stateless.
  • Breaks NAT/Kubernetes: Network Address Translation (NAT) and IP masquerading are heavily dependent on conntrack. Disabling it will break networking for container technologies like Kubernetes and Docker.

Kubernetes Deployments
​

If you run Cribl Stream on Kubernetes, do not disable netfilter or conntrack. Kubernetes networking (including kube-proxy and network policies) depends on conntrack for proper operation. Tune conntrack parameters as described in Method 1 instead.

File Descriptor Limits
​

Applies to: Leader and Worker Nodes

The Cribl Stream systemd unit sets LimitNOFILE=65536 by default, which aligns with our minimum recommendation. This limit applies per process (not systemwide or per user), meaning the API and each Worker Process can each open up to 65,536 file handles. This is generally more than sufficient for high-volume workloads, including those involving:

  • File-based Destinations (S3, Filesystem, etc.), especially with partition expressions that possess a high cardinality. For details, see Amazon S3 Better Practices.
  • Large numbers of concurrent inbound and outbound connections

For more information on running Cribl Stream as a service, see Run Cribl Stream.

Diagnosing Too Many Open Files Error
​

The default value is typically sufficient for normal operation. Increasing the limit is generally not the correct fix; it usually only postpones the necessity of identifying and resolving the root cause of the process hitting the resource ceiling.

Start your investigation by determining which process is affected. The cribl.log file will reflect a too many open files error, referencing one of two error codes:

  • EMFILE (Too many open files): This means the per-process limit has been reached. This is the common case that the LimitNOFILE setting addresses.
  • ENFILE (File table overflow): This means the systemwide (kernel-level) limit for all open files has been reached. The fix for this (increasing the fs.file-max sysctl) is outside the scope of this section, as it’s rarely encountered.

The scenario that leads to the common EMFILE error is usually the API Process being overwhelmed due to Worker Processes being too busy. The API Process always receives the connection attempt first, then hands the file descriptor off to a Worker Process.

The EMFILE condition typically occurs when a large number of clients (on the order of thousands) establish connections that are not closed have excessively long timeouts. Since Worker Processes are often unable to accept the handoff of these file descriptors from the API, the API Process accumulates thousands of connections waiting to be serviced. This accumulation of active or stuck connections rapidly exhausts the file descriptor limit for the API Process. The proper fix, therefore, is to address why the Worker Processes are too busy.

Start by analyzing your Pipelines using the Pipeline Profiling feature to identify bottlenecks and reduce Worker load. For details, see Pipeline Profiling.

How to Increase the Per-Process Open File Limit (if Necessary)
​

If, after investigation, you’ve confirmed that the workload legitimately requires more file handles (such as, in extremely large, high-volume, or unique deployments), you can increase this per process limit via a systemd override:

# Create or edit the systemd override
sudo systemctl edit cribl

Add the following (example) content:

[Service]
LimitNOFILE=262144

Save the file and restart the Cribl Stream service for the change to take effect:

sudo systemctl restart cribl

You can verify the current limit for running processes by checking /proc/<pid\>/ limits for any Cribl Stream process.

# For exampalefor PID 7284
cat /proc/7284/limits

UDP Receive Buffers
​

Applies to: Worker and Edge Nodes (Nodes hosting UDP input Sources).

For high-rate UDP ingest (such as Syslog, NetFlow, or IPFIX), the default kernel UDP receive buffers may be insufficient, leading to packet drops before data reaches Cribl Stream. Increasing these buffers can significantly reduce packet loss.

When to Tune
​

Cribl recommends tuning UDP buffers to accommodate short-lived but large data spikes. Tuning UDP buffers is recommended to accommodate short-lived but large data spikes.

Before tuning the buffer size, you should confirm that the data is being properly distributed across all the Worker Nodes in the Group. If the data is improperly load-balanced and only hitting a subset of your Nodes, you must address the distribution issue first. Fixing the load balancing might resolve the packet drop problem entirely, making buffer adjustments unnecessary.

For permanent solutions to consistently high data volume, scaling your environment (adding more worker processes or nodes) is the correct approach.

Monitor for UDP packet drops using:

# On modern operating systems, check for packet receive errors using ss
ss -ua
# Alternatively, if ss is unavailable
netstat -su

Look for increases in packet receive errors or RcvbufErrors under the UDP section.

Method 1: Kernel-Level Tuning (System-Wide OS Configuration)
​

This method applies the buffer increase to all UDP sockets on the system.

Increase the kernel’s UDP receive buffer sizes:

# Set to 25 MB (adjust based on your volume)
echo "net.core.rmem_max=26214400" >> /etc/sysctl.conf
echo "net.core.rmem_default=26214400" >> /etc/sysctl.conf
# Apply the changes immediately without rebooting:
sysctl -p

Method 2: Per-Input via Cribl Stream Configuration
​

We generally recommend configuring the buffer size within Cribl Stream (on the specific UDP Source) for two main benefits:

  • Portability: The setting becomes part of your Cribl Stream configuration, ensuring it is automatically applied to newly deployed Nodes without requiring manual OS tuning on each one.
  • Scope: The larger buffer is applied only to the specific UDP sockets used by Stream Sources that have this setting defined, avoiding the general application to all system sockets.

Start with 25 MB and adjust based on your data rates and observed drop rates.

For detailed UDP tuning guidance specific to Cribl Stream, see:

SELinux Configuration
​

Applies to: Leader, Worker, and Edge Nodes.

In hardened environments running SELinux in enforcing mode, ensure proper SELinux contexts are set on Cribl directories to allow Stream to operate correctly. Use SELinux audit logs to troubleshoot any denials and adjust policies as needed.

For detailed guidance on running Cribl Stream with SELinux, see SELinux Configuration.