Storage and OS Tuning

Storage and OS tuning is the host-level configuration that gives a validator enough disk throughput, network buffers, memory mappings, and file descriptors to keep up with the cluster. Agave performance problems are often storage, networking, or operating-system-limit problems before they are client problems.

Storage layout

Keep operating-system, ledger, and accounts data on separate storage where possible. Ledger and accounts are write-heavy and recovery-sensitive; fast local NVMe is the default target for those paths. Record disk growth per day for the actual workload and set capacity alerts before the disk becomes an incident.

The upstream Agave guidance also separates accounts, ledger, snapshots, and optionally the OS. Its baseline recommendation is high-end NVMe for accounts, ledger, and snapshots; accounts and ledger should not share a disk in a production validator because of their IO profile. See the Agave requirements for current hardware guidance.

Measure disk growth

Record a daily sample for every mounted path:

df -h <ledger-dir> <accounts-dir> <snapshots-dir>
du -sh <ledger-dir> <accounts-dir> <snapshots-dir>

Do not publish a generic “GB per day” number until it is measured against a named network, version, enabled RPC indexes, snapshot retention policy, and transaction load. All of those inputs materially change disk growth.

Applied host limits

These values form a host-service baseline. A Kubernetes container does not receive a systemd unit or host PAM limits automatically, so verify effective limits from inside the actual validator container as part of its deployment acceptance.

The official validator setup additionally sets fs.nr_open=1000000 and recommends a memory-lock limit for the service/session. A containerized validator may require suitable capabilities and runtime limits to achieve the same result. Confirm all active limits from the process that starts Agave; writing a configuration file without reloading systemd or logging in again does not change an already-running process, and it does not prove the container inherited the intended limits.

systemctl show <validator-service> -p LimitNOFILE -p LimitMEMLOCK
ulimit -n
sysctl net.core.rmem_max net.core.wmem_max vm.max_map_count fs.nr_open

For a Kubernetes validator, execute the equivalent ulimit -n and sysctl checks in the running container and record the result. Do not rely on the presence of the host script alone.

Additional checks

• Disable swap for validator hosts unless there is a deliberate, tested reason not to.
• Use a performance-oriented CPU governor and verify the setting survives reboot.
• Measure disk latency and throughput on the actual mounted paths, not only on an unused device.
• Alert on filesystem usage, inode pressure, IO wait, and validator restart count.

These settings are a baseline, not a substitute for capacity testing on the hardware that will run the node.

Failure signatures