Capacity and Performance Testing
Measure the workloads that make a Solana validator or RPC tier fail before users discover the limit.
Capacity planning establishes the workload a validator or RPC tier can serve while preserving its availability and data-correctness promises. There is no useful universal “Solana RPC requests per second” number: a cached getBalance and a wide historical getProgramAccounts query may both be one request while consuming radically different CPU, memory, disk, network, and retention resources.
Use the Agave hardware guidance as a starting point, then make local measurements the source of truth. It recommends fast NVMe storage and separates account, ledger, snapshot, and operating-system concerns because storage isolation affects failure and recovery behavior as much as raw throughput.
Establish the test contract
Write a test charter before choosing a load generator:
| Field | Example question |
|---|---|
| Cluster and build | Which Agave version, genesis, feature state, OS/kernel, instance type, and disk layout are being measured? |
| Node role | Voting validator, private RPC, public RPC, history RPC, or indexer-backed API? |
| API contract | Which methods, commitment levels, pagination limits, transaction sizes, and account-index features are advertised? |
| Traffic mix | What proportion of light reads, account scans, historical reads, simulations, and sends reflects production? |
| Success condition | What p95/p99 latency, error rate, slot lag, and recovery headroom are acceptable? |
| Stop conditions | Which CPU, memory, disk latency, packet-loss, queue-depth, or lag signals make more load unsafe? |
Record the date, test-data shape, warm/cold cache condition, and generated load definition beside every result. Otherwise a future version comparison only proves that two unlabelled experiments differed.
Test each role separately
Voting validator
The meaningful outcome is continued consensus participation, not a fast HTTP response. During a controlled test, measure vote behavior, root/slot advancement, skip performance, leader-slot behavior, process restarts, disk latency, NIC errors, CPU steal time on virtualized environments, and memory pressure. Do not use a voting validator as the target of an unbounded public-RPC benchmark.
Run disruptive maintenance and recovery exercises outside leader windows. The Agave best-practices guidance specifically calls out waiting for a validator to be outside leader slots before stopping it; agave-validator exit is intended to wait until it is no longer a leader.
RPC backend
Measure at least three states:
- Warm steady state: normal cache and a representative mix.
- Cold or post-restart: snapshot/replay complete but application caches are empty.
- Degraded dependency: one backend removed, high disk latency, or a history/index workload competing with ordinary reads.
Track slot lag against a reference endpoint. An RPC backend that returns HTTP success while serving an old slot is not healthy for all callers. Define whether traffic is removed at a fixed lag threshold, a time threshold, or a combination of getHealth, observed slot progress, and lag.
Public edge
Test connection churn, TLS handshakes, large bodies, batches, unauthenticated traffic, invalid JSON, quota exhaustion, and client retry behavior. Confirm the gateway enforces limits before the backend sees excess work. A proxy benchmark that uses only a cached health endpoint does not establish public RPC capacity.
Workload matrix
Build the matrix from actual application behavior. The following is a useful minimum set:
| Workload | What to vary | Signals to watch |
|---|---|---|
| Light reads | Concurrency, keep-alive, cache warmth | Edge latency, backend CPU, success rate |
| Account reads | Account size and getMultipleAccounts cardinality | Response bytes, memory, tail latency |
| Program scans | Filter types, result cardinality, index availability | Disk reads, IO wait, timeout rate |
| Token-owner/mint queries | Index choice and pagination | Index disk growth, replay/startup cost |
| Historical reads | Transaction/block age and range | History retention, disk pressure, p99 latency |
| Simulation/send | Transaction size, compute profile, duplicate/retry rate | Queue depth, upstream errors, submission reconciliation |
| Batch abuse | Batch cardinality and mixed cost | Actual backend work per HTTP request |
| Failure recovery | Snapshot restore, restart, catch-up | RTO, disk headroom, lag convergence |
For every test, preserve the exact command/configuration and a histogram or time series, not just a single maximum number. Include the lowest setting that causes an SLO breach; that knee is often more actionable than peak throughput.
Resource measurements
At a practical sampling interval, capture:
- CPU utilization broken down by user/system/iowait and, on VMs, steal time.
- Resident memory, cgroup limits, OOM events, page faults, and swap state.
- Per-device read/write throughput, latency, queue depth, capacity, inode availability, and filesystem errors for ledger, accounts, and snapshots.
- NIC bandwidth, drops/errors, UDP receive errors, retransmits where applicable, and connection counts at the edge.
- RPC method latency/error/response-size distributions and edge queue/concurrency state.
- Validator/RPC slot progression, reference-slot lag, vote status, leader performance, and restart count.
The storage and OS tuning page contains the baseline limits and verification commands. A test result without host-resource telemetry cannot reliably distinguish a software regression from an IO or network limit.
Find a safe operating point
Increase one dimension at a time until the first relevant SLO fails, then leave headroom below that point. Headroom must cover routine variance, leader periods, snapshot/replay, a backend loss, traffic bursts, and an upgrade, not merely the average observed load.
Do not set a rate limit at the highest successful benchmark rate. Choose a sustainable threshold after accounting for the worst repeatable latency, backend removal, and recovery state. Make the policy granular: a service may safely admit many light reads while needing a much lower concurrent limit for scans, simulations, or historical requests.
Regression suite
Run a smaller repeatable suite when any of these change:
- Agave/client version or feature configuration.
- Kernel, instance shape, CPU governor, NIC, cgroup, or filesystem configuration.
- Ledger/accounts/snapshot disks, retention policy, account indexes, or history settings.
- Proxy configuration, authentication layer, method allowlist, timeout, or rate limits.
- Application traffic mix or supported API methods.
Compare the same test charter, not raw results from mismatched hardware or data. Flag a regression in tail latency, error rate, slot lag, disk latency, recovery time, or resource headroom even when average throughput improved.
Publish the useful result
For an external service, publish a stable contract with supported methods, limits, retention semantics, and error/retry behavior, rather than hardware details or an unqualified RPS claim. Internally, retain the benchmark report, dashboards, load definition, configuration hash, and decision log. That separation gives users a dependable interface while preserving the evidence operators need to evolve it.