Operational Checklists

Operational checklists turn an otherwise ad-hoc validator procedure into a repeatable set of observable checks. They are not a substitute for observation, but they make the minimum evidence consistent during a restart, upgrade, or incident when it is easiest to skip a dependency. Adapt thresholds to the cluster and record the owner for every unchecked item.

New-node preflight

• Hardware, kernel, storage layout, mount options, nofile, vm.max_map_count, network buffers, CPU governor, and swap state satisfy the host baseline.
• Ledger, accounts, and snapshots have distinct, documented capacity budgets; the filesystem owner can write every configured path.
• Identity, vote, authorized-voter, stake, faucet, and withdraw authority locations follow the key hierarchy. The withdraw authority is not present on a routine remote node or Pod.
• Binary version and checksums/provenance are recorded, with a known-good rollback artifact available.
• Firewall rules permit the documented gossip, dynamic UDP, TPU/TVU, RPC, metrics, and administration flows, with no broader ranges.
• Entrypoints, known validators, expected genesis hash, and RPC exposure mode have been checked against the intended cluster.
• Systemd service limits, restart policy, log retention, time synchronization, monitoring, and alert routing are active before the node is trusted.

Join and readiness acceptance

solana cluster-version --url <local-rpc-url>
solana genesis-hash --url <local-rpc-url>
solana slot --url <local-rpc-url>
sleep 10
solana slot --url <local-rpc-url>
solana validators --url <local-rpc-url>

• The reported genesis matches the intended cluster.
• Slot advances over the sample interval and is within the documented lag threshold of a reference endpoint.
• Gossip/trusted-peer configuration is in effect; a follower did not join through an unintended cluster.
• The identity and, for a voter, vote account are the expected public keys.
• RPC, metrics, and public edge exposure match the documented role; direct backend access is denied where applicable.
• Dashboards record a baseline for disk capacity/latency, slot lag, vote status, restarts, and RPC behavior.

Daily health review

• Slot progresses and lag is within threshold; inspect a trend, not just one value.
• Voting validators are not delinquent and show expected vote / leader behavior.
• Ledger, accounts, and snapshot volumes have sufficient capacity and normal latency; no filesystem or NVMe errors are present.
• CPU iowait/steal, memory pressure/OOM events, UDP drops, NIC errors, and process restarts are normal.
• Snapshot creation/retention behaves as expected and the latest usable restore point is known.
• Public RPC error rates, tail latency, queue/concurrency, response sizes, method mix, and rate-limit events are normal.
• Alerts are delivered to a tested destination; disabled/silenced alerts have an owner and expiry.

Planned restart or upgrade

1. Capture version, genesis, slot, vote/validator state, host resource baseline, configuration revision, and a current log window.
2. Confirm snapshot/restore viability, disk headroom, rollback package, and the exact target release.
3. For a voting node, check the leader schedule and choose a safe window. Avoid a hard process kill; follow the maintenance sequence.
4. Drain public RPC traffic from the node or backend pool before changing it.
5. Start one node, observe replay/catch-up and acceptance checks, then proceed in a controlled order.
6. Compare before/after metrics and leave the change record with timing, config diff, rollback decision, and anomalies.

Snapshot restore exercise

• Declare the RPO and RTO being tested.
• Preserve the original failed state long enough to capture logs and configuration evidence.
• Restore into a controlled target with the correct genesis and enough ledger/accounts/snapshot capacity.
• Time each stage: artifact acquisition, verification, startup, replay/catch-up, and ready-for-traffic.
• Verify solana genesis-hash, slot progression, lag convergence, vote behavior if applicable, and API behavior before admitting traffic.
• Record the actual wall-clock result and the reason for variance from the RTO. Update snapshots and restore when the procedure changes.

Security review

• Review SSH, cloud, Kubernetes, and service-account access to hosts, volumes, keys, dashboards, and backup locations.
• Confirm the validator/RPC process cannot read unnecessary private keys and that key file modes/ownership remain correct after automation runs.
• Confirm public RPC has TLS, authentication where required, per-key/source quotas, body/batch/time limits, method policy, and privacy-reviewed logs as described in RPC security.
• Rotate credentials through a rehearsed procedure; do not discover authority relationships during an incident.
• Revalidate firewall/security-group intent and remove expired temporary access.

Incident handover

When handing an incident to another operator, include the incident start time, cluster/genesis, node role, identity public key, current and reference slot, current version/configuration revision, last known healthy time, logs/metrics links, changes made, and explicit next action. Do not share private key material, raw API credentials, or unrestricted infrastructure access in the handover record.