Troubleshooting
Diagnose Agave bootstrap, network, storage, snapshot, and validator-health failures without jumping straight to a reset.
Troubleshooting is the evidence-first process of locating a validator failure across process, storage, network, snapshot, and consensus state. Start with evidence: a process restart or ledger wipe can erase the exact state needed to explain a failure.
First five checks
Repeat the slot check after ten seconds. If it does not advance, record that result before changing anything.
Symptom guide
| Symptom | Likely classes of cause | Evidence to collect |
|---|---|---|
| Follower cannot join cluster | Wrong entrypoint, blocked UDP range, wrong expected genesis, or bootstrap unavailable | Gossip output, firewall rules, genesis hash, bootstrap service logs. |
| RPC listens but node is unhealthy | Replay/catch-up delay, storage pressure, stalled process, or wrong cluster | Slot progression, disk latency/space, logs, reference endpoint comparison. |
| Node repeatedly restarts | Service limits, permission problems, invalid flags, OOM, or storage failure | systemd status, exit code, kernel log, effective limits, mount ownership. |
| Leader slots are skipped | CPU scheduling, VM steal time, network loss, or disk latency | CPU governor, steal/IO wait, NIC counters, slot/performance samples. |
| Snapshot download never completes | Bad peer reachability, insufficient disk, incompatible genesis, or corrupted local state | Snapshot logs, disk capacity, genesis hash, peer/gossip visibility. |
| Validator is delinquent | Voting key/configuration problem, excessive lag, or service failure | getVoteAccounts, validator logs, local/reference slot lag. |
Safe escalation path
- Read logs and collect status.
- Check configuration drift: version, flags, genesis hash, entrypoints, trusted validators, and mounts.
- Check host limits and resource pressure.
- Check network reachability and peer discovery.
- Attempt a controlled restart only when the failure mode supports it.
- Restore from snapshot or rebuild only after preserving evidence and confirming the target genesis.
Dangerous shortcuts
- Do not delete ledger/accounts directories to fix a generic “not healthy” alert.
- Do not set
--wal-recovery-mode skip_any_corrupted_recordas a reflex; understand the data-integrity consequence for the node and version in use. - Do not disable known-validator or expected-genesis protections just to make a follower connect.
- Do not expose RPC publicly to test a networking problem.