Proviroll

Snapshots and Restore

Snapshot retention, ledger limits, and a restore workflow for Agave nodes.

A snapshot is a serialized checkpoint of validator state that a node can use to avoid replaying the full ledger from genesis during recovery or bootstrap. Snapshots reduce catch-up time, but only when their cadence, retention, storage, and restore path are tested before an incident.

Snapshot choices

The reference development profiles configure a full snapshot every 500 slots. Production values should be selected from measured recovery objectives and available storage rather than copied unchanged.

ControlWhat it changes
--full-snapshot-interval-slotsFrequency of full snapshots.
incremental snapshot settingsFrequency and retention of smaller deltas between full snapshots.
full/incremental retention flagsNumber of snapshots retained locally.
--limit-ledger-sizeBounds local ledger retention and disk use.

A conservative local policy may retain one full snapshot and two incremental snapshots. --maximum-local-snapshot-age and --no-snapshot-fetch are different controls: one limits the age of a local snapshot and the other changes whether a node fetches a snapshot at startup. Retention must be large enough to survive an interrupted download, a bad local snapshot, and the time needed to diagnose an incident.

Snapshot inventory

At least once per day, record the newest full and incremental snapshot slot, local size, storage location, and the last successful restore test. The important questions are not simply “does a snapshot file exist?” but “is it compatible, retrievable, and fast enough to meet the recovery objective?”

Restore runbook

  1. Preserve logs and record the failure time, validator version, and last observed slot.
  2. Stop the validator cleanly; do not delete data until the recovery path is chosen.
  3. Verify available disk capacity and the expected genesis hash.
  4. Bootstrap from a trusted compatible snapshot or allow the node to catch up according to the selected recovery strategy.
  5. Use solana-ledger-tool to verify the recovered ledger, inspect its bounds, and minimize it only when the recovery plan calls for it. The exact invocation varies by Agave release; confirm the available verify, bounds, and minimize subcommands with solana-ledger-tool --help before operating on a production ledger.
  6. Start the validator and verify health, slot progression, peer connectivity, and voting before returning it to service.

Validation after restore

solana cluster-version --url <local-rpc-url>
solana slot --url <local-rpc-url>
sleep 10
solana slot --url <local-rpc-url>
solana validators --url <local-rpc-url>

For a voting validator, also confirm the expected vote account is present and not delinquent. For an RPC-only node, verify the methods and history/index behavior that the application actually depends on.

Test the clock

For every environment, record the wall-clock time to restore from the intended snapshot source. A snapshot policy without a measured restore time is only a hypothesis.

Record time to download, unpack, replay, reach the allowed slot-lag threshold, and, where relevant, resume voting. Those are different stages and one slow stage tells you where to improve the runbook.

On this page