Troubleshooting
Diagnose rollup incidents by locating the failed stage of the execution-to-settlement pipeline.
Start with the user symptom, then locate the pipeline stage where progress stopped. Do not restart every service at once: that destroys the timing and queue evidence needed to distinguish an execution issue from a settlement, proving, data availability, or bridge incident.
| Symptom | First checks | Likely boundary |
|---|---|---|
| Transactions accepted but no new blocks | Sequencer health, queue depth, execution logs, database capacity. | Sequencing or execution. |
| Blocks advance but state is not settling | Batch queue, submitter wallet funds, L1 RPC, pending transaction state. | Batch submission or settlement. |
| Proofs or challenges are delayed | Worker capacity, queue age, circuit version, verifier response. | Proving or challenge system. |
| Deposits or withdrawals are delayed | Bridge events, relay queue, finality state, liquidity, destination-chain status. | Bridge and settlement. |
| Users cannot reconstruct state | Data-publication status, retrieval path, retained batches, provider health. | Data availability. |
For every incident, preserve logs, queue state, transaction hashes, contract events, configuration version, and the timeline of operator actions. Reconcile user-facing balances and messages before declaring recovery complete.
Triage order
- Establish the affected chain, component, time window, and user operation.
- Check the last successful event at each boundary: execution, publication, proof or challenge, settlement, relay, and indexing.
- Compare the component's local state with an independent source such as a settlement-layer transaction, DA reader, or bridge event.
- Decide whether to continue intake, rate-limit, pause a specific route, or declare a user-facing incident.
- Apply the smallest reversible corrective action and observe the backlog before making another change.
Evidence checklist
Capture the effective configuration, deployed artifact digest, database and queue state, logs with timestamps, settlement transaction hashes, bridge message identifiers, provider responses, and current access-control state. If a key or credential may be involved, rotate or revoke it through the approved process. Do not paste it into an incident channel or a debugging document.