Blockchain event backfill is what keeps a temporary listener disconnect from becoming a permanent hole in your database. A live subscription can resume with new transfers while yesterday’s missing events stay unnoticed.
For an EVM indexer, treat subscriptions as a prompt to look for work. Durable checkpoints and range queries establish what you’ve actually processed. Let’s build that recovery path before adding more subscription reconnect logic.
Checkpoint completed work
Store the chain ID, filter version, last completed block number and its hash. Advance the checkpoint only after the corresponding events and application updates are committed or durably scheduled.
If you update the checkpoint first, a crash can skip events. If you store events first and the checkpoint update is lost, replay is acceptable when your ingestion is idempotent.
Keep separate checkpoints for different filters or consumers. Adding a new contract address today doesn’t mean the old checkpoint proves its historical events were scanned.
Read a bounded range after reconnecting
Start from a point before the last checkpoint according to your overlap policy, then query a bounded block range. Providers may limit range size or result count, so split requests and handle explicit limit errors.
Use the EVM JSON-RPC log fields and filter semantics your client supports. A subscription notification alone doesn’t prove that all previous logs were delivered. Ethereum JSON-RPC reference.
The overlap intentionally rereads some events. That’s simpler to reason about than pretending the connection always ended at an exact, fully applied boundary.
Give observations a full identity
Include chain, block hash, transaction hash and log index. Keep block number for range operations, but don’t use it as the sole branch identity.
An event reappearing in an overlapping scan should not create another business effect. An observation from an orphaned block should remain inspectable while no longer contributing to the canonical view.
Build overlap and branch replacement
This example scans a range, scans it again and then replaces one provisional block. The new canonical view keeps only events from the current branch.
import assert from 'node:assert/strict';
const observed=new Map();
let headers=new Map([[1,'a'],[2,'b']]),checkpoint=0;
const key=e=>JSON.stringify([1,e.hash,e.tx,e.index]);
function ingest(events,to){for(const e of events)observed.set(key(e),{...e});checkpoint=to;}
const old=[{height:1,hash:'a',tx:'t1',index:0},{height:2,hash:'b',tx:'t2',index:0}];
ingest(old,2);ingest(old,2);
assert.equal(observed.size,2);
const current=()=>[...observed.values()].filter(e=>headers.get(e.height)===e.hash);
headers.set(2,'c');
ingest([{height:2,hash:'c',tx:'t3',index:0}],2);
assert.deepEqual(current().map(e=>e.tx),['t1','t3']);
assert.equal(observed.size,3);assert.equal(checkpoint,2);
console.log('PASS: overlapping backfill deduplicates observations and filters orphaned logs');
The checkpoint follows the completed range, and repeated ingestion doesn’t duplicate an observation. The fixture doesn’t connect to an RPC service. It tests the local identity and projection rules that a real scanner needs.
Commit event ingestion and progress together
For a small range, store the events and checkpoint in one database transaction. For larger work, use a durable batch record and advance the checkpoint only when the batch is complete.
Downstream effects may happen asynchronously. In that case, distinguish “indexed through block X” from “all business effects applied through block X”. A single misleading checkpoint can hide a large consumer backlog.
Use an outbox or equivalent recoverable publication when moving ingested events into a broker. Don’t acknowledge the scan batch as fully scheduled before that obligation is durable.
Detect reorganisation rather than only duplicates
Compare stored block hashes with the canonical chain as you rescan. If a mismatch appears, find the common ancestor and invalidate affected observations before rebuilding the projection.
An overlap window is a practical recovery aid, not a proof that deeper changes cannot happen. Define a response when a mismatch extends beyond the retained window or contradicts your trusted finality checkpoint.
For deposits and other financial effects, separate provisional observations from released credit. Recovering a missed log and releasing a balance are different operations.
Make provider problems distinguishable
A request can fail because of rate limiting, a large range, a lagging node or an unsupported historical query. Record the provider, requested range and error class.
Don’t advance the checkpoint on an empty result if the request actually failed and your adapter replaced the error with an empty array. That small convenience can create a silent data gap.
When switching providers, verify chain identity and the relevant block hash. A working endpoint on another network is worse than a clear outage.
Test gaps, repeats and partial commits
Disconnect the live listener while fixture transfers occur, then recover through range scans. Deliver the same logs twice and replace a provisional block. Check the canonical projection and checkpoint after each step.
Crash after storing part of a batch and before progress commits. On restart, the scanner should replay safely rather than assume the whole range was completed.
Also change the contract filter and verify that a new backfill can cover its required history. A filter version in the checkpoint makes that operational decision visible.
Start by making your last-completed checkpoint durable and hash-aware. Once a reconnect can explain exactly which range it will reread, subscriptions become a latency improvement rather than the only thing protecting your data.
Run the example and adapt it
Save the code as 26-recover-missed-chain-events.mjs and run it with Node.js 24 or later:
node 26-recover-missed-chain-events.mjs
The code above includes assertions you can run locally. It tests the local decision logic with controlled inputs; it doesn’t connect to a live provider or establish production performance. Keep that boundary when adapting it to your application.
