feat(host-reth): replace ExEx backfill with direct DB reads by prestwich · Pull Request #136 · init4tech/node-components

prestwich · 2026-04-10T16:34:50Z

Summary

Replace ExEx-driven backfill (which re-executes blocks) with direct DB reads from the reth provider, fixing slow startup and memory issues
New DbBackfill<P> reads blocks+receipts in batches up to finalized, then hands off to the ExEx stream for live blocks
HostChain enum unifies backfill and live chain segments behind Extractable
set_head documented and enforced as once-only across both notifier implementations

Details

The ExEx backfill mechanism re-executes historical blocks on startup, which is slow and has caused memory issues. The reth DB already contains executed results, making re-execution unnecessary.

New types (signet-host-reth):

DbBlock / DbChainSegment — owned block+receipts from DB, implements Extractable
DbBackfill<P> — batch reader using spawn_blocking for MDBX reads
HostChain — enum wrapping DbChainSegment (backfill) and RethChain (live)

Notifier changes:

RethHostNotifier::next_notification is now two-phase: drain DB backfill, then switch to ExEx
set_head creates a DbBackfill instead of calling ExEx set_with_head
ExEx stream is initiated after backfill completes, pointed at the last backfilled block
reth-stages-types dependency removed

Cross-crate:

HostNotifier::set_head trait doc clarified as once-only
RpcHostNotifier::set_head guards against repeated calls

Review follow-ups

Fraser: set_backfill_thresholds(None) now resets to DEFAULT_BATCH_SIZE via a new DbBackfill::reset_batch_size, matching the trait contract and RpcHostNotifier.
Evalir: Removed the genesis fallback after backfill completion. The ExEx startup race it was defending against was fixed upstream in reth (#19665 / #22168, merged Feb 2026), and at our call site DbBackfill has just successfully read last_backfilled from the same provider — so a missing header there now indicates DB-level failure and returns RethHostError::MissingHeader.

Closes ENG-1784

Test plan

cargo clippy passes (both --all-features and --no-default-features)
RUSTDOCFLAGS="-D warnings" cargo doc passes
All existing tests pass (signet-host-reth, signet-node-types, signet-host-rpc)
signet-node compiles cleanly with new HostChain type
Integration test on a reth node with historical data (manual)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Introduces three types for reading host-chain blocks from the reth DB: - `DbBlock`: owned block+receipts pair from the provider - `DbChainSegment`: newtype over `Vec<DbBlock>` implementing `Extractable` using the same `RecoveredBlockShim` transmute pattern as `RethChain` - `DbBackfill<P>`: batch reader that walks from a cursor to the finalized block, recording metrics per batch via `crate::metrics` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds `HostChain` enum with `Backfill(DbChainSegment)` and `Live(RethChain)` variants, both delegating to the inner `Extractable` impl. Promotes `DbChainSegment` to `pub` and re-exports both new types from the crate root. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace reth's built-in ExEx backfill with DbBackfill for startup catch-up. The notifier now runs a two-phase loop: phase 1 drains DB batches via DbBackfill, then phase 2 switches to live ExEx notifications. set_head initializes backfill instead of resolving a header directly, and set_backfill_thresholds configures DbBackfill batch size. Chain type changes from RethChain to HostChain enum. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fraser999 · 2026-04-14T08:50:49Z

+        if let Some(backfill) = &mut self.backfill
+            && let Some(max_blocks) = max_blocks
+        {
+            debug!(max_blocks, "configured DB backfill batch size");
+            backfill.set_batch_size(max_blocks);
+        }


The trait method's doc comment says "None means use the backend's default.", and the RpcHostNotifier does this, so we should probably reset the batch size to the default here too in the None case.

Evalir · 2026-04-15T15:40:41Z

+            const {
+                assert!(
+                    size_of::<RecoveredBlockShim>() == size_of::<RethRecovered>(),
+                    "RecoveredBlockShim layout diverged from RethRecovered"
+                );
+                assert!(
+                    align_of::<RecoveredBlockShim>() == align_of::<RethRecovered>(),
+                    "RecoveredBlockShim alignment diverged from RethRecovered"
+                );
+            }


this is very nice

Evalir · 2026-04-15T16:00:14Z

+                    let backfill = self.backfill.take().expect("backfill was Some");
+                    let last_backfilled = backfill.cursor().saturating_sub(1);
+
+                    let head = self


so, this behavior of fetching the last seen block, and if that fails, fall back to genesis, stems from a probably still unresolved bug where Reth starts the exex before its connected to its own db provider (that's roughly what I remember). This ofc complicates the logic quite a bit by adding the genesis fallback path.

Considering that most of the time we might be falling into phase 1 when restarting the node to catch up with a few blocks, maybe we can simplify this?

- `set_backfill_thresholds(None)` now resets the batch size to `DEFAULT_BATCH_SIZE` via a new `DbBackfill::reset_batch_size`, matching the trait contract ("`None` means use the backend's default") and the `RpcHostNotifier` implementation. - Remove the genesis fallback after backfill completion. The documented ExEx startup race it was defending against (reth #19665 / #22168) was fixed upstream, and in any case `DbBackfill` just read `last_backfilled` from the same provider, so a missing header at this point indicates DB-level failure. Now returns `RethHostError::MissingHeader` instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

prestwich and others added 8 commits April 10, 2026 12:17

feat(host-reth): add MissingBlock and MissingReceipts error variants

59b25cf

feat(host-reth): add backfill metrics module

728c4af

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs(node-types): clarify set_head is once-only

34fd03a

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(host-rpc): guard set_head against repeated calls

0a39089

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(host-reth): fix private doc link in DbChainSegment

3ae38bd

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

prestwich requested a review from a team as a code owner April 10, 2026 16:34

prestwich requested review from Evalir and Fraser999 April 10, 2026 16:36

Fraser999 reviewed Apr 14, 2026

View reviewed changes

Evalir reviewed Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(host-reth): replace ExEx backfill with direct DB reads#136

feat(host-reth): replace ExEx backfill with direct DB reads#136
prestwich wants to merge 9 commits intomainfrom
prestwich/db-backfill

prestwich commented Apr 10, 2026 •

edited

Loading

Uh oh!

Fraser999 Apr 14, 2026

Uh oh!

Evalir Apr 15, 2026

Uh oh!

Evalir Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

prestwich commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Review follow-ups

Test plan

Uh oh!

Fraser999 Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Evalir Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Evalir Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prestwich commented Apr 10, 2026 •

edited

Loading