Skip to content

Defer instructionsToState evaluation in toTest (-62% wall time on a 12-worker run)#21

Open
CharlonTank wants to merge 1 commit intolamdera:mainfrom
CharlonTank:perf/lazy-toTest
Open

Defer instructionsToState evaluation in toTest (-62% wall time on a 12-worker run)#21
CharlonTank wants to merge 1 commit intolamdera:mainfrom
CharlonTank:perf/lazy-toTest

Conversation

@CharlonTank
Copy link
Copy Markdown
Contributor

Summary

toTest evaluates instructionsToState instructions eagerly while constructing the Test tree, only to read state.testName. Under elm-test's parallel runner, every worker process forces the whole test tree on startup, so each of the N workers re-runs the full E2E simulation for every test — even though each worker only reports results for its 1/N partition.

This PR defers the simulation to inside the Test.test lambda so that each worker only simulates the tests it actually runs. The test name is read via a new private extractTestName helper that walks the EndToEndTest chain without applying any NextStep / AndThen functions (cheap pattern unwrapping only).

Diff (essentials)

extractTestName : EndToEndTest ... -> String
extractTestName instructions =
    case instructions of
        Start state -> state.testName
        NextStep _ inner -> extractTestName inner
        AndThen _ inner -> extractTestName inner

toTest : EndToEndTest ... -> Test
toTest instructions =
    Test.test (extractTestName instructions)
        (\() ->
            let state = instructionsToState instructions   -- moved inside the lambda
            in case state.testErrors of ...
        )

Impact

Measured on a real-world Lamdera app with 14 E2E tests + 56 unit tests, 12 elm-test workers, cold runs on macOS arm64:

Wall (median of 3) Duration Passed
Before 83.0 s 62.0 s 70/70
After 31.2 s 9.4 s 70/70

That is −62 % wall time, −85 % cumulative test duration, with all tests still passing.

The before-numbers also explain previously-mysterious CPU figures: cumulative user CPU was ~580 s for an 83 s wall, i.e. each of the 12 workers was burning ~50 s of CPU re-doing simulation work it would never report. After the patch, workers only simulate their assigned tests.

Test plan

  • elm-format src/Effect/Test.elm (no diff)
  • lamdera make succeeds with no warnings
  • elm-review passes (no new errors; 4 pre-existing suppressions untouched)
  • Run the existing test suite: elm-test --compiler=lamdera — same passes/failures as before, faster wall time

Backwards compatibility

No public API change. The EndToEndTest constructors are not exposed from the module, the new extractTestName is private, and toTest's type signature is unchanged. Test names still display identically — they're just read via constructor unwrapping rather than running the simulation.

Notes

The test name walk reads the testName from the original Start state set by start. There is currently no public API in Effect.Test that renames a test mid-chain via a NextStep, so this is always the correct name. If a future API ever lets users rename tests via state updates, extractTestName would need to apply such updates — but that case doesn't exist today.

`toTest` was evaluating `instructionsToState instructions` eagerly while
constructing the Test tree, just to read `state.testName`. With elm-test's
parallel runner, every worker process forces the test tree on startup, so
each worker re-runs the full simulation for every E2E test even though it
only reports results for a small subset.

This commit moves the `instructionsToState` call inside the `Test.test`
lambda so the simulation is performed only when the test is actually
executed by its assigned worker. The test name is now extracted via a new
private `extractTestName` helper that walks the EndToEndTest chain
without applying any NextStep/AndThen functions.

Measured on a real-world Lamdera app with 14 E2E tests + 56 unit tests
on a 12-worker elm-test run: median wall time drops from 83s to 31s
(-62%) with all 70 tests still passing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant