Skip to content

chore(bench): add analyzer benchmark suite and PR regression CI#116

Open
veksen wants to merge 1 commit intomainfrom
veksen/add-benchmarks-v3
Open

chore(bench): add analyzer benchmark suite and PR regression CI#116
veksen wants to merge 1 commit intomainfrom
veksen/add-benchmarks-v3

Conversation

@veksen
Copy link
Copy Markdown
Member

@veksen veksen commented Apr 18, 2026

Summary

  • Adds src/remote/optimizer.bench.ts — a vitest bench suite that measures QueryOptimizer.start() on small (3t/5q), medium (20t/100q), and large (300t/1000q) shapes against a single testcontainer Postgres.
  • Adds .github/workflows/benchmark.yaml — runs the bench on both PR HEAD and the merge-base, uses scripts/compare-bench.mjs to diff the two JSON reports, posts/updates a single PR comment with a regression table, and fails the check when any bench's mean regresses by >20%.
  • Wires up npm run bench and a benchmark.include pattern in vitest.config.ts.

Test plan

  • First PR run emits a "no baseline" comment (base commit predates the bench file) — confirms graceful fallback.
  • Subsequent PRs show a diff table with 🔴/🟢/⚪/🆕 verdicts and RME deltas.
  • npm run bench works locally.

🤖 Generated with Claude Code

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Doctor Analysis

View full run details

83 queries analyzed

2 pre-existing issues

Using assumed statistics (10000000 rows/table). For better results, sync production stats.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 18, 2026

Benchmark comparison

Threshold: ±20% on mean. 🔴 regression · 🟢 improvement · ⚪ within noise · 🆕 new/removed.

Benchmark Base mean PR mean Δ RME (base → PR)
src/remote/optimizer.bench.ts > query optimizer > large (300 tables, 1000 queries) 163,677ms 163,558ms -0.1% ±2.9% → ±2.5%
src/remote/optimizer.bench.ts > query optimizer > medium (20 tables, 100 queries) 2,575ms 2,584ms +0.4% ±5.0% → ±7.0%
src/remote/optimizer.bench.ts > query optimizer > small (3 tables, 5 queries) 89ms 93ms +4.6% ±10.2% → ±16.9%

Benchmarks use testcontainers + wall-time; some noise is expected. Treat single-digit deltas as not-significant.
Base commit: ``

@veksen veksen force-pushed the veksen/add-benchmarks-v3 branch from 992a893 to a9062f2 Compare April 20, 2026 07:56
Adds a vitest bench suite that measures QueryOptimizer.start() across
small/medium/large DB shapes via a single testcontainer Postgres, plus
a PR workflow that runs the bench on both HEAD and base, posts a
comparison comment, and fails the check on >20% regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@veksen veksen force-pushed the veksen/add-benchmarks-v3 branch from a9062f2 to 457f91e Compare April 20, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant