Skip to content

fix: report unique queries (matches site) instead of log-entry count in CI comment#107

Merged
veksen merged 3 commits intomainfrom
veksen/ci-vs-site-query-count
Apr 20, 2026
Merged

fix: report unique queries (matches site) instead of log-entry count in CI comment#107
veksen merged 3 commits intomainfrom
veksen/ci-vs-site-query-count

Conversation

@veksen
Copy link
Copy Markdown
Member

@veksen veksen commented Apr 3, 2026

Summary

The CI comment's "X queries analyzed" disagreed with the query list on the site — most dramatically on the sin-bot PR 12 run, which reported 4236 queries analyzed in the PR comment while the site showed 74.

Root cause: the reported count was computed at ingest time from the number of plan: log entries that passed filtering, not from the number of unique queries we actually analyze. The same 74 distinct queries ran ~57× each across 457 vitest specs, inflating the headline by ~57×.

This change:

  • Computes queryStats.analyzed (renamed from queryStats.total) using buildQueries(allResults, config).length — the exact same list used to upload to the site, so the headline always matches what the user sees.
  • Drops the separate total counter from the ingest loop.
  • Drops the countUploadableQueries / UPLOADABLE_STATES helpers — they duplicated the filter that already lives in buildQueries.
  • Restores the Matched N unique queries out of M log entries debug log by using recentQueries.length (no extra counter needed).
  • Updates ReportStatistics.totalReportStatistics.analyzed (docstring: "Number of unique queries analyzed and uploaded to the site"). Consumers updated in the success template and test fixtures.

Why this is the right fix

Before deciding on the dedup approach, we added a diagnostic step to sin-bot's CI workflow that split the log by phase and counted plan: / pg_catalog / information_schema / @qd_introspection lines. Results on the sin-bot run:

Phase plan: entries pg_catalog hits info_schema hits @qd_introspection
drizzle-kit migrate 41 38 0 0
npm test (457 specs) 4220 95 0 0

Key takeaways that shaped this PR:

  1. Migration phase is tiny. 41 plan entries. The initial theory that drizzle-kit introspection was inflating the count does not hold.
  2. No information_schema traffic reaches plan: entries in either phase.
  3. The 4220 test-phase entries are real application queries, executed by the test suite. 457 specs × ~9 queries each ≈ 4100.
  4. The headline mismatch is purely about dedup, not filtering. A pre-count isSystemQuery / information_schema filter would not have materially moved the number.

Given that, delegating to buildQueries is strictly better than any heuristic filter: there is one filter definition (in site-api.ts), and the CI comment is guaranteed to report whatever we actually uploaded.

Test plan

  • npx vitest run src/runner.test.ts src/reporters/github/github.test.ts passes (20/20)
  • Existing template rendering tests updated for the totalanalyzed rename
  • New runner.test.ts asserts buildQueries(allResults).length counts exactly the queries reported to the site on a mixed-state input
  • Re-run sin-bot PR 12 against this branch and confirm the PR comment reports 74 (or whatever the current analyzed count is) rather than a 4-digit number

Follow-up (not in this PR)

  • Consider surfacing the ingest-count elsewhere (e.g. in the Query Doctor run page) if users find the "executions vs. uniques" ratio useful for understanding test-suite shape.
  • The diagnostic write-up lives at .context/query-count-handoff.md in this workspace and documents the ruled-out hypotheses so we don't retrace them.

🤖 Generated with Claude Code

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Doctor Analysis

View full run details

2 queries analyzed

2 pre-existing issues

Using assumed statistics (10000000 rows/table). For better results, sync production stats.

@veksen veksen force-pushed the veksen/ci-vs-site-query-count branch from 4825d2a to 2d5aaa4 Compare April 3, 2026 05:58
Comment thread src/runner.ts Outdated
@Xetera Xetera force-pushed the main branch 2 times, most recently from 3aac4a7 to 7b0a1b9 Compare April 16, 2026 22:50
@veksen veksen changed the title fix: match CI query count to site by computing total from uploadable results fix: report unique queries (matches site) instead of log-entry count in CI comment Apr 20, 2026
veksen and others added 3 commits April 20, 2026 11:53
The `total` counter was incrementing during log parsing, counting raw
entries before deduplication and optimization. This caused the CI comment
to show a different query count than the site, which only displays
queries with uploadable states (improvements_available,
no_improvement_found, error).

Compute `total` from the deduplicated, optimized results filtered to
uploadable states so the CI comment and site always agree.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract UPLOADABLE_STATES and countUploadableQueries as exported
functions so they can be tested. Tests verify:
- Only improvements_available, no_improvement_found, error are counted
- not_supported, timeout, waiting, optimizing are excluded
- countUploadableQueries agrees with buildQueries for every state

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…alyzed

Drop UPLOADABLE_STATES / countUploadableQueries — they duplicated the
filter in buildQueries. Use buildQueries(allResults, config).length
directly so the CI comment count is derived from the same list we
upload to the site.

Rename queryStats.total to queryStats.analyzed across the type, the
success template, and test fixtures.

Restore the `Matched N unique queries out of M log entries` debug
log using recentQueries.length (no separate counter).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@veksen veksen force-pushed the veksen/ci-vs-site-query-count branch from 9984e70 to d007327 Compare April 20, 2026 07:54
@ChrisHarris2012
Copy link
Copy Markdown
Contributor

Consider surfacing the ingest-count elsewhere (e.g. in the Query Doctor run page) if users find the "executions vs. uniques" ratio useful for understanding test-suite shape.

Perhaps the text 2 queries analyzed should be more along the lines of 2 unique query shapes analyzed or something?

@veksen
Copy link
Copy Markdown
Member Author

veksen commented Apr 20, 2026

Consider surfacing the ingest-count elsewhere (e.g. in the Query Doctor run page) if users find the "executions vs. uniques" ratio useful for understanding test-suite shape.

Perhaps the text 2 queries analyzed should be more along the lines of 2 unique query shapes analyzed or something?

Happy to review the text, created follow-up: #120

@veksen veksen merged commit fc62327 into main Apr 20, 2026
6 checks passed
@veksen veksen deleted the veksen/ci-vs-site-query-count branch April 20, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants