Skip to content

Align snapshot regeneration workflow with VRT runtime#1358

Merged
lukasoppermann merged 2 commits intomainfrom
copilot/fix-vrt-errors
Apr 21, 2026
Merged

Align snapshot regeneration workflow with VRT runtime#1358
lukasoppermann merged 2 commits intomainfrom
copilot/fix-vrt-errors

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 21, 2026

Summary

VRT failures were caused by environment drift: the snapshot update workflow generated baselines differently than the VRT verification workflow.
This change makes snapshot regeneration run with the same sharding, dependency setup, build scope, and Playwright runtime as VRT.

List of notable changes:

  • updated .github/workflows/update_visual_snapshots.yml to use 5 shards ([1,2,3,4,5]) to match visual_regression_test.yml
  • updated install/build steps to mirror VRT runner:
    • npm ci && npm run storybook:install
    • npm run build
  • updated snapshot test execution to run in mcr.microsoft.com/playwright:v1.56.1-jammy with host-bridge STORYBOOK_URL, matching VRT networking model
  • updated workflow docs inline with a short note explaining why Docker uses 172.17.0.1
- name: Run Visual Regression Tests
  uses: docker://mcr.microsoft.com/playwright:v1.56.1-jammy
  env:
    # Tests run in Docker; Storybook runs on host runner
    STORYBOOK_URL: 'http://172.17.0.1:6006'
  with:
    args: npx playwright test --update-snapshots --shard="${{ matrix.shard }}/${{ strategy.job-total }}"

What should reviewers focus on?

  • Workflow parity between:
    • .github/workflows/update_visual_snapshots.yml
    • .github/workflows/visual_regression_test.yml
  • Correctness of Docker/host networking assumption for Storybook access
  • Whether snapshot update output now reflects what VRT actually validates

Steps to test:

  1. On a PR with color-token changes, add the update snapshots label.
  2. Confirm Snapshots workflow runs and commits regenerated snapshots.
  3. Re-run Visual Regression Testing and verify prior false diffs are resolved.

Supporting resources (related issues, external links, etc):

Contributor checklist:

  • All new and existing CI checks pass
  • Tests prove that the feature works and covers both happy and unhappy paths
  • Any drop in coverage, breaking changes or regressions have been documented above
  • All developer debugging and non-functional logging has been removed
  • Related issues have been referenced in the PR description

Reviewer checklist:

  • Check that pull request and proposed changes adhere to our contribution guidelines and code of conduct
  • Check that tests prove the feature works and covers both happy and unhappy paths
  • Check that there aren't other open Pull Requests for the same update/change
  • Verify the design tokens changed in this PR are expected using the diffing results in this PR

Copilot AI and others added 2 commits April 21, 2026 12:54
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 21, 2026

⚠️ No Changeset found

Latest commit: 1056a75

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@lukasoppermann lukasoppermann marked this pull request as ready for review April 21, 2026 13:00
@lukasoppermann lukasoppermann requested a review from a team as a code owner April 21, 2026 13:00
@lukasoppermann lukasoppermann added the skip changeset Apply to PRs that should not result in a version bump. label Apr 21, 2026
@github-actions github-actions Bot temporarily deployed to Preview (Storybook) April 21, 2026 13:03 Inactive
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Aligns the snapshot regeneration workflow with the Visual Regression Testing (VRT) runtime to prevent baseline drift and reduce false diffs by running snapshots under the same environment and sharding model.

Changes:

  • Updates snapshot regeneration to use 5 shards to match VRT.
  • Mirrors VRT install/build steps (npm ci && npm run storybook:install, then npm run build).
  • Runs Playwright snapshot updates inside the same Playwright Docker image and uses the same host-bridge STORYBOOK_URL.
Show a summary per file
File Description
.github/workflows/update_visual_snapshots.yml Brings snapshot regeneration sharding, dependency setup, build, and Playwright runtime/networking in line with the VRT workflow.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

.github/workflows/update_visual_snapshots.yml:33

  • npm run storybook:install already runs npm ci inside docs/storybook (with --legacy-peer-deps). The additional npm ci --legacy-peer-deps in the Build storybook step is redundant, adds time, and can reintroduce drift vs visual_regression_test.yml (which just runs npm run build-storybook). Consider removing the extra npm ci here and relying on storybook:install for dependency setup.
      - name: Build storybook
        working-directory: docs/storybook
        run: |
          npm ci --legacy-peer-deps
          npm run build-storybook
  • Files reviewed: 1/1 changed files
  • Comments generated: 1

@@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The snapshot update job's matrix strategy doesn't set fail-fast: false. With sharding enabled, the default fail-fast behavior can cancel remaining shards on the first failure, which makes snapshot regeneration less reliable and can leave the PR with only a partial set of updated snapshots. Consider matching visual_regression_test.yml by explicitly setting fail-fast: false under strategy:.

Suggested change
strategy:
strategy:
fail-fast: false

Copilot uses AI. Check for mistakes.
@lukasoppermann lukasoppermann merged commit cd9f884 into main Apr 21, 2026
35 of 36 checks passed
@lukasoppermann lukasoppermann deleted the copilot/fix-vrt-errors branch April 21, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip changeset Apply to PRs that should not result in a version bump.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants