Now that we have a benchmarks branch that will run nightly, we can build upon this. It is not uncommon for someone to introduce a change, and for us to want to validate the benchmarks before merging. We should create automation where you can add a tag to a PR, and benchmarks will run, followed by a bot that will publish a comment on the PR with the results and much they deviate from the baseline on the benchmarks branch.
We will have to see how the final results look in terms of how accurate they will be, since we don't have much control over the hosts running the tests.
Requirements:
- When a "Run benchmarks" label is added to a PR, a github action job is triggered
- The action will run the "ci" benchmark job for the particular branch
- The action will diff the results with the latest info on the
benchmarks branch
- The action will leave a comment on the PR with a summary of the results and their delta compared to the baseline
Now that we have a
benchmarksbranch that will run nightly, we can build upon this. It is not uncommon for someone to introduce a change, and for us to want to validate the benchmarks before merging. We should create automation where you can add a tag to a PR, and benchmarks will run, followed by a bot that will publish a comment on the PR with the results and much they deviate from the baseline on thebenchmarksbranch.We will have to see how the final results look in terms of how accurate they will be, since we don't have much control over the hosts running the tests.
Requirements:
benchmarksbranch