Whenever a likely regression is detected for a benchmark, relevant people should be notified about this in an email.
A notification feature is considered important to increase the usage of the BM tool and thereby also collect more/better feedback and user requirements.
A rudimentary initial version of the feature could be to add a special "notification admin page" in the web app that allows anyone to edit the contents of the following table:
|Benchmark pattern||Platform pattern||Branch pattern|
Whenever a new regression is detected for a particular benchmark, an email with the appropriate details is sent to all subscribers matching the benchmark, platform, and branch. The notification event is then recorded to prevent retriggering.
The email could contain the following:
- A link to the time series of the change (such as this)
- A link to a page where the change could be classified/annotated:
- The change is valid (e.g. confirmed by manual inspection)
- The change is invalid (for this or that reason)
In order to minimize reporting false positives, it is important to let testing of a candidate benchmark run for a few more snapshots before reporting. I.e. if the regression was just an outlier (caused by an external effect, e.g. in the test environment), the results would soon fall back to the previous level.
Ideally, the notification mechanism should also check if the regression occurs on more than one platform, and include details for each of these in the email.