Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-217

Add notification feature to BM tool

    XMLWordPrintable

Details

    • Task
    • Resolution: Out of scope
    • P1: Critical
    • 2012q4
    • None
    • Performance tracking
    • None

    Description

      Whenever a likely regression is detected for a benchmark, relevant people should be notified about this in an email.

      A notification feature is considered important to increase the usage of the BM tool and thereby also collect more/better feedback and user requirements.

      A rudimentary initial version of the feature could be to add a special "notification admin page" in the web app that allows anyone to edit the contents of the following table:

      Email Benchmark pattern Platform pattern Branch pattern
      user.name1@nokia.com tst_QGraphicsView:* linux* *4.7*
      user.name1@nokia.com QObjectBenchmark:* linux* *4.7*
      user.name2@nokia.com * * qt/qt-script-v8#master-v8

      ...

      Whenever a new regression is detected for a particular benchmark, an email with the appropriate details is sent to all subscribers matching the benchmark, platform, and branch. The notification event is then recorded to prevent retriggering.

      The email could contain the following:

      • A link to the time series of the change (such as this)
      • A link to a page where the change could be classified/annotated:
        • The change is valid (e.g. confirmed by manual inspection)
        • The change is invalid (for this or that reason)
        • ...

      In order to minimize reporting false positives, it is important to let testing of a candidate benchmark run for a few more snapshots before reporting. I.e. if the regression was just an outlier (caused by an external effect, e.g. in the test environment), the results would soon fall back to the previous level.

      Ideally, the notification mechanism should also check if the regression occurs on more than one platform, and include details for each of these in the email.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              bdo Jo Asplin (Inactive)
              bdo Jo Asplin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 4 weeks
                  4w
                  Remaining:
                  Remaining Estimate - 4 weeks
                  4w
                  Logged:
                  Time Spent - Not Specified
                  Not Specified

                  Gerrit Reviews

                    There are no open Gerrit changes