Bug 1838925 - Browsertime tests for IndexedDB. r=perftest-reviewers,s…

…parky,janv,canaltinova,dom-storage-reviewers,taskgraph-reviewers,ahal

Differential Revision: https://phabricator.services.mozilla.com/D181330

dom/indexedDB/test/perfdocs/config.yml

@@ -0,0 +1,8 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+---
+name: IndexedDB
+manifest: None
+static-only: True
+suites: {}

dom/indexedDB/test/perfdocs/index.rst

@@ -0,0 +1,66 @@
+=============================
+IndexedDB Performance Testing
+=============================
+
+How to run tests on CI:
+-----------------------
+* Windows: ``mach try perf --show-all -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"``
+* Linux: ``mach try perf --show-all -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"``
+* Mac: ``mach try perf --show-all -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"``
+* All but 32-bit jobs: ``mach try perf --chrome --safari --show-all -q 'shippable-browsertime-indexeddb !32'``
+* In general:
+
+  * Open test selection interface with ``mach try perf --show-all``
+  * Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above)
+  * Note down the string used as a filter for rerunning the job (or rerun it with ``mach try again --list-tasks`` and ``mach try again --index``)
+
+How to run tests locally with the profiler?
+-------------------------------------------
+* Build the browser with release or release with debug symbols flags (not in debug mode)
+* Use ``mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000`` where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
+* After the test is complete, the generated profile is opened with the default browser.
+* The generated profile file path is listed also in the command line output.
+* For best symbolication results, it may help to
+
+  * run the same browser build that was used for the tests with ``./mach run``
+  * navigate to "profiler.firefox.com"
+  * use the "Load a profile from file" button
+
+How to compare performance to a different browser?
+--------------------------------------------------
+* The test outputs a ``time_duration`` value for all supported browsers
+* Using Chrome as an example,
+
+  * ``mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"``
+  * where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
+  * browser executable path after the ``-b`` argument varies locally
+  * in some cases, a test driver argument such as ``--browsertime-chromedriver`` may be required
+
+How to add more tests?
+----------------------
+* For the test boilerplate, copy and rename an old test script such as ``testing/raptor/browsertime/indexeddb_write.js`` under the ``testing/raptor/browsertime/`` directory
+* Modify the test case script argument of ``commands.js.run`` / Selenium's ``executeAsyncScript``
+
+  * Test parameters can be passed to such script with syntax ``${variable_name}`` where ``variable_name`` represents the parameter in the context of ``executeAsyncScript`` or ``commands.js.run``.
+  * Use quotes to capture a string value, for example ``"${variable_name}"``
+  * TIP: Debugging the test case could be simpler by serving it locally without the boilerplate
+
+* Add ``[test_name]`` section to file ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` where ``test_name`` **must be 10 characters or less** in order to be a valid ``Treeherder`` test name
+
+  * Under the ``[test_name]`` section, specify the test script name as a value of ``test_script =``
+  * Under the ``[test_name]`` section, specity the test parameters as a sequence of ``--browsertime.key=value`` arguments as a value of ``browsertime_args =``
+  * Under the ``[test_name]`` section, override any other values as needed
+
+* Add test as a subtest to run for Desktop ``taskcluster/ci/test/browsertime-desktop.yml`` (maybe also for mobile)
+* Add test documentation to ``testing/raptor/raptor/perfdocs/config.yml``
+
+* Generated files:
+
+  * Run ``./mach lint --warnings --outgoing --fix`` to regenerate the documentation and task files, and warn about omissions
+  * Running ``./mach lint -l perfdocs --fix .`` may also be needed
+
+* Testing:
+
+  * Test the new test by running it with the profiler
+  * Test the new test by running it with a different browser
+  * Test the new test by triggering it on CI

taskcluster/ci/test/browsertime-desktop.yml

@@ -802,3 +802,49 @@ browsertime-throttled:
     run-on-projects: []
     tier: 3
     treeherder-symbol: Btime(throttled)
+
+browsertime-indexeddb:
+    description: Raptor (browsertime) tests for IndexedDB
+    max-run-time: 2700
+    treeherder-symbol: Btime(idb)
+    limit-platforms:
+        by-app:
+            chrome:
+                - .*shippable.*
+            chromium:
+                - .*shippable.*
+            custom-car:
+                - .*64.*shippable.*
+            default: []
+    raptor:
+        apps: [firefox, chrome, chromium]
+        run-visual-metrics: false
+        subtests:
+            - addMab1
+            - addMabN
+            - addMar1
+            - addMarN
+            - addMbl1
+            - addMblN
+            - addkAB1
+            - addkABN
+            - addkAR1
+            - addkARN
+            - addkBL1
+            - addkBLN
+            - getkeyrng
+    run-on-projects:
+        by-app:
+            firefox:
+                by-test-platform:
+                    windows.*-32.*: []
+                    .*64.*shippable-qr/.*: [autoland, mozilla-central]
+                    default: []
+            default: []
+    tier:
+        by-app:
+            firefox: 2
+            default: 3
+    mozharness:
+        extra-options:
+            - --extra-profiler-run

taskcluster/ci/test/test-sets.yml

@@ -99,6 +99,7 @@ browsertime:
     - browsertime-speculative
     - browsertime-throttled
     - browsertime-pageload-benchmark
+    - browsertime-indexeddb
 
 browsertime-clang:
     - browsertime-tp6

testing/perfdocs/generated/IndexedDB.rst

@@ -0,0 +1,66 @@
+=============================
+IndexedDB Performance Testing
+=============================
+
+How to run tests on CI:
+-----------------------
+* Windows: ``mach try perf --show-all -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"``
+* Linux: ``mach try perf --show-all -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"``
+* Mac: ``mach try perf --show-all -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"``
+* All but 32-bit jobs: ``mach try perf --chrome --safari --show-all -q 'shippable-browsertime-indexeddb !32'``
+* In general:
+
+  * Open test selection interface with ``mach try perf --show-all``
+  * Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above)
+  * Note down the string used as a filter for rerunning the job (or rerun it with ``mach try again --list-tasks`` and ``mach try again --index``)
+
+How to run tests locally with the profiler?
+-------------------------------------------
+* Build the browser with release or release with debug symbols flags (not in debug mode)
+* Use ``mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000`` where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
+* After the test is complete, the generated profile is opened with the default browser.
+* The generated profile file path is listed also in the command line output.
+* For best symbolication results, it may help to
+
+  * run the same browser build that was used for the tests with ``./mach run``
+  * navigate to "profiler.firefox.com"
+  * use the "Load a profile from file" button
+
+How to compare performance to a different browser?
+--------------------------------------------------
+* The test outputs a ``time_duration`` value for all supported browsers
+* Using Chrome as an example,
+
+  * ``mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"``
+  * where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
+  * browser executable path after the ``-b`` argument varies locally
+  * in some cases, a test driver argument such as ``--browsertime-chromedriver`` may be required
+
+How to add more tests?
+----------------------
+* For the test boilerplate, copy and rename an old test script such as ``testing/raptor/browsertime/indexeddb_write.js`` under the ``testing/raptor/browsertime/`` directory
+* Modify the test case script argument of ``commands.js.run`` / Selenium's ``executeAsyncScript``
+
+  * Test parameters can be passed to such script with syntax ``${variable_name}`` where ``variable_name`` represents the parameter in the context of ``executeAsyncScript`` or ``commands.js.run``.
+  * Use quotes to capture a string value, for example ``"${variable_name}"``
+  * TIP: Debugging the test case could be simpler by serving it locally without the boilerplate
+
+* Add ``[test_name]`` section to file ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` where ``test_name`` **must be 10 characters or less** in order to be a valid ``Treeherder`` test name
+
+  * Under the ``[test_name]`` section, specify the test script name as a value of ``test_script =``
+  * Under the ``[test_name]`` section, specity the test parameters as a sequence of ``--browsertime.key=value`` arguments as a value of ``browsertime_args =``
+  * Under the ``[test_name]`` section, override any other values as needed
+
+* Add test as a subtest to run for Desktop ``taskcluster/ci/test/browsertime-desktop.yml`` (maybe also for mobile)
+* Add test documentation to ``testing/raptor/raptor/perfdocs/config.yml``
+
+* Generated files:
+
+  * Run ``./mach lint --warnings --outgoing --fix`` to regenerate the documentation and task files, and warn about omissions
+  * Running ``./mach lint -l perfdocs --fix .`` may also be needed
+
+* Testing:
+
+  * Test the new test by running it with the profiler
+  * Test the new test by running it with a different browser
+  * Test the new test by triggering it on CI

testing/perfdocs/generated/index.rst

@@ -8,6 +8,7 @@ Performance Testing
   :glob:
 
   DAMP
+  IndexedDB
   awsy
   fxrecord
   mach-try-perf
@@ -23,6 +24,7 @@ but `mozperftest </testing/perfdocs/mozperftest.html>`_ aims to provide this in
 For more detailed information about each test suite and project, see their documentation:
 
   * :doc:`DAMP`
+  * :doc:`IndexedDB`
   * :doc:`awsy`
   * :doc:`fxrecord`
   * :doc:`mach-try-perf`