enable complex arguments for "npm run render-goldens" command (google…

…#4542)

* enable complex line arguments for update-renderer command, including support for --missing-only, --quiet and --dry-run

* remove debug messages.

* allow setting the port.

* support --scenario being a subset of the scenario name and case insensitive

* add cli for render fidelity tests, allow dry run, quiet modes.

* better description

* enable quite mode in CI.  add missing types for yargs

* improve readme for new update-screenshots cli.

* update readme with new "npm run test" CLI options.

* rename "npm run update-screenshots" to "npm run render-goldens", more accurate name.

* make which renderer you fidelity testing a CLI parameter, defaults to "model-viewer"

* be clear which renderers are supported, just web-based ones.

* update readme with new renderer choice for fidelity test.

* remove debug code.

.github/workflows/fidelity-tests.yml

@@ -29,4 +29,4 @@ jobs:
     - name: Fidelity tests
       uses: GabrielBB/[email protected]
       with:
-        run: npm run test --workspace=@google/model-viewer-render-fidelity-tools
+        run: npm run test --workspace=@google/model-viewer-render-fidelity-tools -- --quiet

packages/render-fidelity-tools/README.md

@@ -12,7 +12,7 @@ on [modelviewer.dev](https://modelviewer.dev).
 
 To get started, follow the instructions in [the main README.md file](../../README.md).
 
-In order to run `npm run update-screenshots`, you must have `imagemagick` installed.
+In order to run `npm run render-goldens`, you must have `imagemagick` installed.
 
 On macOS, you can install `imagemagick` with homebrew: `brew install imagemagick`
 
@@ -25,9 +25,49 @@ Command                         | Description
 `npm run build`                 | Build artifacts required to run testing and review tools
 `npm run clean`                 | Deletes all build artifacts
 `npm run test`                  | Run a render fidelity check comparing `<model-viewer>` to other renderers
-`npm run update-screenshots`    | Update the "golden" screenshots for all renderers
+`npm run render-goldens`        | Render the "golden" images for all renderers
 
-You can specify the scenarios when running `npm run test` and `npm run update-screenshots` by appending scenarios. (e.g. `npm run update-screenshots khronos-BoomBox khronos-FlightHelmet` will only update screenshots of these two scenarios) For `npm run update-screenshots`, you can further specify which renderers you want to update screenshots for. (e.g. `npm run update-screenshots khronos-BoomBox babylon` means update screenshots for khronos-BoomBox only with babylon)
+### Fidelity Test CLI
 
-After running `npm run test`, you can look at the results by starting a local web server
-(e.g. `npx http-server`) in this folder, then opening `test/results-viewer.html` in a browser.
+For `npm run test`, we support these command line options:
+
+Command            | Default             | Description
+--------------------|----------- | -----------
+  -c, --config      |  | Path to configuration json.
+  -r, --renderer    | model-viewer | The renderer to fidelity test, must be a web-based renderer.
+  -s, --scenario    |  | Limit to specific scenarios. This now also allows you to specify multiple scenarios in the whitelist.  You can use a full name or a partial name of scenarios and it will match against all that contain that scenario substring.
+  -p, --port        | 9040  | Port for web server.
+  -d, --dry-run      | false | Lists which images would be rendered but doesn't render.  Useful when trying to figure out which tests will run given that command line.
+  -q, --quiet        | false | Hide the puppeteer controlled browser.  This can allow you to work on the computer while it is running in the background.
+
+To run a subset of scenarios do something like this:
+
+```
+% npm run test -- --scenario=texture --quiet
+```
+
+*NOTE: In the above example, the `--` is required to separate the arguments to `npm run test` from the arguments to the script itself.*
+
+After running `npm run test`, you can look at the results by starting a local web server (e.g. `npx http-server`) in this folder, then opening `test/results-viewer.html` in a browser.
+
+### Render Goldens CLI
+
+For `npm run render-goldens`, we support these command line options:
+
+Command            | Default             | Description
+--------------------|----------- | -----------
+  -c, --config      |  | Path to configuration json.
+  -r, --renderer    |  | Limit to specific renderers. This now allows you to specify multiple renderers in the whitelist, rather than only one.
+  -s, --scenario    |  | Limit to specific scenarios. This now also allows you to specify multiple scenarios in the whitelist.  You can use a full name or a partial name of scenarios and it will match against all that contain that scenario substring.
+  -p, --port        | 9040  | Port for web server.
+  -m, --missing-only | false | Only render if an output image is missing.  Very useful when adding new tests.
+  -d, --dry-run      | false | Lists which images would be rendered but doesn't render.  Useful when trying to figure out which tests will run given that command line.
+  -q, --quiet        | false | Hide the puppeteer controlled browser.  This can allow you to work on the computer while it is running in the background.
+
+To run a subset of renders or scenarios do something like this:
+
+```
+% npm run render-goldens -- --renderer=filament --renderer=model-viewer --scenario=clearcoat -q 
+```
+
+*NOTE: In the above example, the `--` is required to separate the arguments to `npm run render-goldens` from the arguments to the script itself.*

packages/render-fidelity-tools/package.json

@@ -5,9 +5,9 @@
   "version": "0.0.1",
   "description": "<model-viewer> rendering fidelity test suite and tools",
   "scripts": {
-    "test": "node --experimental-modules ./lib/workflows/test-fidelity.js ./test/config.json",
+    "test": "node --experimental-modules ./lib/workflows/test-fidelity.js --config=./test/config.json",
     "test:ci": "./scripts/ci-check-fidelity.sh",
-    "update-screenshots": "node --experimental-modules ./lib/workflows/update-screenshots.js ./test/config.json",
+    "render-goldens": "node --experimental-modules ./lib/workflows/render-goldens.js --config=./test/config.json",
     "build": "tsc && rollup -c",
     "prepare": "if [ ! -L './shared-assets' ]; then ln -s ../shared-assets ./shared-assets; fi",
     "clean": "rm -rf ./dist ./lib ./test/results ./test/renderers/filament/{cmgen,ktx}"
@@ -44,15 +44,17 @@
     "puppeteer": "^21.3.4",
     "rimraf": "^3.0.2",
     "three": "^0.157.0",
-    "three-gpu-pathtracer": "^0.0.13"
+    "three-gpu-pathtracer": "^0.0.13",
+    "yargs": "^17.7.2"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^22.0.1",
     "@rollup/plugin-node-resolve": "^13.3.0",
     "@rollup/plugin-replace": "^4.0.0",
+    "@types/yargs": "^17.0.29",
     "polymer-build": "^3.1.4",
     "rollup": "^2.77.2",
     "rollup-plugin-external-globals": "^0.6.1",
     "typescript": "4.8.4"
   }
-}
+}