-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update snapshots and test configs #16126
base: master
Are you sure you want to change the base?
Conversation
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/16126/merge/index.html#WGZLGJ#4600 Links to test babylon tools with this snapshot: https://playground.babylonjs.com/?snapshot=refs/pull/16126/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/16126/merge#BCU1XR#0 |
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/16126/merge/ |
You have made possible changes to the playground. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/16126/merge/ The snapshot playground with the CDN snapshot (only when available): Note that neither Babylon scenes nor textures are uploaded to the snapshot directory, so some playgrounds won't work correctly. |
WebGL2 visualization test reporter: |
Visualization tests for WebGPU |
cc @MiiBond to look into the pbr based tests who look different following the latest changes. Goal is to validate they are not regressions but expected changes. |
WebGL2 tests fail I tested “water-material”, “FrameGraph nrge depth of field” and “GLTF Extension KHR_materials_volume with attenuation” locally, after making the same modifications as in your PR (threshold = 0.03, maxDiffPixelRatio = 1% - I rebuilt the sources by running They also worked in WebGPU, except for “GLTF Extension KHR_materials_volume with attenuation”, which had a ratio of 3% different pixels. As this test is not flagged as an error by the CI in WebGPU mode, I'd tend to think that the errors we get for these tests are due to the difference between the backend we use on the CI and the backend used when generating the expected screenshot (usually Chrome or Edge on a desktop browser). What do you think? WebGPU tests that fail You can update the “terrain-erosion” image, it seems to be outdated, and the differences are not due to the latest changes in lighting conditions, I can see a difference with a version as old as 7.30.0. "FrameGraph glow layer“ and "FrameGraph highlight layer" tests: the base model (”NeonPipe.gltf") renders slightly differently between WebGL and WebGPU. You can see this by browsing https://playground.babylonjs.com/#9YU4C5#2 and https://playground.babylonjs.com/?webgpu#9YU4C5#2 and switching between tabs: we can see differences on the inscription (front of the plate), and on the sides of the neon (the elbows). I think we should try to find out why we have such differences (at least on the inscription, where it's quite visible), but I don't have the time right now, I'll try to have a look at it next week. [...] I had a look, it's a problem with anisotropy, I will create a PR for it. |
The glTF Animation Node test also exhibits some expected differences. The cubes all have roughness=1 and metallic=1 so the change is more pronounced. |
Thanks to @MiiBond, we validated the PBR changes and they all look closer to ray tracing :-) |
There was a change in lighting lately, but our tests didn't catch the changes due to the configuration of color-difference per pixel (threshold was 10%). This threshold was now reduced to 3%, and the number of changed pixels was lowered to 1% (strict!).
This resulted in many tests slightly failing.
All font-based tests have received a higher trheshold and other tests as well, depending on the test in hand.
Please review each of those, and check if it is in your responsibility. If it is, please check if the change makes sense or if something is wrong!
I also had to update a few playgrounds that simply didn't work correctly.
Note that some tests are still failing - deliberatly. I wanted to ask the people who added them to go over them and see what the reason for such a high change in pixels.