Bug 1773223 - Make the GN processor an independent script. r=firefox-…

…build-system-reviewers,mjf,ahochheiden

The way the processor works currently is that it relies on two different
build backends, one of which produces json files for specific
configurations, and the other which produces moz.build files from the
aggregate of all those configs.

Each of these json files is huge, and we actually don't have enough to
support all the platforms we're supposed to be supporting. Adding more
files is not enticing.

Now that we've made the first step described above work in a single pass
on a single machine (as opposed to multiple passes on multiple machines
previously), we can actually merge both steps and avoid producing the
intermediate json files altogether. This will allow to add more
configurations without having to worry about the weight of those files.

And because this all doesn't need to depend on having the first step
hooked up in the build system, we make the whole an independent script
rather than a build backend.

Differential Revision: https://phabricator.services.mozilla.com/D149210

build/docs/gn.rst

@@ -11,42 +11,7 @@ we have support in the build system for translating GN configuration
 files into moz.build files. In most cases these moz.build files will be like any
 others in the tree (except that they shouldn't be modified by hand), however
 those updating vendored code or building on platforms not supported by
-Mozilla automation may need to re-generate these files. This is a two-step
-process, described below.
-
-Generating GN configs as JSON
-=============================
-
-The first step must take place on a machine with access to the ``GN`` tool, which
-is specified in a mozconfig with::
-
-    export GN=<path/to/gn>
-
-With this specified, and the tree configured, run::
-
-    $ ./mach build-backend -b GnConfigGen
-
-This will run ``gn gen`` on projects found in ``GN_DIRS`` specified in moz.build
-files, injecting variables from the current build configuration and specifying
-the result be written to a JSON file in ``$objdir/gn-output``. The file will
-have a name derived from the arguments passed to ``gn gen``, for instance
-``x64_False_x64_linux.json``.
-
-If updating upstream sources or vendoring a new project, this step must be
-performed for each supported build configuration. If adding support for a
-specific configuration, the generated configuration may be added to existing
-configs before re-generating the ``moz.build`` files, which should be found under
-the ``gn-configs`` directory under the vendored project's top-level directory.
-
-Generating moz.build files from GN JSON configs
-===============================================
-
-Once the relevant JSON configs are present under a project's ``gn-configs``
-directory, run::
-
-    $ ./mach build-backend -b GnMozbuildWriter
-
-This will combine the configuration files present into a set of moz.build files
-that will build the given project. Once the result is verified, the resulting
-moz.build files should be checked in and should build like any other part of
-mozilla-central.
+Mozilla automation may need to re-generate these files. This is a per-project
+process, described in dom/media/webrtc/third_party_build/gn-configs/README.md for
+webrtc. As of writing, it is very specific to webrtc, and likely doesn't work as-is
+for other projects.

dom/media/webrtc/moz.build

@@ -23,6 +23,9 @@ with Files("jsapi/**"):
     BUG_COMPONENT = ("Core", "WebRTC: Signaling")
 with Files("sdp/**"):
     BUG_COMPONENT = ("Core", "WebRTC: Signaling")
+with Files("third_party_build/**"):
+    BUG_COMPONENT = ("Core", "WebRTC")
+
 
 EXPORTS += [
     "CubebDeviceEnumerator.h",
@@ -76,10 +79,13 @@ if CONFIG["MOZ_WEBRTC_SIGNALING"]:
         "jsep",
         "libwebrtcglue",
         "sdp",
-        "third_party_build",
         "transportbridge",
+        "/third_party/libwebrtc",
     ]
 
+    if CONFIG["MOZ_WIDGET_TOOLKIT"] == "gtk":
+        DIRS += ["/third_party/pipewire/libpipewire"]
+
     # Avoid warnings from third-party code that we can not modify.
     if CONFIG["CC_TYPE"] == "clang-cl":
         CXXFLAGS += ["-Wno-invalid-source-encoding"]

dom/media/webrtc/third_party_build/gn-configs/README.md

@@ -3,29 +3,14 @@
 /!\ This is only supported on Linux and macOS. If you are on Windows, you can run
 the script under [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
 
-1. Download a version of the `gn` executable that corresponds to
-    `Thu Nov 19 14:14:00 2020`.  In our case, that is version `1889 (8fe02009)`.
-
-   - [Linux](https://chrome-infra-packages.appspot.com/p/gn/gn/linux-amd64/+/bvBFKgehaepiKy_YhFnbiOpF38CK26N2OyE1R1jXof0C)
-   - [macOS](https://chrome-infra-packages.appspot.com/p/gn/gn/mac-amd64/+/nXvMRpyJhLhisAcnRmU5s9UZqovzMAhKAvWjax-swioC)
+1. The script should be run from the top directory of our firefox tree.
 
-   Find the downloaded `.zip` file, unzip and export the location of the
-   executable:
+   ```
+   ./mach python python/mozbuild/mozbuild/gn_processor.py dom/media/webrtc/third_party_build/gn-configs/webrtc.json
+   ```
 
-        unzip gn-mac-amd64.zip && export GN=`pwd`/gn
-        unzip gn-linux-amd64.zip && export GN=`pwd`/gn
-
-2. It is time to generate the build files.  The script should be run from the
-   top directory of our firefox tree.
-
-        bash ./dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
-
-   Debugging the generate script itself may prove useful, and one can do this by
-   setting the DEBUG_GEN environment variable to a non-empty value. This will 
-   print everything that the script executes.
-
-3. Checkin all the generated/modified files and try your build!
+2. Checkin all the generated/modified files and try your build!
 
 # Adding new configurations to the build
 
-Edit the `GnConfigGenBackend` class in `python/mozbuild/mozbuild/gn_processor.py` file.
+Edit the `main` function in the `python/mozbuild/mozbuild/gn_processor.py` file.

dom/media/webrtc/third_party_build/gn-configs/webrtc.json

@@ -0,0 +1,73 @@
+{
+  "target_dir": "third_party/libwebrtc",
+  "gn_target": "//:webrtc",
+  "gn_sandbox_variables": {
+    "COMPILE_FLAGS": {
+      "WARNINGS_AS_ERRORS": []
+    },
+    "FINAL_LIBRARY": "webrtc"
+  },
+  "mozilla_flags": [
+    "-fobjc-arc",
+    "-mavx2",
+    "-mfma",
+    "-mfpu=neon",
+    "-msse2"
+  ],
+  "write_mozbuild_variables": {
+    "INCLUDE_TK_CFLAGS_DIRS": [
+      "third_party/libwebrtc/modules/desktop_capture/desktop_capture_generic_gn"
+    ]
+  },
+  "non_unified_sources": [
+    "third_party/libwebrtc/api/audio_codecs/opus/audio_encoder_opus_config.cc",
+    "third_party/libwebrtc/audio/channel_send_frame_transformer_delegate.cc",
+    "third_party/libwebrtc/common_audio/vad/vad_core.c",
+    "third_party/libwebrtc/common_audio/vad/webrtc_vad.c",
+    "third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/decode_plc.c",
+    "third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/lpc_masking_model.c",
+    "third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter.c",
+    "third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter_c.c",
+    "third_party/libwebrtc/modules/audio_coding/neteq/audio_vector.cc",
+    "third_party/libwebrtc/modules/audio_device/android/audio_manager.cc",
+    "third_party/libwebrtc/modules/audio_device/android/audio_record_jni.cc",
+    "third_party/libwebrtc/modules/audio_device/android/audio_track_jni.cc",
+    "third_party/libwebrtc/modules/audio_device/android/opensles_player.cc",
+    "third_party/libwebrtc/modules/audio_device/linux/audio_device_pulse_linux.cc",
+    "third_party/libwebrtc/modules/audio_device/linux/audio_mixer_manager_pulse_linux.cc",
+    "third_party/libwebrtc/modules/audio_device/win/audio_device_core_win.cc",
+    "third_party/libwebrtc/modules/audio_processing/aecm/aecm_core.cc",
+    "third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_c.cc",
+    "third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_mips.cc",
+    "third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_neon.cc",
+    "third_party/libwebrtc/modules/audio_processing/aecm/echo_control_mobile.cc",
+    "third_party/libwebrtc/modules/audio_processing/echo_control_mobile_impl.cc",
+    "third_party/libwebrtc/modules/audio_processing/echo_detector/normalized_covariance_estimator.cc",
+    "third_party/libwebrtc/modules/audio_processing/gain_control_impl.cc",
+    "third_party/libwebrtc/modules/audio_processing/rms_level.cc",
+    "third_party/libwebrtc/modules/desktop_capture/linux/base_capturer_pipewire.cc",
+    "third_party/libwebrtc/modules/desktop_capture/win/screen_capturer_win_gdi.cc",
+    "third_party/libwebrtc/modules/remote_bitrate_estimator/remote_bitrate_estimator_single_stream.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/flexfec_receiver.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbn.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbr.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_sender.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtp_format_vp8.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtp_rtcp_impl2.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_egress.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_video.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/ulpfec_generator.cc",
+    "third_party/libwebrtc/modules/rtp_rtcp/source/video_rtp_depacketizer_vp9.cc",
+    "third_party/libwebrtc/modules/third_party/g722/g722_encode.c",
+    "third_party/libwebrtc/modules/video_capture/windows/device_info_ds.cc",
+    "third_party/libwebrtc/modules/video_capture/windows/help_functions_ds.cc",
+    "third_party/libwebrtc/modules/video_capture/windows/sink_filter_ds.cc",
+    "third_party/libwebrtc/modules/video_coding/codecs/vp8/screenshare_layers.cc",
+    "third_party/libwebrtc/third_party/abseil-cpp/absl/strings/numbers.cc",
+    "third_party/libwebrtc/third_party/abseil-cpp/absl/synchronization/blocking_counter.cc",
+    "third_party/libwebrtc/third_party/abseil-cpp/absl/time/internal/cctz/src/time_zone_posix.cc",
+    "third_party/libwebrtc/third_party/abseil-cpp/absl/time/time.cc",
+    "third_party/libwebrtc/video/rtp_video_stream_receiver2.cc",
+    "third_party/libwebrtc/video/video_receive_stream.cc"
+  ]
+}