Backed out 6 changesets (bug 1738845, bug 1773223, bug 1760484) for c…

…ausing multiple failures CLOSED TREE

Backed out changeset e759ac4fb646 (bug 1773223)
Backed out changeset 20c541be9a37 (bug 1760484)
Backed out changeset 653d2024a187 (bug 1738845)
Backed out changeset fd9540114bfd (bug 1773223)
Backed out changeset f24f3dfa49aa (bug 1773223)
Backed out changeset 11f3375a4a70 (bug 1773223)

build/docs/gn.rst

@@ -11,7 +11,42 @@ we have support in the build system for translating GN configuration
 files into moz.build files. In most cases these moz.build files will be like any
 others in the tree (except that they shouldn't be modified by hand), however
 those updating vendored code or building on platforms not supported by
-Mozilla automation may need to re-generate these files. This is a per-project
-process, described in dom/media/webrtc/third_party_build/gn-configs/README.md for
-webrtc. As of writing, it is very specific to webrtc, and likely doesn't work as-is
-for other projects.
+Mozilla automation may need to re-generate these files. This is a two-step
+process, described below.
+
+Generating GN configs as JSON
+=============================
+
+The first step must take place on a machine with access to the ``GN`` tool, which
+is specified in a mozconfig with::
+
+    export GN=<path/to/gn>
+
+With this specified, and the tree configured, run::
+
+    $ ./mach build-backend -b GnConfigGen
+
+This will run ``gn gen`` on projects found in ``GN_DIRS`` specified in moz.build
+files, injecting variables from the current build configuration and specifying
+the result be written to a JSON file in ``$objdir/gn-output``. The file will
+have a name derived from the arguments passed to ``gn gen``, for instance
+``x64_False_x64_linux.json``.
+
+If updating upstream sources or vendoring a new project, this step must be
+performed for each supported build configuration. If adding support for a
+specific configuration, the generated configuration may be added to existing
+configs before re-generating the ``moz.build`` files, which should be found under
+the ``gn-configs`` directory under the vendored project's top-level directory.
+
+Generating moz.build files from GN JSON configs
+===============================================
+
+Once the relevant JSON configs are present under a project's ``gn-configs``
+directory, run::
+
+    $ ./mach build-backend -b GnMozbuildWriter
+
+This will combine the configuration files present into a set of moz.build files
+that will build the given project. Once the result is verified, the resulting
+moz.build files should be checked in and should build like any other part of
+mozilla-central.

build/gn.mozbuild

@@ -0,0 +1,42 @@
+# -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
+# vim: set filetype=python:
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+gn_vars = {}
+
+if CONFIG['MOZ_DEBUG']:
+   gn_vars['is_debug'] = True
+else:
+   gn_vars['is_debug'] = False
+
+os = CONFIG['OS_TARGET']
+
+if CONFIG['MOZ_X11']:
+    gn_vars['use_x11'] = True
+elif os == "Linux":
+    # Only set use_x11 explicitly to false on Linux for no-x11 variants
+    gn_vars['use_x11'] = False
+
+flavors = {
+    'WINNT': 'win',
+    'Android': 'android',
+    'Linux': 'linux',
+    'Darwin': 'mac' if CONFIG['MOZ_WIDGET_TOOLKIT'] == 'cocoa' else 'ios',
+    'SunOS': 'solaris',
+    'GNU/kFreeBSD': 'freebsd',
+    'DragonFly': 'dragonfly',
+    'FreeBSD': 'freebsd',
+    'NetBSD': 'netbsd',
+    'OpenBSD': 'openbsd',
+}
+gn_vars['target_os'] = flavors.get(os)
+
+arches = {
+    'x86_64': 'x64',
+    'aarch64': 'arm64',
+}
+
+gn_vars['host_cpu'] = arches.get(CONFIG['HOST_CPU_ARCH'], CONFIG['HOST_CPU_ARCH'])
+gn_vars['target_cpu'] = arches.get(CONFIG['CPU_ARCH'], CONFIG['CPU_ARCH'])

dom/media/webrtc/moz.build

@@ -23,9 +23,6 @@ with Files("jsapi/**"):
     BUG_COMPONENT = ("Core", "WebRTC: Signaling")
 with Files("sdp/**"):
     BUG_COMPONENT = ("Core", "WebRTC: Signaling")
-with Files("third_party_build/**"):
-    BUG_COMPONENT = ("Core", "WebRTC")
-
 
 EXPORTS += [
     "CubebDeviceEnumerator.h",
@@ -79,13 +76,10 @@ if CONFIG["MOZ_WEBRTC_SIGNALING"]:
         "jsep",
         "libwebrtcglue",
         "sdp",
+        "third_party_build",
         "transportbridge",
-        "/third_party/libwebrtc",
     ]
 
-    if CONFIG["MOZ_WIDGET_TOOLKIT"] == "gtk":
-        DIRS += ["/third_party/pipewire/libpipewire"]
-
     # Avoid warnings from third-party code that we can not modify.
     if CONFIG["CC_TYPE"] == "clang-cl":
         CXXFLAGS += ["-Wno-invalid-source-encoding"]

dom/media/webrtc/third_party_build/gn-configs/README.md

@@ -3,14 +3,29 @@
 /!\ This is only supported on Linux and macOS. If you are on Windows, you can run
 the script under [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
 
-1. The script should be run from the top directory of our firefox tree.
+1. Download a version of the `gn` executable that corresponds to
+    `Thu Nov 19 14:14:00 2020`.  In our case, that is version `1889 (8fe02009)`.
+
+   - [Linux](https://chrome-infra-packages.appspot.com/p/gn/gn/linux-amd64/+/bvBFKgehaepiKy_YhFnbiOpF38CK26N2OyE1R1jXof0C)
+   - [macOS](https://chrome-infra-packages.appspot.com/p/gn/gn/mac-amd64/+/nXvMRpyJhLhisAcnRmU5s9UZqovzMAhKAvWjax-swioC)
 
-   ```
-   ./mach python python/mozbuild/mozbuild/gn_processor.py dom/media/webrtc/third_party_build/gn-configs/webrtc.json
-   ```
+   Find the downloaded `.zip` file, unzip and export the location of the
+   executable:
 
-2. Checkin all the generated/modified files and try your build!
+        unzip gn-mac-amd64.zip && export GN=`pwd`/gn
+        unzip gn-linux-amd64.zip && export GN=`pwd`/gn
+
+2. It is time to generate the build files.  The script should be run from the
+   top directory of our firefox tree.
+
+        bash ./dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
+
+   Debugging the generate script itself may prove useful, and one can do this by
+   setting the DEBUG_GEN environment variable to a non-empty value. This will 
+   print everything that the script executes.
+
+3. Checkin all the generated/modified files and try your build!
 
 # Adding new configurations to the build
 
-Edit the `main` function in the `python/mozbuild/mozbuild/gn_processor.py` file.
+Edit the `GnConfigGenBackend` class in `python/mozbuild/mozbuild/gn_processor.py` file.

dom/media/webrtc/third_party_build/gn-configs/fixup_json.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+import os
+import re
+import sys
+
+
+def main():
+    # Check and retrieve command-line arguments
+    if len(sys.argv) != 2:
+        print(__doc__)
+        sys.exit(1)  # Return a non-zero value to indicate abnormal termination
+    filein = sys.argv[1]
+
+    if not os.path.isfile(filein):
+        print("error: {} does not exist".format(filein))
+        sys.exit(1)
+
+    f = open(filein, "r")
+    file = f.read()
+
+    # In practice, almost all of the entries in the cflags section have no affect
+    # on the moz.build output files when running ./mach build-backend -b GnMozbuildWriter
+    # There are few exceptions which do: -msse2, -mavx2, -mfma, -fobjc-arc
+    # However, since we're really concerned about removing differences between development
+    # machines, we only need remove the reference to osx sdk.  Removing it doesn't change
+    # the generated moz.build files and makes diffs much easier to see.
+    file = re.sub(
+        r' *"-isysroot",\n *".*/Contents/Developer/Platforms/MacOSX\.platform/Developer/SDKs/'
+        'MacOSX([0-9]+\.[0-9]+)?\.sdk",\n',
+        r"",
+        file,
+    )
+    f.close()
+
+    f = open(filein, "w")
+    f.write(file)
+    f.close()
+
+
+if __name__ == "__main__":
+    main()

dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+
+# Print an Error message if `set -eE` causes the script to exit due to a failed command
+trap 'echo "*** ERROR *** $? $LINENO Generation did not complete successfully!"' ERR 
+
+# If DEBUG_GEN is set all commands should be printed as they are executed
+if [ ! "x$DEBUG_GEN" = "x" ]; then
+  set -x
+fi
+
+if [ "x$GN" = "x" ]; then
+  echo "GN is not defined, see README.md"
+  exit
+fi
+
+if [ -f $GN ]; then
+  echo "GN is $GN"
+else
+  echo "Path $GN is not found, see README.md"
+  exit
+fi
+
+# After this point:
+# * eE: All commands should succede.
+# * u: All variables should be defined before use.
+# * o pipefail: All stages of all pipes should succede.
+set -eEuo pipefail
+
+SYS_NAME=`uname`
+
+# Check for modified files and abort if present.
+MODIFIED_FILES=`hg status --modified --added --exclude "**/moz.build" --exclude "dom/media/webrtc/third_party_build/**.json"`
+if [ "x$MODIFIED_FILES" = "x" ]; then
+  # Completely clean the mercurial checkout before proceeding
+  hg update -C -r .
+  hg purge
+else
+  echo "There are modified files in the checkout. Cowardly aborting!"
+  echo "$MODIFIED_FILES"
+  exit 1
+fi
+
+CONFIG_DIR=dom/media/webrtc/third_party_build/gn-configs
+echo "CONFIG_DIR is $CONFIG_DIR"
+
+export MOZ_OBJDIR=$(mktemp -d -p . obj-XXXXXXXXXX)
+./mach configure
+if [ ! -d $MOZ_OBJDIR ]; then
+  echo "Expected build output directory $MOZ_OBJDIR is missing"
+  exit 1
+fi
+./mach build-backend -b GnConfigGen --verbose | tee $MOZ_OBJDIR/build-backend.log
+cp $MOZ_OBJDIR/third_party/libwebrtc/gn-output/*.json $CONFIG_DIR
+
+# run some fixup (mostly removing dev-machine dependent info) on json files
+for THIS_CONFIG in $CONFIG_DIR/*.json
+do
+  echo "fixup file: $THIS_CONFIG"
+  ./$CONFIG_DIR/fixup_json.py $THIS_CONFIG
+done
+
+#  The symlinks are no longer needed after generating the .json files.
+if [ -L third_party/libwebrtc/buildtools ]; then
+  rm third_party/libwebrtc/buildtools
+else
+  rm -rf third_party/libwebrtc/buildtools
+fi
+
+if [ -L third_party/libwebrtc/buildtools ]; then
+  rm third_party/libwebrtc/.git
+else
+  rm -rf third_party/libwebrtc/.git
+fi
+
+# After all the json files are generated they are used to generate moz.build files.
+echo "Building moz.build files from gn json files"
+./mach build-backend -b GnMozbuildWriter --verbose
+
+rm -rf $MOZ_OBJDIR
+
+echo
+echo "Done generating gn build files. You should now be able to build with ./mach build"