Skip to content

Commit

Permalink
DEPS: Sync Git subtree mirrors instead of symlinking into chromium/src
Browse files Browse the repository at this point in the history 
This changes the way we pull in dependencies WebRTC shares with
Chromium. The base, build, tools and third_party directories from
Chromium are now synced as Git subtree mirrors in the DEPS file.

All symlinks to directories that were previously created by the
setup_links.py are replaced with proper DEPS entries.

One downside with this solution is that we get a lot of directories
in tools/ and third_party/ that we currently don't use. Going forward
it might be possible to improve this but as long as the BUILD.gn files
are stored in the Chromium repo rather at each dependency's repo,
this will be very cumbersome.

The DEPS file will be kept auto-rolled by the script in
https://chromium.googlesource.com/external/webrtc/+/master/tools-webrtc/autoroller/roll_deps.py
which is periodically executed by a bot.

This change brings back the Google Play Services download for Android,
which displays a license confirmation dialog to the user at the first sync.
By running it as a proper hook instead of inside sync_chromium.py, the
problems with that the interactive prompt gets hidden/stuck should be
fixed (now the behavior is identical to Chromium).

Some measurements on the size savings for a clean, newly created checkout:
Linux: 15GB -> 6.4GB (-8.6GB)
Linux (with Android): 25 GB -> 16 GB (-9GB). 8.4GB of this is Android SDK+NDK.
Mac (with iOS): 14 GB -> 5.6GB (-8.4GB)

Note that for all of the above, 1GB is occupied by the resources/ dir.

BUG=webrtc:5006, webrtc:5578
NOTRY=True
[email protected], [email protected], [email protected]

Review-Url: https://codereview.webrtc.org/1414343008 .
Cr-Commit-Position: refs/heads/master@{#15754}
  • Loading branch information
[email protected] committed Dec 22, 2016
1 parent 5262487 commit 177567c
Show file tree
Hide file tree
Showing 11 changed files with 157 additions and 945 deletions.
107 changes: 2 additions & 105 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,121 +40,18 @@
/base
/build
/buildtools
/chromium/.gclient.tmp
/chromium/.gclient.tmp_entries
/chromium/.last_sync_chromium
/chromium/_bad_scm
/chromium/src
/links
/links.db
/mojo
/out
/testing
/third_party/BUILD.gn
/third_party/WebKit
/third_party/WebKit/Tools/Scripts
/third_party/accessibility_test_framework
/third_party/afl
/third_party/android_platform
/third_party/android_support_test_runner
/third_party/android_tools
/third_party/apache_velocity
/third_party/appurify-python
/third_party/asan
/third_party/ashmem
/third_party/binutils
/third_party/boringssl
/third_party/bouncycastle
/third_party/byte_buddy
/third_party/catapult
/third_party/ced
/third_party/clang_format
/third_party/class-dump
/third_party/closure_compiler
/third_party/colorama
/third_party/cygwin
/third_party/directxsdk
/third_party/espresso
/third_party/expat
/third_party/ffmpeg
/third_party/gaeunit
/third_party/gflags
/third_party/google-visualization-python
/third_party/gtest-parallel
/third_party/guava
/third_party/hamcrest
/third_party/icu
/third_party/icu4j
/third_party/ijar
/third_party/instrumented_libraries
/third_party/intellij
/third_party/jsoncpp
/third_party/javax_inject
/third_party/jsr-305
/third_party/junit
/third_party/libFuzzer
/third_party/libc++
/third_party/libc++-static
/third_party/libc++abi
/third_party/libjingle
/third_party/libjpeg
/third_party/libjpeg_turbo
/third_party/libsrtp
/third_party/libvpx
/third_party/libxml
/third_party/libyuv
/third_party/llvm
/third_party/llvm-build
/third_party/lss
/third_party/mockito
/third_party/modp_b64
/third_party/oauth2
/third_party/objenesis
/third_party/ocmock
/third_party/openh264
/third_party/openmax_dl
/third_party/opus
/third_party/ow2_asm
/third_party/proguard
/third_party/protobuf
/third_party/requests
/third_party/robolectric
/third_party/sqlite
/third_party/sqlite4java
/third_party/syzygy
/third_party/tcmalloc
/third_party/usrsctp
/third_party/valgrind
/third_party/winsdk_samples
/third_party/yasm
/third_party/zlib
/tools/android
/tools/clang
/tools/clang_format_merge_driver
/tools/determinism
/tools/generate_library_loader
/tools/generate_stubs
/tools/gn
/tools/grit
/tools/gyp
/tools/isolate_driver.py
/tools/luci-go
/tools/memory
/tools/protoc_wrapper
/tools/python
/tools/sanitizer_options
/tools/swarming_client
/tools/telemetry
/tools/tsan_suppressions
/tools/valgrind
/third_party
/tools
/tools-webrtc/video_quality_toolchain/linux/ffmpeg
/tools-webrtc/video_quality_toolchain/linux/zxing
/tools-webrtc/video_quality_toolchain/mac/ffmpeg
/tools-webrtc/video_quality_toolchain/mac/zxing
/tools-webrtc/video_quality_toolchain/win/*.dll
/tools-webrtc/video_quality_toolchain/win/*.exe
/tools/vim
/tools/win
/tools/whitespace.txt
/x86-generic_out/
/xcodebuild
155 changes: 131 additions & 24 deletions DEPS
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,97 @@
# This file contains dependencies for WebRTC that are not shared with Chromium.
# If you wish to add a dependency that is present in Chromium's src/DEPS or a
# directory from the Chromium checkout, you should add it to setup_links.py
# instead.
# This file contains dependencies for WebRTC.

vars = {
'chromium_git': 'https://chromium.googlesource.com',
'chromium_revision': 'e6b29f780581dedef421548119586efeb5825d2c',
'boringssl_git': 'https://boringssl.googlesource.com',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling swarming_client
# and whatever else without interference from each other.
'swarming_revision': 'ebc8dab6f8b8d79ec221c94de39a921145abd404',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling openmax_dl
# and whatever else without interference from each other.
'openmax_dl_revision': '7acede9c039ea5d14cf326f44aad1245b9e674a7',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling BoringSSL
# and whatever else without interference from each other.
'boringssl_revision': '33b1d4f575543b42b9c3406b7f8259daf11945dc',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling nss
# and whatever else without interference from each other.
'nss_revision': 'db89506ac363956b4b253db881219eb5d6246781',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling lss
# and whatever else without interference from each other.
'lss_revision': '3f6478ac95edf86cd3da300c2c0d34a438f5dbeb',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling catapult
# and whatever else without interference from each other.
'catapult_revision': '70f42a7c55ca69cdeb9aa6ec7e40ff3f155040b9',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling libFuzzer
# and whatever else without interference from each other.
'libfuzzer_revision': '2d19afdbadb1f20d4254f8a5447fceb30f616157',
}

deps = {
# TODO(kjellander): Move this to be Android-only once the libevent dependency
# in base/third_party/libevent is solved.
'src/base':
Var('chromium_git') + '/chromium/src/base' + '@' + '426683655fbb37d58a331d97121fe5d754d3ea02',
'src/build':
Var('chromium_git') + '/chromium/src/build' + '@' + '53a8a4bbfa7ecf84bba943aa0fdaceb2bd542c85',
'src/buildtools':
Var('chromium_git') + '/chromium/buildtools.git' + '@' + '0ef801087682b271e9ace93cfa93e9d3dea98079',
'src/testing':
Var('chromium_git') + '/chromium/src/testing' + '@' + '97282a50ebdaf10a2844481566c574bcb5776272',
'src/testing/gtest':
Var('chromium_git') + '/external/github.com/google/googletest.git' + '@' + '6f8a66431cb592dad629028a50b3dd418a408c87',
'src/testing/gmock':
Var('chromium_git') + '/external/googlemock.git' + '@' + '0421b6f358139f02e102c9c332ce19a33faf75be', # from svn revision 566
'src/third_party':
Var('chromium_git') + '/chromium/src/third_party' + '@' + '905ae0c419397358564b01df29260c136a0c4aff',
'src/third_party/boringssl/src':
Var('boringssl_git') + '/boringssl.git' + '@' + Var('boringssl_revision'),
'src/third_party/catapult':
Var('chromium_git') + '/external/github.com/catapult-project/catapult.git' + '@' + Var('catapult_revision'),
'src/third_party/colorama/src':
Var('chromium_git') + '/external/colorama.git' + '@' + '799604a1041e9b3bc5d2789ecbd7e8db2e18e6b8',
'src/third_party/ffmpeg':
Var('chromium_git') + '/chromium/third_party/ffmpeg.git' + '@' + 'f309edd7828e3ea500c2891187d15926690ddd27',
'src/third_party/jsoncpp/source':
Var('chromium_git') + '/external/github.com/open-source-parsers/jsoncpp.git' + '@' + 'f572e8e42e22cfcf5ab0aea26574f408943edfa4', # from svn 248

# Used for building libFuzzers (only supports Linux).
'src/third_party/libFuzzer/src':
Var('chromium_git') + '/chromium/llvm-project/llvm/lib/Fuzzer.git' + '@' + Var('libfuzzer_revision'),

'src/third_party/libjpeg_turbo':
Var('chromium_git') + '/chromium/deps/libjpeg_turbo.git' + '@' + '7260e4d8b8e1e40b17f03fafdf1cd83296900f76',
'src/third_party/libsrtp':
Var('chromium_git') + '/chromium/deps/libsrtp.git' + '@' + '0e0936f3013fe5884eac82f95e370c8d460a179f',
'src/third_party/libvpx/source/libvpx':
Var('chromium_git') + '/webm/libvpx.git' + '@' + 'd1eca240fb04fae3039ac28510ba0992eb03248d',
'src/third_party/libyuv':
Var('chromium_git') + '/libyuv/libyuv.git' + '@' + 'b18fd21d3c27fce69b5c1ed44b89131dedc87284',
'src/third_party/nss':
Var('chromium_git') + '/chromium/deps/nss.git' + '@' + Var('nss_revision'),
'src/third_party/openh264/src':
Var('chromium_git') + '/external/github.com/cisco/openh264' + '@' + '0fd88df93c5dcaf858c57eb7892bd27763f0f0ac',
'src/third_party/openmax_dl':
Var('chromium_git') + '/external/webrtc/deps/third_party/openmax.git' + '@' + Var('openmax_dl_revision'),
'src/third_party/usrsctp/usrsctplib':
Var('chromium_git') + '/external/github.com/sctplab/usrsctp' + '@' + '7f9228152ab3d70e6848cc9c67389a0d4218740e',
'src/third_party/yasm/source/patched-yasm':
Var('chromium_git') + '/chromium/deps/yasm/patched-yasm.git' + '@' + '7da28c6c7c6a1387217352ce02b31754deb54d2a',
'src/tools':
Var('chromium_git') + '/chromium/src/tools' + '@' + '4b7f075b7070bd3e422f1da2c526934bc14a7834',
'src/tools/gyp':
Var('chromium_git') + '/external/gyp.git' + '@' + 'e7079f0e0e14108ab0dba58728ff219637458563',
'src/tools/swarming_client':
Var('chromium_git') + '/external/swarming.client.git' + '@' + Var('swarming_revision'),

# WebRTC-only dependencies (not present in Chromium).
'src/third_party/gflags':
Var('chromium_git') + '/external/webrtc/deps/third_party/gflags' + '@' + '892576179b45861b53e04a112996a738309cf364',
'src/third_party/gflags/src':
Expand All @@ -18,9 +101,40 @@ deps = {
}

deps_os = {
'android': {
'src/third_party/android_tools':
Var('chromium_git') + '/android_tools.git' + '@' + 'b43a6a289a7588b1769814f04dd6c7d7176974cc',
'src/third_party/ced/src':
Var('chromium_git') + '/external/github.com/google/compact_enc_det.git' + '@' + 'e57cdc44bd541d10669312a6fdc59fc4bf52d2b9',
'src/third_party/icu':
Var('chromium_git') + '/chromium/deps/icu.git' + '@' + '9cd2828740572ba6f694b9365236a8356fd06147',
'src/third_party/jsr-305/src':
Var('chromium_git') + '/external/jsr-305.git' + '@' + '642c508235471f7220af6d5df2d3210e3bfc0919',
'src/third_party/junit/src':
Var('chromium_git') + '/external/junit.git' + '@' + '64155f8a9babcfcf4263cf4d08253a1556e75481',
'src/third_party/lss':
Var('chromium_git') + '/linux-syscall-support.git' + '@' + Var('lss_revision'),
'src/third_party/mockito/src':
Var('chromium_git') + '/external/mockito/mockito.git' + '@' + 'de83ad4598ad4cf5ea53c69a8a8053780b04b850',
'src/third_party/requests/src':
Var('chromium_git') + '/external/github.com/kennethreitz/requests.git' + '@' + 'f172b30356d821d180fa4ecfa3e71c7274a32de4',
'src/third_party/robolectric/robolectric':
Var('chromium_git') + '/external/robolectric.git' + '@' + 'e38b49a12fdfa17a94f0382cc8ffaf69132fd09b',
'src/third_party/ub-uiautomator/lib':
Var('chromium_git') + '/chromium/third_party/ub-uiautomator.git' + '@' + '00270549ce3161ae72ceb24712618ea28b4f9434',
},
'unix': {
'src/third_party/lss':
Var('chromium_git') + '/linux-syscall-support.git' + '@' + Var('lss_revision'),
},
'win': {
# Dependencies used by libjpeg-turbo
'src/third_party/yasm/binaries':
Var('chromium_git') + '/chromium/deps/yasm/binaries.git' + '@' + '52f9b3f4b0aa06da24ef8b123058bb61ee468881',

# WebRTC-only dependency (not present in Chromium).
'src/third_party/winsdk_samples':
Var('chromium_git') + '/external/webrtc/deps/third_party/winsdk_samples_v71' + '@' + '6e4f93cc0c2f597547449cbaa80a6af629215a63',
Var('chromium_git') + '/external/webrtc/deps/third_party/winsdk_samples_v71' + '@' + '6e4f93cc0c2f597547449cbaa80a6af629215a63',
},
}

Expand All @@ -35,19 +149,6 @@ hooks = [
'_ = os.system("%s %s" % (sys.executable,script)) '
'if os.path.exists(script) else 0')],
},
{
# Clone chromium and its deps.
'name': 'sync chromium',
'pattern': '.',
'action': ['python', '-u', 'src/sync_chromium.py',
'--target-revision', Var('chromium_revision')],
},
{
# Create links to shared dependencies in Chromium.
'name': 'setup_links',
'pattern': '.',
'action': ['python', 'src/setup_links.py'],
},
{
# This clobbers when necessary (based on get_landmines.py). It should be
# an early hook but it will need to be run after syncing Chromium and
Expand All @@ -63,15 +164,21 @@ hooks = [
'src',
],
},
{
# Download Google Play Services SDK (without license prompt).
'name': 'google_play_services_download',
'pattern': '.',
'action': ['python', 'src/webrtc/build/google_play_services_download.py'],
},
# Android dependencies. Many are downloaded using Google Storage these days.
# They're copied from https://cs.chromium.org/chromium/src/DEPS for all
# such dependencies we share with Chromium.
{
# This downloads SDK extras and puts them in the
# third_party/android_tools/sdk/extras directory.
'name': 'sdkextras',
'pattern': '.',
# When adding a new sdk extras package to download, add the package
# directory and zip file to .gitignore in third_party/android_tools.
'action': ['python',
'src/build/android/play_services/update.py',
'download'
],
},
{
'name': 'intellij',
'pattern': '.',
Expand Down
21 changes: 2 additions & 19 deletions PRESUBMIT.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -434,30 +434,13 @@ def _CommonChecks(input_api, output_api):
results.extend(_CheckApprovedFilesLintClean(
input_api, output_api, source_file_filter))
results.extend(input_api.canned_checks.RunPylint(input_api, output_api,
black_list=(r'^.*gviz_api\.py$',
r'^.*gaeunit\.py$',
# Embedded shell-script fakes out pylint.
black_list=(r'^base[\\\/].*\.py$',
r'^build[\\\/].*\.py$',
r'^buildtools[\\\/].*\.py$',
r'^chromium[\\\/].*\.py$',
r'^mojo.*[\\\/].*\.py$',
r'^out.*[\\\/].*\.py$',
r'^testing[\\\/].*\.py$',
r'^third_party[\\\/].*\.py$',
r'^tools[\\\/]clang[\\\/].*\.py$',
r'^tools[\\\/]generate_library_loader[\\\/].*\.py$',
r'^tools[\\\/]generate_stubs[\\\/].*\.py$',
r'^tools[\\\/]gn[\\\/].*\.py$',
r'^tools[\\\/]isolate_driver.py$',
r'^tools[\\\/]mb[\\\/].*\.py$',
r'^tools[\\\/]protoc_wrapper[\\\/].*\.py$',
r'^tools[\\\/]python[\\\/].*\.py$',
r'^tools[\\\/]python_charts[\\\/]data[\\\/].*\.py$',
r'^tools[\\\/]refactoring[\\\/].*\.py$',
r'^tools[\\\/]swarming_client[\\\/].*\.py$',
r'^tools[\\\/]vim[\\\/].*\.py$',
r'^tools[\\\/]valgrind[\\\/].*\.py$',
r'^tools[\\\/]win[\\\/].*\.py$',
r'^tools[\\\/].*\.py$',
# TODO(phoglund): should arguably be checked.
r'^tools-webrtc[\\\/]mb[\\\/].*\.py$',
r'^tools-webrtc[\\\/]valgrind[\\\/].*\.py$',
Expand Down
23 changes: 0 additions & 23 deletions chromium/.gclient
View file

This file was deleted.

1 change: 0 additions & 1 deletion chromium/OWNERS
View file

This file was deleted.

5 changes: 0 additions & 5 deletions chromium/README
View file

This file was deleted.

Loading

0 comments on commit 177567c

Please sign in to comment.