From 2458930a6b03edc18618d0a614493cd589d44d23 Mon Sep 17 00:00:00 2001 From: Jeremy Koritzinsky Date: Fri, 13 Dec 2019 10:30:58 -0800 Subject: [PATCH] Local live-live builds (#494) * Convert libraries and installer to use a live CoreCLR. * Installer uses a live libraries build for DLLs. * Remove newline in path. * Enable live-live builds to build required live dependencies when the dependencies are in the building subset/subset category. * Update path to coreclr output. * Fix typo in coreclr.proj. * Remove last usage of CoreCLRPDBOverridePath. * Update path of binplaced runtime to include OS and Arch. * Fix typo in coreclr/tests/src/Common/Directory.Build.targets. * Restore CoreFX shared framework from live build for CoreCLR test dependencies. * Remove dependency on Microsoft.NETCore.CoreCLR.TestDependencies package. * Enable referencing live builds of the OOB libraries by temporarily adding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries. * Remove reference to System.Drawing.Common from NETClientPrimitives (the reference is only needed at runtime and Core_Root already has System.Drawing.Common). * Remove package-based checked CoreCLR + CoreFX test runs. In the live-live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds). * Only copy the UCRT on Windows. * CoreCLR test build now builds against a live libraries sfx+oob ref assembly set. * Override GetCorePackagePaths target to not reference unused packages (breaks the live-live build on a clean system). * Remove live references to Microsoft.NETCore.Platforms now that it seems to be unneeded since the SDK now packages a RID graph. Use the live RID graph in test_runtime.csproj where we may actually be looking for the live RID graph. * Fix coreclr build arg generation bug that was causing build failures. * Remove references to Microsoft.NETCore.Platforms outside of package references from the installers. * Fix problems in installer build in live-live that were found from a clean build. * Remove unused dependency on Microsoft.NETCore.Targets. * Fix Microsoft.NETCore.App -> Microsoft.NETCore.Platforms dependency. * Remove Microsoft.NETCore.Targets version from Versions.props. * Remove unused deps.csproj file that was the last dependency on the packaged Microsoft.NETCore.Platforms package. * Fix OSGroup calculation for dependent projects off-Windows. * Fix construction of Microsoft.NETCore.App.Internal package. * Remove unneeded CoreRun copying. * Serialize subsets that have dependencies across subset groups. Don't use `` tasks to rebuild subsets since we don't want to rebuild projects multiple times (coreclr was being built multiple times). * Cleanup from PR feedback. * Only copy libraries outputs to artifacts/bin/runtime. * Use the netcoreapp output directories for OOB-included references to libraries outputs. * Remove references to deleted old corefx override setup. * First pass to get new runtime pipeline using the local live-live infra (coreclr+libraries only for now). * Don't generate layout during test native build. * Fix ref assembly default binplacing for netcoreapp. * Remove msbuild goo around importing Subsets.props and Directory.Build.props only once each. This goo isn't needed anymore. * Fix disabling of binplacing for the target vertical to exclude only runtime.depproj and not exclude winrt.depproj. * Fix libraries test build. * Copy all paths used by the "ResolveLibrariesFromLocalBuild" target to publish from libraries build. * For shims, reference the System.Private.CoreLib directly from CoreCLR outputs instead of through runtime.depproj. * Hard-code the libraries configuration to release in coreclr test builds. * Resolve ReferenceFromRuntime items directly from the CoreCLR output instead of via the runtime.depproj project. * Fix test host generation. Fix running the XML Serialization generator as part of libraries test build. * Fix XmlSerializer.Generator.Tests again. * Don't look for CoreCLR files when using ReferenceFromRuntime and not using netcoreapp. * Crossgen-comparison should depend on a libraries build so we can populate Core_Root. * Add job dependency on libraries build from run-test-job to ensure we wait correctly. Otherwise we might have a race since the managed test build might depend on one libraries build while the test run depends on another. * Only resolve coreclr files from local build when targetting netcoreapp. * Add steps to download libraries build for crossgen-comparison. * Revive old method of resolving references to the runtime for building against old netcoreapp (ie netcoreapp 3.0). Fix correctly passing along aliases info. * Fix failing AllConfigurations leg that was due to the netcoreapp runtime.depproj getting binplaced in the runtime output folder. * Update eng/liveBuilds.targets Co-Authored-By: Davis Goodin * HostArch->BuildArchitecture and share with installer. * Fix format nits. * Update YAML variable name. * Remove setting property to import liveBuilds.targets. * Upload docs and packages folders from libraries build * Inclue doc files in the Microsoft.NETCore.App.Internal package * Try making the runtime-installer pipeline live-live using the build-coreclr-and-libraries-job template. * Update artifacts download paths and msbuild properties for installer build. * Mark cross-component files as native. * Remove FrameworkPackageName to remove error about missing $(FrameworkPackageName).versions.txt file that other shared frameworks also don't generate. * Restore GetFilesFromPackageResolve override. * Don't clean the unpack folder for the libraries download. It wipes out the CoreCLR download. * Make sure that we restore the WinRT targetting pack during the installer build. * Add libraries package directory to RestoreSources for the case where the LibrariesConfiguration is different from the installer build configuration. * Don't pass OfficialBuildId unless it is an official build. We want to be consistent with the libraries build so our package versions match. * Give default value for OfficialBuildArg variable * Ensure installer tests can see the libraries-generated packages. Fix typo in TargetPath for CoreCLR assets (runtime instead of runtimes). Fix paths for CoreCLR cross-target files as well as crossgen to place them in the correct directories. * Add back FrameworkPackageName to get the partial platform manifest to generate. --- Directory.Build.props | 3 + Directory.Build.targets | 3 +- eng/Subsets.props | 28 +- eng/Tools.props | 5 +- eng/Version.Details.xml | 12 - eng/Versions.props | 6 - eng/helixcorefxtests.proj | 291 ------------------ eng/liveBuilds.targets | 90 ++++++ eng/packaging.props | 3 +- eng/pipelines/coreclr/pr.yml | 2 + eng/pipelines/coreclr/templates/build-job.yml | 2 +- .../templates/crossgen-comparison-job.yml | 23 +- .../coreclr/templates/run-test-job.yml | 4 +- .../coreclr/templates/xplat-pipeline-job.yml | 11 +- eng/pipelines/installer/azure-pipelines.yml | 11 + eng/pipelines/installer/jobs/base-job.yml | 24 +- eng/pipelines/installer/jobs/bash-build.yml | 2 +- eng/pipelines/installer/jobs/osx-build.yml | 2 +- .../installer/jobs/windows-build.yml | 2 +- eng/pipelines/libraries/base-job.yml | 10 +- eng/pipelines/libraries/build-job.yml | 26 +- eng/referenceFromRuntime.targets | 53 ++-- src/coreclr/build-test.cmd | 56 +--- src/coreclr/build-test.sh | 45 +-- src/coreclr/coreclr.proj | 6 +- src/coreclr/src/.nuget/packaging.props | 1 - src/coreclr/tests/build.proj | 1 - src/coreclr/tests/scripts/patch-corefx.py | 263 ---------------- .../tests/src/Common/CoreFX/CoreFX.csproj | 150 --------- .../tests/src/Common/Directory.Build.targets | 16 +- .../test_dependencies.csproj | 18 +- .../Common/test_runtime/test_runtime.csproj | 9 +- src/coreclr/tests/src/Directory.Build.targets | 33 +- .../Primitives/NETClientPrimitives.csproj | 3 - src/coreclr/tests/src/dirs.proj | 1 - src/coreclr/tests/src/runtest.proj | 14 - src/installer/Directory.Build.props | 3 +- src/installer/Directory.Build.targets | 13 +- src/installer/pkg/Directory.Build.targets | 4 - src/installer/pkg/deps/deps.csproj | 11 - .../pkg/projects/Directory.Build.targets | 24 +- .../Microsoft.NETCore.App.Internal.pkgproj | 5 + .../workaround/Microsoft.NETCore.App.pkgproj | 2 +- .../Microsoft.NETCore.App.SharedFx.sfxproj | 13 +- .../src/localnetcoreapp.override.targets | 202 +++++------- .../netcoreapp/src/netcoreapp.depproj | 91 +++--- .../PrepareTestAssets/PrepareTestAssets.proj | 2 + src/libraries/Directory.Build.props | 10 +- src/libraries/Directory.Build.targets | 12 +- src/libraries/restore/dirs.proj | 1 - src/libraries/restore/runtime/runtime.depproj | 47 +-- .../shims/manual/Directory.Build.props | 10 +- src/libraries/src.builds | 2 + 53 files changed, 494 insertions(+), 1187 deletions(-) delete mode 100644 eng/helixcorefxtests.proj create mode 100644 eng/liveBuilds.targets delete mode 100755 src/coreclr/tests/scripts/patch-corefx.py delete mode 100644 src/coreclr/tests/src/Common/CoreFX/CoreFX.csproj delete mode 100644 src/installer/pkg/deps/deps.csproj diff --git a/Directory.Build.props b/Directory.Build.props index ca3e02475e2b7..6c65af67d082d 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -60,6 +60,7 @@ git://github.com/dotnet/runtime https://github.com/dotnet/runtime https://github.com/dotnet/runtime/blob/master/LICENSE.TXT + $([MSBuild]::NormalizePath('$(LibrariesProjectRoot)', 'pkg', 'Microsoft.NETCore.Platforms', 'runtime.json')) $(RepoRoot)LICENSE.TXT @@ -89,6 +90,8 @@ true Properties + + $([System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture.ToString().ToLowerInvariant()) diff --git a/Directory.Build.targets b/Directory.Build.targets index 7d40eb9ac3020..a1589f9f5d566 100644 --- a/Directory.Build.targets +++ b/Directory.Build.targets @@ -8,6 +8,7 @@ false + @@ -18,4 +19,4 @@ $(MajorVersion).$(MinorVersion) - \ No newline at end of file + diff --git a/eng/Subsets.props b/eng/Subsets.props index a1ca197fe66c0..ba5a44937763f 100644 --- a/eng/Subsets.props +++ b/eng/Subsets.props @@ -76,7 +76,7 @@ - + @@ -96,14 +96,16 @@ - - - - - - + + + + + + + + @@ -119,23 +121,23 @@ - + - + - + - - + + @@ -147,7 +149,7 @@ - + diff --git a/eng/Tools.props b/eng/Tools.props index fa9804318e7c7..4c399288ad250 100644 --- a/eng/Tools.props +++ b/eng/Tools.props @@ -13,14 +13,11 @@ - + - - - diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml index 5470f49201c4b..26ea42ab5bb7f 100644 --- a/eng/Version.Details.xml +++ b/eng/Version.Details.xml @@ -1,13 +1,5 @@ - - https://github.com/dotnet/corefx - 5cee7c97d602f294e27c582d4dab81ec388f1d7b - - - https://github.com/dotnet/corefx - 5cee7c97d602f294e27c582d4dab81ec388f1d7b - https://github.com/dotnet/corefx 5cee7c97d602f294e27c582d4dab81ec388f1d7b @@ -28,10 +20,6 @@ https://github.com/dotnet/corefx cf64918877d98577363bb40d5eafac52beb80a79 - - https://github.com/dotnet/coreclr - 2c4fb3250989f014550882f5d165cdc36ebdbd08 - https://github.com/dotnet/coreclr 2c4fb3250989f014550882f5d165cdc36ebdbd08 diff --git a/eng/Versions.props b/eng/Versions.props index a68129a97b084..3806d733c880d 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -49,12 +49,8 @@ 3.0.0 5.0.0-alpha1.19563.3 - 5.0.0-alpha1.19563.3 5.0.0-alpha1.19563.3 - 5.0.0-alpha.1.19563.6 - 5.0.0-alpha.1.19562.8 - 5.0.0-alpha.1.19563.6 5.0.0-alpha.1.19563.6 5.0.0-alpha.1.19563.6 5.0.0-alpha.1.19563.3 @@ -106,8 +102,6 @@ optimization.linux-x64.ibc.corefx microsoft.private.intellisense Microsoft.DotNet.XUnitConsoleRunner - Microsoft.Private.CoreFx.NETCoreApp - Microsoft.NETCore.Platforms Microsoft.NETCore.App optimization.IBC.CoreCLR optimization.PGO.CoreCLR diff --git a/eng/helixcorefxtests.proj b/eng/helixcorefxtests.proj deleted file mode 100644 index 7b14fb1d0ca70..0000000000000 --- a/eng/helixcorefxtests.proj +++ /dev/null @@ -1,291 +0,0 @@ - - - - - - - - - - - - - - - - - - <_PropertiesToPass> - __BuildArch=$(__BuildArch); - __BuildOS=$(__BuildOS); - __BuildType=$(__BuildType); - Creator=$(_Creator); - HelixAccessToken=$(_HelixAccessToken); - HelixBuild=$(_HelixBuild); - HelixSource=$(_HelixSource); - HelixTargetQueues=$(_HelixTargetQueues); - HelixType=$(_HelixType); - PublishTestResults=$(_PublishTestResults); - RunCrossGen=$(_RunCrossGen); - TimeoutPerTestCollectionInMinutes=$(_TimeoutPerTestCollectionInMinutes) - - - - - - - - - <_Scenarios Include="$(_Scenarios.Split(','))" /> - - - <_ProjectsToBuild Include="$(MSBuildProjectFile)"> - $(_PropertiesToPass);Scenario=%(_Scenarios.Identity) - - - - - <_BuildInParallel>false - <_BuildInParallel Condition=" '@(_ProjectsToBuild->Count())' > '1' ">true - - - - - - - - - - $(_Creator) - $(_HelixAccessToken) - $(_HelixBuild) - $(_HelixSource) - $(_HelixTargetQueues) - $(_HelixType) - - - $(MSBuildThisFileDirectory)..\ - $(ProjectDir)artifacts\bin\coreclr - $(ProjectDir)artifacts\tests\coreclr\$(__BuildOS).$(__BuildArch).$(__BuildType)\ - true - - - $(ProjectDir)src\coreclr\tests\CoreFX\CoreFX.issues.rsp - - $(TestWorkingDir)testhost\ - $(TestWorkingDir)helix\ - - - shared\$(MicrosoftNETCoreAppPackage)\$(ProductVersion) - - https://dotnetfeed.blob.core.windows.net/dotnet-core - - - - - call RunTests.cmd -r %HELIX_CORRELATION_PAYLOAD% --rsp-file %HELIX_CORRELATION_PAYLOAD%\CoreFX.issues.rsp - ./RunTests.sh -r $HELIX_CORRELATION_PAYLOAD --rsp-file $HELIX_CORRELATION_PAYLOAD/CoreFX.issues.rsp - - - - - - - - - - - - SetStressModes_$(Scenario).cmd - SetStressModes_$(Scenario).sh - - - - - - - $(TestHostRootPath)$(TestEnvFileName) - - - - <_ProjectsToBuild Include="$(ProjectDir)src\coreclr\tests\testenvironment.proj"> - Scenario=$(Scenario);TestEnvFileName=$(TestEnvFilePath);TargetsWindows=$(TargetsWindows) - - - - - - - - - - - - - - <_Scenario Include="$(Scenarios.Split(','))" /> - <_ProjectsToBuild Include="$(MSBuildProjectFile)"> - Scenario=%(_Scenario.Identity) - - - - - - - - - - - - - - <_TargetGroup>netcoreapp - <_AssetManifestPath>$(TestAssetBlobFeedUrl)/corefx-tests/$(MicrosoftPrivateCoreFxNETCoreAppVersion)/$(__BuildOS).$(__BuildArch)/$(_TargetGroup)/corefx-test-assets.xml - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @(HelixPreCommand) - - - - $(__BuildArch) - - $(BuildType) - $(BuildType)-$(Scenario) - - CoreFX - $(TestRunNamePrefix)$(BuildOS) $(BuildArch) $(BuildType) @ - $(TestRunNamePrefix)$(BuildOS) $(BuildArch) $(BuildType) $(Scenario) @ - - - $(_PublishTestResults) - false - - true - true - true - true - - - - - - - - - - - - - - $(TestAssetBlobFeedUrl)/%(Identity) - $(HelixCommand) - $([System.TimeSpan]::FromMinutes($(TimeoutPerTestCollectionInMinutes))) - - - - - - - diff --git a/eng/liveBuilds.targets b/eng/liveBuilds.targets new file mode 100644 index 0000000000000..63d8d1b1a2049 --- /dev/null +++ b/eng/liveBuilds.targets @@ -0,0 +1,90 @@ + + + + $(OSGroup) + $(Configuration) + $(OSGroup) + $(Configuration) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'coreclr', '$(CoreCLROSGroup).$(TargetArchitecture).$(CoreCLRConfiguration)')) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'ref', 'microsoft.netcore.app', '$(LibrariesConfiguration)')) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'ref', 'netcoreapp')) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'pkg', 'netcoreapp', 'runtime', '$(LibrariesOSGroup)-$(LibrariesConfiguration)-$(TargetArchitecture)')) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'runtime', 'netcoreapp-$(LibrariesOSGroup)-$(LibrariesConfiguration)-$(TargetArchitecture)')) + $([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin', 'native', 'netcoreapp-$(LibrariesOSGroup)-$(LibrariesConfiguration)-$(TargetArchitecture)')) + x64 + x86 + x64 + + + + + + + $([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)')) + $([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)','sharedFramework')) + $([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)','$(CoreCLRCrossTargetComponentDirName)','sharedFramework')) + + + + + + + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + $(RuntimeIdGraphDefinitionFile) + $(LiveRuntimeIdentifierGraphPath) + + diff --git a/eng/packaging.props b/eng/packaging.props index f15dc257c425f..a7eb2fa7b041a 100644 --- a/eng/packaging.props +++ b/eng/packaging.props @@ -3,7 +3,6 @@ $(PkgDir)descriptions.json $(RepoRoot)LICENSE.TXT $(RepoRoot)THIRD-PARTY-NOTICES.TXT - $(PkgDir)Microsoft.NETCore.Platforms/runtime.json https://go.microsoft.com/fwlink/?LinkID=799421 https://github.com/dotnet/corefx $(CopyrightNetFoundation) @@ -30,7 +29,7 @@ true true - + false diff --git a/eng/pipelines/coreclr/pr.yml b/eng/pipelines/coreclr/pr.yml index f31456d84a43d..0d8691ff7eb0e 100644 --- a/eng/pipelines/coreclr/pr.yml +++ b/eng/pipelines/coreclr/pr.yml @@ -174,6 +174,8 @@ jobs: - Linux_arm helixQueueGroup: pr helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml + jobParameters: + liveLibrariesBuildConfig: Release # # Release test builds diff --git a/eng/pipelines/coreclr/templates/build-job.yml b/eng/pipelines/coreclr/templates/build-job.yml index 1c15f2722f0e4..2a002786640c2 100644 --- a/eng/pipelines/coreclr/templates/build-job.yml +++ b/eng/pipelines/coreclr/templates/build-job.yml @@ -136,7 +136,7 @@ jobs: displayName: Build product # Build native test components - - script: $(coreClrRepoRootDir)build-test$(scriptExt) skipmanaged $(buildConfig) $(archType) $(crossArg) $(priorityArg) $(clangArg) + - script: $(coreClrRepoRootDir)build-test$(scriptExt) skipmanaged $(buildConfig) $(archType) $(crossArg) $(priorityArg) $(clangArg) skipgeneratelayout displayName: Build native test components # Sign on Windows diff --git a/eng/pipelines/coreclr/templates/crossgen-comparison-job.yml b/eng/pipelines/coreclr/templates/crossgen-comparison-job.yml index b90d31ada6967..4c032e6c19de5 100644 --- a/eng/pipelines/coreclr/templates/crossgen-comparison-job.yml +++ b/eng/pipelines/coreclr/templates/crossgen-comparison-job.yml @@ -10,6 +10,13 @@ parameters: variables: {} pool: '' + # When set to a non-empty value (Debug / Release), it determines libraries + # build configuration to use for the tests. Setting this property implies + # a dependency of this job on the appropriate libraries build and is used + # to construct the name of the Azure artifact representing libraries build + # to use for building the tests. + liveLibrariesBuildConfig: '' + ### Crossgen-comparison job ### ### Ensure that the output of cross-architecture, e.g. x64-hosted-arm-targeting, @@ -23,6 +30,7 @@ jobs: osGroup: ${{ parameters.osGroup }} osSubgroup: ${{ parameters.osSubgroup }} stagedBuild: ${{ parameters.stagedBuild }} + liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }} helixType: 'test/crossgen-comparison/' pool: ${{ parameters.pool }} @@ -59,7 +67,10 @@ jobs: - ${{ parameters.variables }} # Test job depends on the corresponding build job - dependsOn: ${{ format('coreclr_product_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }} + dependsOn: + - ${{ format('coreclr_product_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }} + - ${{ if ne(parameters.liveLibrariesBuildConfig, '') }}: + - ${{ format('libraries_build_{0}_{1}{2}_{3}_{4}', 'netcoreapp', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }} # Run all steps in the container. # Note that the containers are defined in platform-matrix.yml @@ -78,6 +89,16 @@ jobs: artifactName: '$(buildProductArtifactName)' displayName: 'product build' + # Optionally download live-built libraries + - ${{ if ne(parameters.liveLibrariesBuildConfig, '') }}: + - template: /eng/pipelines/common/download-artifact-step.yml + parameters: + unpackFolder: $(librariesDownloadDir) + cleanUnpackFolder: false + artifactFileName: '$(librariesBuildArtifactName)$(archiveExtension)' + artifactName: '$(librariesBuildArtifactName)' + displayName: 'live-built libraries' + # Create directories and ensure crossgen is executable - ${{ if ne(parameters.osGroup, 'Windows_NT') }}: diff --git a/eng/pipelines/coreclr/templates/run-test-job.yml b/eng/pipelines/coreclr/templates/run-test-job.yml index 018733b12d5cd..a5db125e48e02 100644 --- a/eng/pipelines/coreclr/templates/run-test-job.yml +++ b/eng/pipelines/coreclr/templates/run-test-job.yml @@ -93,7 +93,7 @@ jobs: - ${{ if eq(variables['System.TeamProject'], 'internal') }}: - group: DotNet-HelixApi-Access - + - ${{ parameters.variables }} # TODO: update these numbers as they were determined long ago @@ -385,4 +385,4 @@ jobs: pathtoPublish: $(Build.SourcesDirectory)/artifacts/log artifactName: '$(LogNamePrefix)_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)_${{ parameters.testGroup }}' continueOnError: true - condition: always() \ No newline at end of file + condition: always() diff --git a/eng/pipelines/coreclr/templates/xplat-pipeline-job.yml b/eng/pipelines/coreclr/templates/xplat-pipeline-job.yml index e0c1eeaaf190d..58a89ddcefc8b 100644 --- a/eng/pipelines/coreclr/templates/xplat-pipeline-job.yml +++ b/eng/pipelines/coreclr/templates/xplat-pipeline-job.yml @@ -41,7 +41,7 @@ jobs: stagedBuild: ${{ parameters.stagedBuild }} strategy: ${{ parameters.strategy }} pool: ${{ parameters.pool }} - + # arcade-specific parameters condition: ${{ parameters.condition }} continueOnError: ${{ parameters.continueOnError }} @@ -50,7 +50,7 @@ jobs: timeoutInMinutes: ${{ parameters.timeoutInMinutes }} enableMicrobuild: ${{ parameters.enableMicrobuild }} gatherAssetManifests: ${{ parameters.gatherAssetManifests }} - + variables: - ${{ if ne(parameters.testGroup, '') }}: - name: testArtifactRootName @@ -89,7 +89,7 @@ jobs: - name: microsoftNetSdkIlFolderPath value: '$(Build.SourcesDirectory)/.packages/microsoft.net.sdk.il' - + - name: microsoftNetSdkIlArtifactName value: 'MicrosoftNetSdkIlPackage_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_$(archType)_$(buildConfig)' @@ -111,10 +111,7 @@ jobs: - ${{ if ne(parameters.liveLibrariesBuildConfig, '') }}: - librariesBuildArtifactName: ${{ format('libraries_bin_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }} - librariesDownloadDir: $(Build.SourcesDirectory)/artifacts - - ${{ if eq(parameters.osGroup, 'Windows_NT') }}: - - librariesOverrideArg: ' localcorefxpath $(Build.SourcesDirectory) localcorefxconfig ${{ parameters.liveLibrariesBuildConfig }}' - - ${{ if ne(parameters.osGroup, 'Windows_NT') }}: - - librariesOverrideArg: ' localcorefxpath=$(Build.SourcesDirectory) localcorefxconfig=${{ parameters.liveLibrariesBuildConfig }}' + - librariesOverrideArg: ' /p:LibrariesConfiguration=${{ parameters.liveLibrariesBuildConfig }}' - ${{ each variable in parameters.variables }}: - ${{insert}}: ${{ variable }} diff --git a/eng/pipelines/installer/azure-pipelines.yml b/eng/pipelines/installer/azure-pipelines.yml index 8504d0acf463e..0f8562c5e801d 100644 --- a/eng/pipelines/installer/azure-pipelines.yml +++ b/eng/pipelines/installer/azure-pipelines.yml @@ -50,6 +50,15 @@ jobs: # - template: /eng/pipelines/common/checkout-job.yml +# +# Build CoreCLR and Libraries +# +- template: /eng/pipelines/common/platform-matrix.yml + parameters: + jobTemplate: /eng/pipelines/common/build-coreclr-and-libraries-job.yml + buildConfig: release + platformGroup: all + # # Build and Test # @@ -61,3 +70,5 @@ jobs: _BuildConfig: Debug release: _BuildConfig: Release + liveCoreClrBuildConfig: release + liveLibrariesBuildConfig: Release diff --git a/eng/pipelines/installer/jobs/base-job.yml b/eng/pipelines/installer/jobs/base-job.yml index 033c518af0e6b..34ba20e8e4c3f 100644 --- a/eng/pipelines/installer/jobs/base-job.yml +++ b/eng/pipelines/installer/jobs/base-job.yml @@ -44,6 +44,9 @@ jobs: - name: ${{ variable.key }} value: ${{ variable.value }} + - name: OfficialBuildArg + value: '' + # Produce test-signed build for PR and Public builds - ${{ if or(eq(variables['System.TeamProject'], 'public'), in(variables['Build.Reason'], 'PullRequest')) }}: - name: SignType @@ -53,15 +56,17 @@ jobs: - ${{ if eq(variables.isOfficialBuild, true) }}: - name: SignType value: $[ coalesce(variables.OfficialSignType, 'real') ] + - name: OfficialBuildArg + value: /p:OfficialBuildId=$(Build.BuildNumber) - name: LiveOverridePathArgs value: >- - $(CoreCLROverridePathArg) - $(CoreFXOverridePathArg) + $(CoreCLRArtifactsPathArg) + $(LibrariesConfigurationArg) - - name: CoreCLROverridePathArg + - name: CoreCLRArtifactsPathArg value: '' - - name: CoreFXOverridePathArg + - name: LibrariesConfigurationArg value: '' - name: CoreClrDownloadPath @@ -78,8 +83,8 @@ jobs: parameters.liveCoreClrBuildConfig) }} - name: CoreClrDownloadPath value: 'artifacts/transport/coreclr' - - name: CoreCLROverridePathArg - value: /p:CoreCLROverridePath=${{ parameters.buildCommandSourcesDirectory }}$(CoreClrDownloadPath) + - name: CoreCLRArtifactsPathArg + value: /p:CoreCLRArtifactsPath=${{ parameters.buildCommandSourcesDirectory }}$(CoreClrDownloadPath) - name: CoreClrArtifactName value: CoreCLRProduct_$(liveCoreClrLegName) @@ -91,11 +96,11 @@ jobs: parameters.archType, parameters.liveLibrariesBuildConfig) }} - name: LibrariesDownloadPath - value: 'artifacts/transport/libraries' - - name: CoreFXOverridePathArg - value: /p:CoreFXOverridePath=${{ parameters.buildCommandSourcesDirectory }}$(LibrariesDownloadPath) + value: 'artifacts' - name: LibrariesArtifactName value: libraries_bin_$(liveLibrariesLegName) + - name: LibrariesConfigurationArg + value: ' /p:LibrariesConfiguration=${{ parameters.liveLibrariesBuildConfig }}' dependsOn: - checkout @@ -152,6 +157,7 @@ jobs: artifactFileName: '$(LibrariesArtifactName)$(archiveExtension)' artifactName: '$(LibrariesArtifactName)' displayName: 'Libraries artifacts' + cleanUnpackFolder: false - ${{ parameters.buildSteps }} diff --git a/eng/pipelines/installer/jobs/bash-build.yml b/eng/pipelines/installer/jobs/bash-build.yml index 0812658c97b1a..c30ed40b2b295 100644 --- a/eng/pipelines/installer/jobs/bash-build.yml +++ b/eng/pipelines/installer/jobs/bash-build.yml @@ -55,8 +55,8 @@ jobs: CommonMSBuildArgs: >- /p:Configuration=$(_BuildConfig) - /p:OfficialBuildId=$(Build.BuildNumber) /p:TargetArchitecture=${{ parameters.archType }} + $(OfficialBuildArg) # Don't put additionalMSBuildArgs as the last line. It may or may not have extra args. If the # parameter is empty, AzDO replaces it with empty space without chomping the extra newline. diff --git a/eng/pipelines/installer/jobs/osx-build.yml b/eng/pipelines/installer/jobs/osx-build.yml index 6c663d36cf5fa..bff6c67b21382 100644 --- a/eng/pipelines/installer/jobs/osx-build.yml +++ b/eng/pipelines/installer/jobs/osx-build.yml @@ -17,9 +17,9 @@ jobs: - script: >- $(Build.SourcesDirectory)/installer.sh --restore --build --ci --test - /p:OfficialBuildId=$(Build.BuildNumber) /p:StripSymbols=true $(LiveOverridePathArgs) $(CommonMSBuildArgs) + $(OfficialBuildArg) displayName: Build condition: succeeded() diff --git a/eng/pipelines/installer/jobs/windows-build.yml b/eng/pipelines/installer/jobs/windows-build.yml index 82f1a6310e401..9126eed56cd8c 100644 --- a/eng/pipelines/installer/jobs/windows-build.yml +++ b/eng/pipelines/installer/jobs/windows-build.yml @@ -15,10 +15,10 @@ jobs: buildVariables: CommonMSBuildArgs: >- /p:Configuration=$(_BuildConfig) - /p:OfficialBuildId=$(Build.BuildNumber) /p:TargetArchitecture=${{ parameters.archType }} /p:PortableBuild=true /p:SkipTests=${{ parameters.skipTests }} + $(OfficialBuildArg) MsbuildSigningArguments: >- /p:CertificateId=400 /p:DotNetSignType=$(SignType) diff --git a/eng/pipelines/libraries/base-job.yml b/eng/pipelines/libraries/base-job.yml index 6f9fc4db1d623..14e30ad1c92a4 100644 --- a/eng/pipelines/libraries/base-job.yml +++ b/eng/pipelines/libraries/base-job.yml @@ -45,10 +45,10 @@ jobs: - ${{ if ne(parameters.testScope, '') }}: - _testScopeArg: -testscope ${{ parameters.testScope }} - + - ${{ if and(eq(parameters.osGroup, 'Linux'), eq(parameters.osSubGroup, '_musl')) }}: - _runtimeOSArg: /p:RuntimeOS=linux-musl - + - ${{ if and(eq(parameters.osGroup, 'Linux'), eq(parameters.osSubGroup, ''), eq(parameters.archType, 'arm')) }}: - _runtimeOSArg: /p:RuntimeOS=ubuntu.16.04 @@ -69,12 +69,12 @@ jobs: - _coreClrArtifactName: '' - _coreClrDownloadPath: '' - - _coreClrOverridePathArg: '' + - _coreClrArtifactsPathArg: '' - ${{ if ne(parameters.liveCoreClrBuildConfig, '') }}: - _coreClrDownloadPath: '$(Build.SourcesDirectory)/artifacts/transport/coreclr' - _coreClrArtifactName: 'CoreCLRProduct_${{ parameters.osGroup }}${{ parameters.osSubgroup }}_${{ parameters.archType }}_${{ parameters.liveCoreClrBuildConfig }}' - - _coreClrOverridePathArg: ' /p:CoreCLROverridePath=$(_coreClrDownloadPath)' + - _coreClrArtifactsPathArg: ' /p:CoreCLRArtifactsPath=$(_coreClrDownloadPath)' # WebAssembly uses linux implementation detail - ${{ if eq(parameters.osGroup, 'WebAssembly') }}: @@ -90,7 +90,7 @@ jobs: - ${{ if eq(parameters.isOfficialBuild, 'true') }}: - _stripSymbolsArg: -stripSymbols - - _buildArguments: -configuration ${{ parameters.buildConfig }} -ci -arch ${{ parameters.archType }} $(_finalFrameworkArg) $(_stripSymbolsArg) $(_testScopeArg) $(_warnAsErrorArg) $(_runtimeOSArg) $(_msbuildCommonParameters) $(_coreClrOverridePathArg) + - _buildArguments: -configuration ${{ parameters.buildConfig }} -ci -arch ${{ parameters.archType }} $(_finalFrameworkArg) $(_stripSymbolsArg) $(_testScopeArg) $(_warnAsErrorArg) $(_runtimeOSArg) $(_msbuildCommonParameters) $(_coreClrArtifactsPathArg) - ${{ parameters.variables }} dependsOn: diff --git a/eng/pipelines/libraries/build-job.yml b/eng/pipelines/libraries/build-job.yml index 76a615ffad44d..ddfa9676fceb8 100644 --- a/eng/pipelines/libraries/build-job.yml +++ b/eng/pipelines/libraries/build-job.yml @@ -112,11 +112,23 @@ jobs: targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/bin/runtime - task: CopyFiles@2 - displayName: Prepare framework runtime folder to publish + displayName: Prepare shared framework ref assemblies to publish + inputs: + sourceFolder: $(Build.SourcesDirectory)/artifacts/bin/ref/microsoft.netcore.app + targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/bin/ref/microsoft.netcore.app + + - task: CopyFiles@2 + displayName: Prepare shared framework runtime folder to publish inputs: sourceFolder: $(Build.SourcesDirectory)/artifacts/bin/pkg/${{ parameters.framework }}/runtime targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/bin/pkg/${{ parameters.framework }}/runtime + - task: CopyFiles@2 + displayName: Prepare docs folder to publish + inputs: + sourceFolder: $(Build.SourcesDirectory)/artifacts/bin/docs + targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/bin/docs + - task: CopyFiles@2 displayName: Prepare native folder to publish inputs: @@ -128,13 +140,19 @@ jobs: inputs: sourceFolder: $(Build.SourcesDirectory)/artifacts/bin/testhost targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/bin/testhost - + - task: CopyFiles@2 displayName: Prepare artifacts toolset folder to publish inputs: sourceFolder: $(Build.SourcesDirectory)/artifacts/toolset targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/toolset - + + - task: CopyFiles@2 + displayName: Prepare artifacts packages folder to publish + inputs: + sourceFolder: $(Build.SourcesDirectory)/artifacts/packages + targetFolder: $(Build.ArtifactStagingDirectory)/artifacts/packages + - template: /eng/pipelines/common/upload-artifact-step.yml parameters: rootFolder: $(Build.ArtifactStagingDirectory)/artifacts @@ -144,7 +162,7 @@ jobs: tarCompression: $(tarCompression) artifactName: $(librariesBuildArtifactName) displayName: Build Assets - + - ${{ if eq(parameters.runTests, true) }}: - template: /eng/pipelines/libraries/helix.yml parameters: diff --git a/eng/referenceFromRuntime.targets b/eng/referenceFromRuntime.targets index 327fab8cc47fa..33af01af354c4 100644 --- a/eng/referenceFromRuntime.targets +++ b/eng/referenceFromRuntime.targets @@ -1,13 +1,6 @@ - - - - - - - + + AddRuntimeProjectReference; $(PrepareProjectReferencesDependsOn); @@ -24,29 +17,52 @@ - + + $([MSBuild]::NormalizePath('$(LibrariesProjectRoot)', 'restore', 'runtime', 'runtime.depproj')) + + false - _referencePathFromRuntime + _referencePathFromRestoredRuntime + + + Condition="'@(ReferenceFromRuntime)' != ''"> + + + + + + + + <_referencePathFromRuntime Include="@(CoreCLRFiles)" Private="false" /> + <_referencePathFromRuntime Include="@(_referencePathFromRestoredRuntime)" Private="false" /> + + <_referencePathFromRuntime Include="@(ReferenceFromRuntime->'$(RuntimePath)%(Identity).dll')" Condition="'$(IsTestProject)' == 'true'" /> <_referencePathFromRuntimeByFileName Include="@(_referencePathFromRuntime->'%(FileName)')" Condition="'%(_referencePathFromRuntime.Extension)' == '.dll'" > %(Identity) + + + + + + + - <_filteredReferencePathFromRuntimeByFileName Include="@(_referencePathFromRuntimeByFileName)" - Condition="'@(_referencePathFromRuntimeByFileName)' == '@(ReferenceFromRuntime)' and '%(Identity)' != ''"> + <_filteredReferencePathFromRuntimeByFileName Include="@(_referencePathFromRuntimeByFileNameFiltered)" + Condition="'@(_referencePathFromRuntimeByFileNameFiltered)' == '@(ReferenceFromRuntime)' and '%(Identity)' != ''"> @(ReferenceFromRuntime->'%(Aliases)') @@ -57,8 +73,8 @@ %(Identity) - <_filteredReferencePathFromRuntimeByFileName Include="@(_referencePathFromRuntimeByFileName)" - Condition="'@(_referencePathFromRuntimeByFileName)' == '@(_remainingReferenceFromRuntimeWithNI)' and '%(Identity)' != ''"> + <_filteredReferencePathFromRuntimeByFileName Include="@(_referencePathFromRuntimeByFileNameFiltered)" + Condition="'@(_referencePathFromRuntimeByFileNameFiltered)' == '@(_remainingReferenceFromRuntimeWithNI)' and '%(Identity)' != ''"> @(_remainingReferenceFromRuntimeWithNI->'%(Aliases)') @@ -66,8 +82,9 @@ - - + + + <_CoreClrBuildArg Condition="$([MSBuild]::IsOsPlatform(Windows))" Include="-skiptests" /> - <_CoreClrBuildArg Include="-$(Platform)" /> + <_CoreClrBuildArg Condition="'$(TargetArchitecture)' != ''" Include="-$(TargetArchitecture)" /> <_CoreClrBuildArg Include="-$(Configuration.ToLower())" /> <_CoreClrBuildArg Condition="'$(ContinuousIntegrationBuild)' == 'true'" Include="-ci" /> - <_CoreClrBuildArg Condition="$([MSBuild]::IsOsPlatform(Windows)) and ('$(Platform)' == 'x86' or '$(Platform)' == 'x64') and '$(Configuration)' == 'Release'" Include="-enforcepgo" /> + <_CoreClrBuildArg Condition="$([MSBuild]::IsOsPlatform(Windows)) and ('$(TargetArchitecture)' == 'x86' or '$(TargetArchitecture)' == 'x64') and '$(Configuration)' == 'Release'" Include="-enforcepgo" /> <_CoreClrBuildArg Condition="!$([MSBuild]::IsOsPlatform(Windows)) and '$(Configuration)' == 'Release'" Include="-stripsymbols" /> - <_CoreClrBuildArg Condition="'$(OfficialBuildId)' == ''" Include="-officialbuildid=$(OfficialBuildId)" /> + <_CoreClrBuildArg Condition="'$(OfficialBuildId)' != ''" Include="-officialbuildid=$(OfficialBuildId)" /> diff --git a/src/coreclr/src/.nuget/packaging.props b/src/coreclr/src/.nuget/packaging.props index 6d16a9848935c..69365bcd15d35 100644 --- a/src/coreclr/src/.nuget/packaging.props +++ b/src/coreclr/src/.nuget/packaging.props @@ -6,7 +6,6 @@ $(CoreclrDir)/LICENSE.TXT $(CoreclrDir)/THIRD-PARTY-NOTICES.TXT - $(NuGetPackageRoot)/microsoft.netcore.platforms/$(MicrosoftNETCorePlatformsVersion)/runtime.json https://go.microsoft.com/fwlink/?LinkID=799421 https://dot.net diff --git a/src/coreclr/tests/build.proj b/src/coreclr/tests/build.proj index 448cc55025ea6..091430dc951e0 100644 --- a/src/coreclr/tests/build.proj +++ b/src/coreclr/tests/build.proj @@ -22,7 +22,6 @@ - diff --git a/src/coreclr/tests/scripts/patch-corefx.py b/src/coreclr/tests/scripts/patch-corefx.py deleted file mode 100755 index 54dd929892e4c..0000000000000 --- a/src/coreclr/tests/scripts/patch-corefx.py +++ /dev/null @@ -1,263 +0,0 @@ -#!/usr/bin/env python -# -# Licensed to the .NET Foundation under one or more agreements. -# The .NET Foundation licenses this file to you under the MIT license. -# See the LICENSE file in the project root for more information. -# -########################################################################## -########################################################################## -# -# Module: patch-corefx.py -# -# Notes: -# -# Script to overwrite the nuget downloaded corefx libraries with ones -# built from a local enlistment. -# -########################################################################## -########################################################################## - -import argparse -import distutils.dir_util -import os -import re -import shutil -import subprocess -import sys - -########################################################################## -# Globals -########################################################################## - -testing = False - -# This should be factored out of build.sh -Unix_name_map = { - 'Linux': 'Linux', - 'Darwin': 'OSX', - 'FreeBSD': 'FreeBSD', - 'OpenBSD': 'OpenBSD', - 'NetBSD': 'NetBSD', - 'SunOS': 'SunOS' -} - -Is_windows = (os.name == 'nt') - -########################################################################## -# Delete protocol -########################################################################## - -def del_rw(action, name, exc): - os.chmod(name, 0o651) - os.remove(name) - -########################################################################## -# Argument Parser -########################################################################## - -description = 'Tool to patch CoreFx tests on the CoreCLR repo' - -parser = argparse.ArgumentParser(description=description) - -parser.add_argument('-arch', dest='arch', default='x64') -parser.add_argument('-build_type', dest='build_type', default='Debug') -parser.add_argument('-clr_core_root', dest='clr_core_root', default=None) -parser.add_argument('-fx_root', dest='fx_root', default=None) - - -########################################################################## -# Helper Functions -########################################################################## - -def validate_args(args): - """ Validate all of the arguments parsed. - Args: - args (argparser.ArgumentParser): Args parsed by the argument parser. - Returns: - (arch, build_type, clr_core_root, fx_root,) - (str, str, str, str) - Notes: - If the arguments are valid then return them all in a tuple. If not, raise - an exception stating x argument is incorrect. - """ - - arch = args.arch - build_type = args.build_type - clr_core_root = args.clr_core_root - fx_root = args.fx_root - - def validate_arg(arg, check): - """ Validate an individual arg - Args: - arg (str|bool): argument to be validated - check (lambda: x-> bool): test that returns either True or False - : based on whether the check passes. - - Returns: - is_valid (bool): Is the argument valid? - """ - - helper = lambda item: item is not None and check(item) - - if not helper(arg): - raise Exception('Argument: %s is not valid.' % (arg)) - - valid_archs = ['x86', 'x64', 'arm', 'arm64'] - valid_build_types = ['Debug', 'Checked', 'Release'] - - arch = next((a for a in valid_archs if a.lower() == arch.lower()), arch) - build_type = next((b for b in valid_build_types if b.lower() == build_type.lower()), build_type) - - validate_arg(arch, lambda item: item in valid_archs) - validate_arg(build_type, lambda item: item in valid_build_types) - - if clr_core_root is None: - raise Exception('No clr_core_root argument provided') - else: - clr_core_root = os.path.normpath(clr_core_root) - validate_arg(clr_core_root, lambda item: os.path.isdir(clr_core_root)) - - if fx_root is None: - raise Exception('No fx_root argument provided') - else: - fx_root = os.path.normpath(fx_root) - - args = (arch, build_type, clr_core_root, fx_root) - - log('Configuration:') - log(' arch: %s' % arch) - log(' build_type: %s' % build_type) - log(' clr_core_root: %s' % clr_core_root) - log(' fx_root: %s' % fx_root) - - return args - -def log(message): - """ Print logging information - Args: - message (str): message to be printed - """ - - print('[%s]: %s' % (sys.argv[0], message)) - -def test_log(message): - """ Print logging information only if testing mode is enabled - Args: - message (str): message to be printed - """ - if testing: - print('[%s]: %s' % (sys.argv[0], message)) - -def copy_files(source_dir, target_dir): - """ Copy any files in the source_dir to the target_dir. - The copy is not recursive. - The directories must already exist. - Args: - source_dir (str): source directory path - target_dir (str): target directory path - Returns: - Nothing - """ - - global testing - assert os.path.isdir(source_dir) - assert testing or os.path.isdir(target_dir) - - for source_filename in os.listdir(source_dir): - source_pathname = os.path.join(source_dir, source_filename) - if os.path.isfile(source_pathname): - target_pathname = os.path.join(target_dir, source_filename) - log('Copy: %s => %s' % (source_pathname, target_pathname)) - if not testing: - shutil.copy2(source_pathname, target_pathname) - -def patch_coreclr_root(core_root, fx_bin): - """ Walk through the fx bin and patch corefx dlls to the core root. - Args: - core_root (str): the core root path - fx_bin (str): the runtime folder from a corefx build - Returns: - nothing - """ - test_log('Patching coreclr core_root') - - forbidden_names = ['coreclr.dll', - 'system.private.corelib.dll', - 'r2rdump.dll', - 'runincontext.dll', - 'mscordaccore.dll', - 'linuxnonjit.dll', - 'protononjit.dll', - 'mscordbi.dll', - 'clrjit.dll', - 'dbgshim.dll', - 'coreshim.dll', - 'clrgc.dll', - 'superpmi-shim-counter.dll', - 'clretwrc.dll', - 'superpmi-shim-collector.dll', - 'superpmi-shim-simple.dll', - 'jitinterface.dll', - 'mscorrc.debug.dll', - 'mscorrc.dll', - 'sos.dll'] - - test_log('forbidden_names = %s' % forbidden_names) - - for file in os.listdir(fx_bin): - test_log('considering file %s' % file) - - filename = os.path.basename(file) - comparename = filename.lower() - if ( comparename.endswith('.dll') and - comparename not in forbidden_names and - not comparename.startswith('api-ms-core') ): - source_pathname = os.path.join(fx_bin, filename) - target_pathname = os.path.join(core_root, filename) - - test_log ('copying file %s to file %s' % (source_pathname, target_pathname)) - - shutil.copy2(source_pathname, target_pathname) - -########################################################################## -# Main -########################################################################## - -def main(args): - """ - The way this script decides what to patch is by looking at the core - root for a list of dlls, then filtering out any ones built by coreclr. - This leaves us with a list of non-coreclr build dlls. Now we can use - that list to go through the corefx repo and any ones that also exist - in the corefx bin folder are copied over. - """ - - log('Patching CoreFX binaries from local enlistment.') - - arch, build_type, clr_core_root, fx_root = validate_args(args) - - clr_os = 'Windows_NT' if Is_windows else Unix_name_map[os.uname()[0]] - - if not os.path.exists(clr_core_root): - raise Exception('Core root path %s does not exist.' % (clr_core_root)) - - fx_bin = os.path.join(fx_root, - 'artifacts', - 'bin', - 'runtime', - 'netcoreapp-%s-%s-%s' % (clr_os, build_type, arch)) - - if not os.path.exists(fx_bin): - raise Exception('CoreFX bin path %s does not exist.' % (fx_bin)) - - patch_coreclr_root(clr_core_root, fx_bin) - - -########################################################################## -# setup for Main -########################################################################## - -if __name__ == '__main__': - Args = parser.parse_args(sys.argv[1:]) - - main(Args) diff --git a/src/coreclr/tests/src/Common/CoreFX/CoreFX.csproj b/src/coreclr/tests/src/Common/CoreFX/CoreFX.csproj deleted file mode 100644 index 36dccb97ada49..0000000000000 --- a/src/coreclr/tests/src/Common/CoreFX/CoreFX.csproj +++ /dev/null @@ -1,150 +0,0 @@ - - - - - - $(SourceDir)Common\CoreFX\obj - C# - netcoreapp3.0 - SharedLibrary - false - 1.0.5 - Library - true - Microsoft.NETCore.App.deps.json - true - false - .NETCoreApp,Version=v3.0 - netcoreapp3.0 - true - $(MicrosoftNETCoreAppVersion) - - - - - win-$(Platform) - linux-$(Platform) - osx-$(Platform) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - hostfxr - dll - dotnet.exe - hostpolicy - dll - - - - libhostfxr - so - dylib - libhostpolicy - $(HostFxrFileExtension) - dotnet - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - $(SourceDir)Common\CoreFX\obj\project.assets.json - - - - - - - - - - - diff --git a/src/coreclr/tests/src/Common/Directory.Build.targets b/src/coreclr/tests/src/Common/Directory.Build.targets index fae63241b062d..43424eab03de0 100644 --- a/src/coreclr/tests/src/Common/Directory.Build.targets +++ b/src/coreclr/tests/src/Common/Directory.Build.targets @@ -16,20 +16,20 @@ DependsOnTargets="ResolveAssemblyReferences"> - - - - + + + + %(Identity) - - - + + + true true win-arm;win-arm64;win-x64;win-x86;$(TargetRid) + $(BuildArch) + true + Release - - - - - - - - @@ -24,4 +19,11 @@ $(SourceDir)Common\test_dependencies\obj\project.assets.json + + + + + + + diff --git a/src/coreclr/tests/src/Common/test_runtime/test_runtime.csproj b/src/coreclr/tests/src/Common/test_runtime/test_runtime.csproj index a09ddaae79f58..e52382ea35194 100644 --- a/src/coreclr/tests/src/Common/test_runtime/test_runtime.csproj +++ b/src/coreclr/tests/src/Common/test_runtime/test_runtime.csproj @@ -8,15 +8,20 @@ $(NoWarn);NU1603 $(NoWarn);NU1701 true + true - - + + + + + + diff --git a/src/coreclr/tests/src/Directory.Build.targets b/src/coreclr/tests/src/Directory.Build.targets index 275d70b85fd76..920348b6cb082 100644 --- a/src/coreclr/tests/src/Directory.Build.targets +++ b/src/coreclr/tests/src/Directory.Build.targets @@ -44,8 +44,8 @@ - - + + <_CLRTestNeedsProjectToRun>false <_CLRTestNeedsProjectToRun Condition=" '$(_CLRTestNeedsToRun)' and '!$(_CLRTestBuildsExecutable)' ">true @@ -80,10 +80,10 @@ <_CopyNativeProjectBinaries Condition="'$(_WillCLRTestProjectBuild)' != 'true'">false <_CopyNativeProjectBinaries Condition="'$(_CopyNativeProjectBinaries)' == ''">true - + - + @@ -105,7 +105,7 @@ - @@ -199,6 +199,12 @@ + + + + false @@ -207,6 +213,13 @@ + + + + + + - + true + + true + $(BuildArch) + Release + + + + diff --git a/src/coreclr/tests/src/Interop/COM/NETClients/Primitives/NETClientPrimitives.csproj b/src/coreclr/tests/src/Interop/COM/NETClients/Primitives/NETClientPrimitives.csproj index c8ac76c7ac124..0c7d448ed4fe8 100644 --- a/src/coreclr/tests/src/Interop/COM/NETClients/Primitives/NETClientPrimitives.csproj +++ b/src/coreclr/tests/src/Interop/COM/NETClients/Primitives/NETClientPrimitives.csproj @@ -24,8 +24,5 @@ - - $(MicrosoftPrivateCoreFxNETCoreAppVersion) - diff --git a/src/coreclr/tests/src/dirs.proj b/src/coreclr/tests/src/dirs.proj index 36961b47130ae..b6dfdfa8f5f5d 100644 --- a/src/coreclr/tests/src/dirs.proj +++ b/src/coreclr/tests/src/dirs.proj @@ -16,7 +16,6 @@ - diff --git a/src/coreclr/tests/src/runtest.proj b/src/coreclr/tests/src/runtest.proj index 9e578fc1156eb..9488cc0e7f262 100644 --- a/src/coreclr/tests/src/runtest.proj +++ b/src/coreclr/tests/src/runtest.proj @@ -366,16 +366,6 @@ namespace $([System.String]::Copy($(Category)).Replace(".","_").Replace("\",""). Properties="Language=C#;TargetRid=$(TargetRid);RuntimeIdentifier=$(TargetRid)" /> - - - - - - @@ -394,10 +384,6 @@ namespace $([System.String]::Copy($(Category)).Replace(".","_").Replace("\",""). Targets="CreateTestOverlay" Condition=" '$(GenerateRuntimeLayout)'=='true' "/> - - diff --git a/src/installer/Directory.Build.props b/src/installer/Directory.Build.props index 0fb04e36e8866..9670824f2a128 100644 --- a/src/installer/Directory.Build.props +++ b/src/installer/Directory.Build.props @@ -45,7 +45,6 @@ - $([System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture.ToString().ToLowerInvariant()) $(BuildArchitecture) x64 $(TargetArchitecture) @@ -351,7 +350,7 @@ lib - .so + .so .dll .dylib lib diff --git a/src/installer/Directory.Build.targets b/src/installer/Directory.Build.targets index 724a3f5557e2c..3d45d9ee49c92 100644 --- a/src/installer/Directory.Build.targets +++ b/src/installer/Directory.Build.targets @@ -11,6 +11,13 @@ + + + $([MSBuild]::NormalizeDirectory('$(ArtifactsDir)', 'packages', '$(LibrariesConfiguration)')) + $([MSBuild]::NormalizeDirectory('$(LibrariesPackagesDir)', 'Shipping')) + $([MSBuild]::NormalizeDirectory('$(LibrariesPackagesDir)', 'NonShipping')) + + @@ -128,12 +135,12 @@ $(ProductionVersion) - + 0.1.$(VersionSuffix) $([System.String]::Copy('$(HostPackageRelease)').Replace('-', '_')) - + $(ProductionVersion) - + 0.1.$(VersionSuffix) $([System.String]::Copy('$(HostResolverPackageRelease)').Replace('-', '_')) diff --git a/src/installer/pkg/Directory.Build.targets b/src/installer/pkg/Directory.Build.targets index cbe076d1e7f61..d457972bfd7f5 100644 --- a/src/installer/pkg/Directory.Build.targets +++ b/src/installer/pkg/Directory.Build.targets @@ -1,10 +1,6 @@ - - $(NuGetPackageRoot)$(PlatformPackageId.ToLowerInvariant())\$(MicrosoftNETCorePlatformsVersion)\runtime.json - - diff --git a/src/installer/pkg/deps/deps.csproj b/src/installer/pkg/deps/deps.csproj deleted file mode 100644 index f105b8f3826f7..0000000000000 --- a/src/installer/pkg/deps/deps.csproj +++ /dev/null @@ -1,11 +0,0 @@ - - - - netstandard1.0 - - - - - - - diff --git a/src/installer/pkg/projects/Directory.Build.targets b/src/installer/pkg/projects/Directory.Build.targets index b40cddb9dfd36..f8ca05a3026db 100644 --- a/src/installer/pkg/projects/Directory.Build.targets +++ b/src/installer/pkg/projects/Directory.Build.targets @@ -336,4 +336,26 @@ - \ No newline at end of file + + + + + + + + + + + + + diff --git a/src/installer/pkg/projects/netcoreapp/pkg/legacy/Microsoft.NETCore.App.Internal.pkgproj b/src/installer/pkg/projects/netcoreapp/pkg/legacy/Microsoft.NETCore.App.Internal.pkgproj index 8a92a5716b84c..bea575e85c83b 100644 --- a/src/installer/pkg/projects/netcoreapp/pkg/legacy/Microsoft.NETCore.App.Internal.pkgproj +++ b/src/installer/pkg/projects/netcoreapp/pkg/legacy/Microsoft.NETCore.App.Internal.pkgproj @@ -18,5 +18,10 @@ false + + + + + diff --git a/src/installer/pkg/projects/netcoreapp/pkg/workaround/Microsoft.NETCore.App.pkgproj b/src/installer/pkg/projects/netcoreapp/pkg/workaround/Microsoft.NETCore.App.pkgproj index 5a0d9501e728d..d9d6f3423c3dd 100644 --- a/src/installer/pkg/projects/netcoreapp/pkg/workaround/Microsoft.NETCore.App.pkgproj +++ b/src/installer/pkg/projects/netcoreapp/pkg/workaround/Microsoft.NETCore.App.pkgproj @@ -20,7 +20,7 @@ - + diff --git a/src/installer/pkg/projects/netcoreapp/sfx/Microsoft.NETCore.App.SharedFx.sfxproj b/src/installer/pkg/projects/netcoreapp/sfx/Microsoft.NETCore.App.SharedFx.sfxproj index c24c1a8db2e47..3ea98ed5d6c64 100644 --- a/src/installer/pkg/projects/netcoreapp/sfx/Microsoft.NETCore.App.SharedFx.sfxproj +++ b/src/installer/pkg/projects/netcoreapp/sfx/Microsoft.NETCore.App.SharedFx.sfxproj @@ -1,4 +1,5 @@ - + + Microsoft.NETCore.App @@ -42,4 +43,14 @@ + + + + + $(RestoreSources);$(LibrariesShippingPackagesDir) + $(RestoreSources);$(LibrariesNonShippingPackagesDir) + diff --git a/src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets b/src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets index 401fd102d5371..52f64d3386d39 100644 --- a/src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets +++ b/src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets @@ -1,152 +1,98 @@ - - - - + + - $([MSBuild]::NormalizeDirectory('$(CoreCLROverridePath)')) + <_winmdPackageDir>$(NuGetPackageRoot)$(MicrosoftTargetingPackPrivateWinRTPackage.ToLowerInvariant())/$(MicrosoftTargetingPackPrivateWinRTVersion)/ + <_diaSymReaderPackageDir>$(NuGetPackageRoot)microsoft.diasymreader.native/$(MicrosoftDiaSymReaderNativeVersion)/ + + + + <_crossHostArch>x86 + + + <_crossHostArch>x64 + + + <_crossHostArch>x64 - x64 - x86 - x64 - $([MSBuild]::NormalizeDirectory('$(CoreCLROverridePath)','sharedFramework')) - $([MSBuild]::NormalizeDirectory('$(CoreCLROverridePath)','$(CoreCLRCrossTargetComponentDirName)','sharedFramework')) + <_crossDir Condition="'$(_crossHostArch)' != ''">/$(_crossHostArch)_$(TargetArchitecture) - - - - - - true - - - - - - - - - - - - runtimes/$(CoreCLRCrossTargetComponentDirName)_$(TargetArchitecture)/native - - - - - - - + + <_requiredProperty Include="_winmdPackageDir" /> + - + + + + - + + + + <_windowsWinMD Include="$(_winmdPackageDir)**/Windows.winmd" /> + <_diaSymReaderAssembly Include="$(_diaSymReaderPackageDir)**\Microsoft.DiaSymReader.Native.*.dll" /> - - + + <_runtimeDirectory>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'sharedFramework')) + <_crossGenPath>$([MSBuild]::NormalizePath('$(CoreCLRArtifactsPath)', '$(CoreCLRCrossTargetComponentDirName)', 'sharedFramework', 'crossgen$(ApplicationFileExtension)')) + - - + + <_coreLibDirectory>$(CoreCLRArtifactsPath) + <_fxLibDirectory>$(LibrariesSharedFrameworkBinArtifactsPath) + - - Windows_NT - Linux - OSX - FreeBSD - $(CoreCLROSGroup) + <_jitPath>$([MSBuild]::NormalizePath('$(CoreCLRArtifactsPath)', '$(CoreCLRCrossTargetComponentDirName)', 'sharedFramework', '$(LibraryFilePrefix)clrjit$(LibraryFileExtension)')) - - - - - + + <_windowsWinMDDirectory>%(_windowsWinMD.RootDir)%(_windowsWinMD.Directory) + - - + + <_diaSymReaderToolDir>%(_diaSymReaderAssembly.RootDir)%(_diaSymReaderAssembly.Directory) + - - - @(CoreFXReferenceItems) - - - - - - - @(CoreFXReferenceCopyLocalItems) - - - - true - + + + <_diaSymReaderToolDir>$(IntermediateOutputPath) + - - - + + + - - <_coreFXOverrideDocFiles - Condition="'$(CoreFXOverridePath)' != '' And '%(NuGetPackageId)' == '$(MicrosoftPrivateCoreFxNETCoreAppPackage)'" - Include="@(OverriddenFrameworkReferenceFiles->'$(CoreFXOverridePath)/../../bin/docs/%(FileName).xml')" /> - <_docFilesToPackage Include="@(_coreFXOverrideDocFiles)" Condition="Exists('%(Identity)')" /> + + + <_crossgenPlatformDirectories Include="%(_filesToCrossGen.RootDir)%(_filesToCrossGen.Directory)" /> + + <_crossgenPlatformDirectories Include="$(_runtimeDirectory)" /> + <_crossgenPlatformDirectories Include="$(_coreLibDirectory)" /> + <_crossgenPlatformDirectories Include="$(_fxLibDirectory)" /> - - - - <_runtimeDirectory>$(CoreCLROverridePath) - <_crossgenPath>$([MSBuild]::NormalizePath('$(CoreCLROverridePath)', '$(_crossHostArch)', 'crossgen$(ApplicationFileExtension)')) - <_CoreLibSharedDirectory>$(CoreCLROverridePath) - <_jitPath>$([MSBuild]::NormalizePath('$(CoreCLROverridePath)', '$(_crossHostArch)', '$(LibraryFilePrefix)clrjit$(LibraryFileExtension)')) + + + <_pathSeparatorEscaped>$([MSBuild]::Escape($([System.IO.Path]::PathSeparator.ToString()))) + <_crossgenPlatformAssemblies>@(_crossgenPlatformDirectories->'%(Identity)', '$(_pathSeparatorEscaped)') - @@ -155,9 +101,9 @@ calculate it by looking for native in the path --> true - + - + runtimes/$(PackageRID)/lib/$(PackageTargetFramework) runtimes/$(PackageRID)/native @@ -192,6 +138,4 @@ - - diff --git a/src/installer/pkg/projects/netcoreapp/src/netcoreapp.depproj b/src/installer/pkg/projects/netcoreapp/src/netcoreapp.depproj index cc80981a53fbf..b5edf4df303b1 100644 --- a/src/installer/pkg/projects/netcoreapp/src/netcoreapp.depproj +++ b/src/installer/pkg/projects/netcoreapp/src/netcoreapp.depproj @@ -1,15 +1,16 @@ - Microsoft.NETCore.App + false + Windows_NT + Linux + OSX + FreeBSD + $(CoreCLROSGroup) - - - - @@ -19,11 +20,8 @@ - + - - - - - <_ToolsToPackage Include="$(_runtimePackageDir)tools/**/*.*"/> - - $(_runtimePackageId) - - $([System.String]::new('tools/%(RecursiveDir)').TrimEnd('/')) - true - - - runtimes$(_crossDir)/native - true - + + <_docFilesToPackage Include="$(ArtifactsBinDir)/docs/%(LibrariesRefAssemblies.FileName).xml" Condition="Exists('$(ArtifactsBinDir)/docs/%(LibrariesRefAssemblies.FileName).xml')"/> diff --git a/src/installer/test/PrepareTestAssets/PrepareTestAssets.proj b/src/installer/test/PrepareTestAssets/PrepareTestAssets.proj index b2ac07d4cebfc..46b4ba4710734 100644 --- a/src/installer/test/PrepareTestAssets/PrepareTestAssets.proj +++ b/src/installer/test/PrepareTestAssets/PrepareTestAssets.proj @@ -49,6 +49,8 @@ + + false - + true @@ -142,6 +142,7 @@ <_packageRID Condition="'$(PortableBuild)' == 'true'">$(_portableOS)-$(ArchGroup) $(_packageRID) $(RuntimeOS)-$(ArchGroup) + $(ArchGroup) @@ -182,14 +183,14 @@ - + false $(NoWarn);nullable - + @@ -266,9 +267,6 @@ $(TestArchiveTestsRoot)$(OSPlatformConfig)/ $(TestArchiveRoot)runtime/ - - $(MSBuildThisFileDirectory)restore\runtime\runtime.depproj - $(ArtifactsBinDir)pkg\netcoreapp\ref $(ArtifactsBinDir)pkg\netcoreapp\lib diff --git a/src/libraries/Directory.Build.targets b/src/libraries/Directory.Build.targets index 04cb558a8148b..6536d32bc16d5 100644 --- a/src/libraries/Directory.Build.targets +++ b/src/libraries/Directory.Build.targets @@ -1,4 +1,9 @@ + + $(_bc_OSGroup) + $(_bc_ConfigurationGroup) + + @@ -40,15 +45,16 @@ true true + true - + $(BuildConfigurationRefPath) $(RuntimePath) - + $(NETCoreAppPackageRefPath) $(NETCoreAppPackageRuntimePath) @@ -82,7 +88,7 @@ $(RefRootPath)%(Identity)/ - + $(ArtifactsBinDir)runtime/%(Identity)-$(ConfigurationGroup)-$(ArchGroup) diff --git a/src/libraries/restore/dirs.proj b/src/libraries/restore/dirs.proj index cf1b99f6e1e23..9c11cf6e8230e 100644 --- a/src/libraries/restore/dirs.proj +++ b/src/libraries/restore/dirs.proj @@ -12,7 +12,6 @@ - diff --git a/src/libraries/restore/runtime/runtime.depproj b/src/libraries/restore/runtime/runtime.depproj index bfb02bccfcd63..d6520703f7972 100644 --- a/src/libraries/restore/runtime/runtime.depproj +++ b/src/libraries/restore/runtime/runtime.depproj @@ -5,22 +5,16 @@ $(ToolRuntimeRID) $(NoWarn);NU1603;NU1605 true + $(DefaultOSGroup) + $(ConfigurationGroup) + false - - - 3.0.0-rc2.19462.14 - 4.6.0-rc2.19462.14 - - - - - @@ -43,7 +37,7 @@ hostfxr libhostfxr - + true false @@ -68,39 +62,16 @@ - - - $(CoreCLROverridePath)/PDB - - - - - - - - - - - - <_CoreLibFile Include="@(ReferenceCopyLocalPaths)" Condition="'%(FileName)' == 'System.Private.CoreLib'" /> - - - <_CoreLibFilePath>%(_CoreLibFile.FullPath) - <_CoreLibPackagePath>$(_CoreLibFilePath.SubString(0, $(_CoreLibFilePath.IndexOf('runtimes')))) - - - + - - + @@ -115,5 +86,5 @@ - + diff --git a/src/libraries/shims/manual/Directory.Build.props b/src/libraries/shims/manual/Directory.Build.props index 57cdeb82eef66..d133e1342d10b 100644 --- a/src/libraries/shims/manual/Directory.Build.props +++ b/src/libraries/shims/manual/Directory.Build.props @@ -3,7 +3,7 @@ manual.$(MSBuildProjectName) - + @@ -13,6 +13,8 @@ true $(ArtifactsBinDir)runtime/$(TargetGroup)-$(_bc_OSGroup)-$(_bc_ConfigurationGroup)-$(ArchGroup)/ + $(_bc_OSGroup) + $(_bc_ConfigurationGroup) @@ -20,4 +22,10 @@ Include="$(RefPath)System.*.dll;$(RefPath)Microsoft.Win32.*.dll;$(RefPath)netstandard.dll" Exclude="$(RefPath)$(MSBuildProjectName).dll" /> + + + + + + diff --git a/src/libraries/src.builds b/src/libraries/src.builds index 508f68f077a86..427c01d8e7c25 100644 --- a/src/libraries/src.builds +++ b/src/libraries/src.builds @@ -9,6 +9,8 @@ + +