Skip to content

Commit

Permalink
Simplify webassembly targets (dotnet#41869)
Browse files Browse the repository at this point in the history
Add a subset for compiling only the wasm runtime: `mono.wasmruntime`
This allows us to rebuild the wasm runtime without building the full libraries subset.

Updated documentation with new guidance.

Updated how we build the dotnet.timezones.blat.
  • Loading branch information
akoeplinger authored Sep 4, 2020
1 parent a9f6fd6 commit e658ddb
Show file tree
Hide file tree
Showing 19 changed files with 128 additions and 337 deletions.
8 changes: 6 additions & 2 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,24 @@

<DocsDir>$([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'docs'))</DocsDir>
<ManPagesDir>$([MSBuild]::NormalizeDirectory('$(DocsDir)', 'manpages'))</ManPagesDir>

<!-- System.Private.CoreLib -->
<CoreLibSharedDir>$([MSBuild]::NormalizeDirectory('$(LibrariesProjectRoot)', 'System.Private.CoreLib', 'src'))</CoreLibSharedDir>
<CoreLibProject Condition="'$(RuntimeFlavor)' == 'CoreCLR'">$([MSBuild]::NormalizePath('$(CoreClrProjectRoot)', 'src', 'System.Private.CoreLib', 'System.Private.CoreLib.csproj'))</CoreLibProject>
<CoreLibProject Condition="'$(RuntimeFlavor)' == 'Mono'">$([MSBuild]::NormalizePath('$(MonoProjectRoot)', 'netcore', 'System.Private.CoreLib', 'System.Private.CoreLib.csproj'))</CoreLibProject>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetsMobile)' == 'true'">
<AppleAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'AppleAppBuilder', 'Debug', '$(NetCoreAppCurrent)'))</AppleAppBuilderDir>
<AndroidAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'AndroidAppBuilder', 'Debug', '$(NetCoreAppCurrent)'))</AndroidAppBuilderDir>
<WasmAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmAppBuilder', 'Debug', '$(NetCoreAppCurrent)', 'publish'))</WasmAppBuilderDir>
<GenerateWasmBundleDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'GenerateWasmBundle', 'Debug', '$(NetCoreAppCurrent)', 'publish'))</GenerateWasmBundleDir>
<WasmBuildTasksDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmBuildTasks', 'Debug', '$(NetCoreAppCurrent)', 'publish'))</WasmBuildTasksDir>
<MonoAOTCompilerDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'MonoAOTCompiler', 'Debug', '$(NetCoreAppCurrent)'))</MonoAOTCompilerDir>

<AppleAppBuilderTasksAssemblyPath>$([MSBuild]::NormalizePath('$(AppleAppBuilderDir)', 'AppleAppBuilder.dll'))</AppleAppBuilderTasksAssemblyPath>
<AndroidAppBuilderTasksAssemblyPath>$([MSBuild]::NormalizePath('$(AndroidAppBuilderDir)', 'AndroidAppBuilder.dll'))</AndroidAppBuilderTasksAssemblyPath>
<WasmAppBuilderTasksAssemblyPath>$([MSBuild]::NormalizePath('$(WasmAppBuilderDir)', 'WasmAppBuilder.dll'))</WasmAppBuilderTasksAssemblyPath>
<GenerateWasmBundlesAssemblyPath>$([MSBuild]::NormalizePath('$(GenerateWasmBundleDir)', 'GenerateWasmBundle.dll'))</GenerateWasmBundlesAssemblyPath>
<WasmBuildTasksAssemblyPath>$([MSBuild]::NormalizePath('$(WasmBuildTasksDir)', 'WasmBuildTasks.dll'))</WasmBuildTasksAssemblyPath>
<MonoAOTCompilerTasksAssemblyPath>$([MSBuild]::NormalizePath('$(MonoAOTCompilerDir)', 'MonoAOTCompiler.dll'))</MonoAOTCompilerTasksAssemblyPath>
</PropertyGroup>

Expand Down
79 changes: 43 additions & 36 deletions docs/workflow/building/libraries/webassembly-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,102 +4,109 @@

If you haven't already done so, please read [this document](../../README.md#Build_Requirements) to understand the build requirements for your operating system.

An installation of the emsdk needs to be installed. Follow the installation guide [here](https://emscripten.org/docs/getting_started/downloads.html#sdk-download-and-install). Once installed the EMSDK_PATH needs to be set:
The Emscripten SDK (emsdk) needs to be installed. Follow the installation guide [here](https://emscripten.org/docs/getting_started/downloads.html#sdk-download-and-install) or run `make -C src/mono/wasm provision-wasm` to install emsdk into `src/mono/wasm/emsdk`.

On Linux and MacOSX:
Once installed the `EMSDK_PATH` environment variable needs to be set:

On Linux and macOS:

```bash
export EMSDK_PATH=PATH_TO_SDK_INSTALL/emsdk
export EMSDK_PATH=<FULL_PATH_TO_SDK_INSTALL>/emsdk
```

## Building everything

At this time no other build configurations are necessary to start building for WebAssembly. The CoreLib for WebAssembly build configurations will be built by default using the WebAssembly configuration shown below.
At this time no other build dependencies are necessary to start building for WebAssembly.

This document explains how to work on libraries. In order to work on library projects or run library tests it is necessary to have built the runtime to give the libraries something to run on. If you haven't already done so, please read [this document](../../README.md#Configurations) to understand configurations.
This document explains how to work on the runtime or libraries. If you haven't already done so, please read [this document](../../README.md#Configurations) to understand configurations.

For Linux and macOS:

For Linux and MacOSX:
```bash
./build.sh --arch wasm --os Browser --configuration release
./build.sh -os Browser -configuration Release
```

Detailed information about building and testing runtimes and the libraries is in the documents linked below.
Artifacts will be placed in `artifacts/bin/microsoft.netcore.app.runtime.browser-wasm/Release/`. When rebuilding with `build.sh` after a code change, you need to ensure that the `mono.wasmruntime` and `libs.pretest` subsets are included even for a Mono-only change or this directory will not be updated (details below).

## How to build native components only
## Building Mono's System.Private.CoreLib or runtime

The libraries build contains some native code. This includes shims over libc, openssl, gssapi, and zlib. The build system uses CMake to generate Makefiles using clang. The build also uses git for generating some version information.
If you are working on core parts of Mono you will probably need to build the Mono runtime and [System.Private.CoreLib](../../../design/coreclr/botr/corelib.md) which can be built with the following:

**Examples**
```bash
./build.sh mono -os Browser -c Debug|Release
```

To build just System.Private.CoreLib without the Mono runtime you can use the `Mono.CoreLib` subset:

- Building in debug mode for platform wasm and Browser operating system
```bash
./build.sh libs.native --arch wasm --os Browser
./build.sh mono.corelib -os Browser -c Debug|Release
```

- Building in release mode for platform wasm and Browser operating system
To build just the Mono runtime without System.Private.CoreLib use the `Mono.Runtime` subset:

```bash
./build.sh libs.native --arch wasm --os Browser -c Release
./build.sh mono.runtime -os Browser -c Debug|Release
```

## How to build mono System.Private.CoreLib
Building both Mono/System.Private.CoreLib and the managed libraries:

If you are working on core parts of mono libraries you will probably need to build the [System.Private.CoreLib](../../../design/coreclr/botr/corelib.md) which can be built with the following:
```bash
./build.sh mono+libs -os Browser -c Debug|Release
```

## Building the WebAssembly runtime files

The WebAssembly implementation files are built after the libraries source build and made available in the artifacts folder. If you are working on the code base and need to compile just these modules then building the `Mono.WasmRuntime` subset will allow one to do that:

```bash
./build.sh mono --arch wasm --os Browser -c Release
./build.sh mono.wasmruntime -os Browser -c Debug|Release
```

To build just SPC without mono you can use the Mono.CoreLib subset.
## Updating in-tree runtime pack

If you don't run the full `Libs` subset then you can use the `Libs.PreTest` subset to copy updated runtime/corelib binaries to the runtime pack which is used for running tests:

```bash
./build.sh mono.corelib --arch wasm --os Browser -c Release
./build.sh libs.pretest -os Browser -c Debug|Release
```

Building the managed libraries as well:
## Building libraries native components only

The libraries build contains some native code. This includes shims over libc, openssl, gssapi, and zlib. The build system uses CMake to generate Makefiles using clang. The build also uses git for generating some version information.

```bash
./build.sh mono+libs --arch wasm --os Browser -c Release
./build.sh libs.native -os Browser -c Debug|Release
```

## Building individual libraries

Individual projects and libraries can be build by specifying the build configuration.

Building individual libraries
**Examples**

- Build all projects for a given library (e.g.: System.Net.Http) including the tests

```bash
./build.sh --arch wasm --os Browser -c Release --projects src/libraries/System.Net.Http/System.Net.Http.sln
./build.sh -os Browser -c Release --projects <full-repository-path>/src/libraries/System.Net.Http/System.Net.Http.sln
```

- Build only the source project of a given library (e.g.: System.Net.Http)

```bash
./build.sh --arch wasm --os Browser -c Release --projects src/libraries/System.Net.Http/src/System.Net.Http.csproj
./build.sh -os Browser -c Release --projects <full-repository-path>/src/libraries/System.Net.Http/src/System.Net.Http.csproj
```

More information and examples can be found in the [libraries](./README.md#building-individual-libraries) document.

## Building the WebAssembly runtime files

The WebAssembly implementation files are built and made available in the artifacts folder. If you are working on the code base and need to compile just these modules then the following will allow one to do that.

For Linux and MacOSX:
```bash
./dotnet.sh build /p:Configuration=Debug|Release /p:TargetArchitecture=wasm /p:TargetOS=Browser src/libraries/src.proj /t:BuildWasmRuntimes
```
## Notes

__Note__: A `Debug` build sets the following environment variables by default. When built from the command line this way the `Configuration` value is case sensitive.
A `Debug` build sets the following environment variables by default:

- debugging and logging which will log garbage collection information to the console.

```
monoeg_g_setenv ("MONO_LOG_LEVEL", "debug", 0);
monoeg_g_setenv ("MONO_LOG_MASK", "gc", 0);
MONO_LOG_LEVEL=debug
MONO_LOG_MASK=gc
```

#### Example:
Expand Down
26 changes: 1 addition & 25 deletions docs/workflow/building/mono/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,31 +51,7 @@ The build has a number of options that you can learn about using build -?.

### WebAssembly

In addition to the normal build requirements, WebAssembly builds require a local emsdk to be downloaded. This can either be external or acquired via a make target.

To acquire it externally, move to a directory outside of the runtime repository and run:
```bash
git clone https://github.com/emscripten-core/emsdk.git
```

To use the make target, from the root of the runtime repo:
```bash
cd src/mono/wasm
make provision-wasm
cd ../../..
```

When building for WebAssembly, regardless of the machine architecture, you must set the `EMSDK_PATH` environmental variable and architecture/os, calling build.sh like so:
```bash
EMSDK_PATH={path to emsdk repo} ./build.sh --subset mono+libs --arch wasm --os browser -c release
```

If using the locally provisioned emsdk, this will be:
```bash
EMSDK_PATH={path to runtime repo}/src/mono/wasm/emsdk ./build.sh --subset mono+libs --arch wasm --os browser -c release
```

Artifacts will be placed in `artifacts/bin/microsoft.netcore.app.runtime.browser-wasm/Release/`. When rebuilding with `build.sh`, you _must_ rebuild with `mono+libs` even for mono-only changes, or this directory will not be updated. Alternative, you can rebuild just the runtime-specific bits from the `src/mono/wasm` directory by running either `make runtime` or `make corlib` when modifying Mono or System.Private.CoreLib respectively.
See the instructions in [../libraries/webassembly-instructions.md].

## Packages

Expand Down
4 changes: 2 additions & 2 deletions docs/workflow/testing/libraries/testing-wasm.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@ PATH=/Users/<your_user>/.jsvu/:$PATH V8

Now we're ready to build everything for WebAssembly (for more details, please read [this document](../../building/libraries/webassembly-instructions.md#building-everything)):
```bash
./build.sh --arch wasm --os Browser -c release
./build.sh -os Browser -c Release
```
and even run tests one by one for each library:
```
./build.sh --subset libs.tests -t --arch wasm --os Browser -c release
./build.sh libs.tests -test -os Browser -c Release
```

### Running individual test suites
Expand Down
6 changes: 6 additions & 0 deletions eng/Subsets.props
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
<DefaultCoreClrSubsets>clr.runtime+linuxdac+clr.corelib+clr.nativecorelib+clr.tools+clr.packages</DefaultCoreClrSubsets>

<DefaultMonoSubsets Condition="'$(MonoEnableLLVM)' == 'true' and '$(MonoLLVMDir)' == ''">mono.llvm+</DefaultMonoSubsets>
<DefaultMonoSubsets Condition="'$(TargetOS)' == 'Browser'">$(DefaultMonoSubsets)mono.wasmruntime+</DefaultMonoSubsets>
<DefaultMonoSubsets>$(DefaultMonoSubsets)mono.runtime+mono.corelib+mono.packages</DefaultMonoSubsets>

<DefaultLibrariesSubsets>libs.native+libs.ref+libs.src+libs.pretest+libs.packages</DefaultLibrariesSubsets>
Expand Down Expand Up @@ -94,6 +95,7 @@
<SubsetName Include="Mono.Runtime" Description="The Mono .NET runtime." />
<SubsetName Include="Mono.CoreLib" Description="The managed System.Private.CoreLib library for Mono." />
<SubsetName Include="Mono.Packages" Description="The projects that produce NuGet packages for the Mono runtime." />
<SubsetName Include="Mono.WasmRuntime" Description="The WebAssembly runtime." />

<!-- Libs -->
<SubsetName Include="Libs" Description="The libraries native part, refs and source assemblies, test infra and packages, but NOT the tests (use Libs.Tests to request those explicitly)" />
Expand Down Expand Up @@ -195,6 +197,10 @@
<ProjectToBuild Include="$(LibrariesProjectRoot)src.proj" Category="libs" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+mono.wasmruntime+'))">
<ProjectToBuild Include="$(MonoProjectRoot)wasm\wasm.proj" Category="mono" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+libs.pretest+'))">
<ProjectToBuild Include="$(LibrariesProjectRoot)pretest.proj" Category="libs" />
</ItemGroup>
Expand Down
5 changes: 5 additions & 0 deletions eng/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,11 @@ if [ ${#actInt[@]} -eq 0 ]; then
arguments="-restore -build $arguments"
fi

if [ "$os" = "Browser" ] && [ "$arch" != "wasm" ]; then
# override default arch for Browser, we only support wasm
arch=wasm
fi

initDistroRid $os $arch $crossBuild $portableBuild

# URL-encode space (%20) to avoid quoting issues until the msbuild call in /eng/common/tools.sh.
Expand Down
20 changes: 1 addition & 19 deletions eng/testing/tests.mobile.targets
Original file line number Diff line number Diff line change
Expand Up @@ -151,24 +151,6 @@
AssemblySearchPaths="@(AssemblySearchPaths)" />
</Target>

<Import Project="$(MonoProjectRoot)wasm\timezones.props" Condition="'$(TargetOS)' == 'Browser'" />
<UsingTask TaskName="CompileTimeZoneData"
AssemblyFile="$(GenerateWasmBundlesAssemblyPath)" />
<Target Condition="'$(TargetOS)' == 'Browser'" Name="LoadTimeZone" >
<CompileTimeZoneData
TimeZones="@(TimeZoneToBundle)"
InputDirectory="$([MSBuild]::NormalizePath('$(GenerateWasmBundleDir)', '..', 'data'))"
OutputDirectory="$(BundleDir)obj/data/output"/>
</Target>

<UsingTask TaskName="GenerateWasmBundle"
AssemblyFile="$(GenerateWasmBundlesAssemblyPath)" />
<Target Condition="'$(TargetOS)' == 'Browser'" Name="BundleWasmTestData" DependsOnTargets="LoadTimeZone">
<GenerateWasmBundle
InputDirectory="$(BundleDir)obj/data/output"
OutputFileName="$(MicrosoftNetCoreAppRuntimePackRidDir)dotnet.timezones.blat" />
</Target>

<Target Name="AddTestRunnersToPublishedFiles"
AfterTargets="ComputeResolvedFilesToPublishList">
<ItemGroup>
Expand All @@ -184,5 +166,5 @@
<Target Name="PublishTestAsSelfContained"
Condition="'$(IsCrossTargetingBuild)' != 'true'"
AfterTargets="Build"
DependsOnTargets="Publish;BundleTestAppleApp;BundleTestAndroidApp;BundleTestWasmApp;BundleWasmTestData;ArchiveTests" />
DependsOnTargets="Publish;BundleTestAppleApp;BundleTestAndroidApp;BundleTestWasmApp;ArchiveTests" />
</Project>
4 changes: 0 additions & 4 deletions src/libraries/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -245,10 +245,6 @@
<NetCoreAppCurrentRefPath>$([MSBuild]::NormalizeDirectory('$(RefRootPath)', '$(NetCoreAppCurrent)'))</NetCoreAppCurrentRefPath>
<NetCoreAppCurrentRuntimePath>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'runtime', '$(NetCoreAppCurrent)-$(TargetOS)-$(Configuration)-$(TargetArchitecture)'))</NetCoreAppCurrentRuntimePath>

<!-- System.Private.CoreLib -->
<CoreLibProject Condition="'$(RuntimeFlavor)' == 'CoreCLR'">$([MSBuild]::NormalizePath('$(CoreClrProjectRoot)', 'src', 'System.Private.CoreLib', 'System.Private.CoreLib.csproj'))</CoreLibProject>
<CoreLibProject Condition="'$(RuntimeFlavor)' == 'Mono'">$([MSBuild]::NormalizePath('$(MonoProjectRoot)', 'netcore', 'System.Private.CoreLib', 'System.Private.CoreLib.csproj'))</CoreLibProject>

<!-- Helix properties -->
<OSPlatformConfig>$(TargetOS).$(Platform).$(Configuration)</OSPlatformConfig>
<AnyOSPlatformConfig>AnyOS.AnyCPU.$(Configuration)</AnyOSPlatformConfig>
Expand Down
2 changes: 0 additions & 2 deletions src/libraries/src.proj
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@
Properties="$(TraversalGlobalProperties)" />
</Target>

<Import Condition="'$(TargetOS)' == 'Browser'" Project="$(MonoProjectRoot)wasm\wasm.targets" />

<Target Name="BuildManualShims"
AfterTargets="BuildNonNetCoreAppProjects"
Condition="'@(ManualShimProject)' != ''">
Expand Down
Loading

0 comments on commit e658ddb

Please sign in to comment.