[SUGGEST] Introduce sbt

[i10416]

I Introduced sbt in order to make it easier to run test and build scripts.

• Windows 10
  • run
  • assembly
  • package
• Linux Ubuntu 20.04 (Example.java setTitle() throws nullPointerException for some reasons.)
  • run
  • assembly
  • package : need ubuntu 18.04 to run jpackage.
• macos
  This PR enables these features below.

// at project root
// run cmake and ninja, then copy the binary at <os>/src/main/resources
sbt packArtifact

// compile
sbt compile

// clean shared/build, shared/target, <os>/build,<os>/target
sbt clean

// clean linux/build, linux/target
sbt linux/clean

// publish to local ivy
sbt publishLocal

// publish to local m2
sbt publishM2

// at examples/metrics

// run example application
sbt run

// generate fat jar
sbt assembly

// package example application. generate linux pacakge at target/linux ,msi at target/windows
sbt packageApp
// note: For some reasons, java 16's jpackage does not work on windows, so I use sbt-native-packager feature.

In this PR

• Python scripts are replaced with build tasks.
• jwm-shared, jwm-<platform>, and the example project are also managed by sbt.
• To use jpackage mentioned Build demo app package #104, I use sbt native packager on Windows as a work around because Java 16's jpackage fails on Windows for some reasons.

It is possible to replace sbt with maven since sbt uses (almost) the same directory structure as maven or gradle. But I think sbt is much suitable for cross builid application like this.

[tonsky]

Hi! I’d be happy to move to something more industry-standard than python scripts, but I have two major concerns:

• Performance. With python I can usually get new built up and running under a second after I make a change in the code. For maven, for example, it only takes a second to start the maven itself. This requirement rules out Maven and Gradle.
• Portability. My primary machine is M1 mac, looks like SBT is not available there:

Caused by: scala.MatchError: aarch64 (of class java.lang.String)
	at BuildNative$.getArch(BuildNative.scala:11)
	at BuildNative$.arch$lzycompute(BuildNative.scala:5)
	at BuildNative$.arch(BuildNative.scala:5)
	at BuildNative$.getLibName(BuildNative.scala:20)

[~/ws/jwm] brew install sbt                            
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 1 formula.

sbt: The x86_64 architecture is required for this software.
Error: sbt: An unsatisfied requirement failed this build.

A few questions I have for you:

• How is SBT performance? E.g. changing one native file or one Java file, recompiling and running an example, how long does the whole thing takes?
• You said that “make it easier to run test and build scripts”. I would like to hear more, what is hard with current Python scripts? Can we maybe address those problems somehow?

This is python scripts performance numbers for comparison (I changed Log.cc and Example.java):

[~/ws/jwm] time ./script/run.py
Building libjwm_arm64.dylib
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/tonsky/ws/jwm/macos/build
[2/2] Linking CXX shared library libjwm_arm64.dylib
Compiling 1 java files to examples/target/classes: ['examples/java/org/jetbrains/jwm/examples/Example.java']
./script/run.py  2.08s user 0.17s system 178% cpu 1.262 total

[i10416]

Portability

sbt from brew is not up to date.
I found that the latest sbt supports m1 mac. You need to download universal package version from github

Excuse me for taking the time. I should check that in advance.

https://github.com/sbt/sbt/releases/

I am to blame for that scala.MatchError you give, not sbt compatibility. sbt itself has good portability.

project/BuildNative.scala

  def getArch(): String = {
    System.getProperty("os.arch").toLowerCase match {
      // here should be "arm64" | "aarch64"
      case "arm64" => "arm64"
      case "amd64" | "x86_64" => "x64"
    }
  }

Performance

sbt takes about 1s > for stating up😖 but taking these facts

1. sbt is supposed to be kept running while development
2. sbt can watch Java source changes and automatically execute incremental compile (run with tilde~ like ~compile)
3. "watch and re-run" is available from other commands too
4. sbt can efficiently schedule build tasks (independent tasks can run parallel, e.g. sbt can concurrently check Java, cmake, ninja presence)

into consideration, I think performance is decent enough.

If you keep running ~compile, it does not take much time.

tl;dr see this build performance after editing Log.cc and Example.java
#155 (comment)

run example application at examples/metrics

 run
[info] compiling 14 Java sources toJWM-dev\examples\metrics\target
  | => metrics / update 0s                                                                                                        t\classes ...
  | => metrics / Compile / compileIncremental 2s                                                                                  0m
[info] running org.jetbrains.jwm.examples.Example 

(These metrics below are measured in sbt shell on Windows. Linux is faster by around 2-5s and I think significant figures should be more precise.)

general

clean all(shared, win, macos, linux) including native build.

sbt:jwm> clean
[success] Total time: 1 s, completed Sep 23, 2021,

java stuffs

Compile for the first time

shared/c                                                                                                                 c
ompile
[info] compiling 39 Java sources to C:JWM-dev\shared\targ
                                                                                                                                  get\classes ...
[success] Total time: 3 s, completed 2021/09/23 

windows/compile
[info] compiling 2 Java sources to JWM-dev\windows\target\classes ...
[success] Total time: 1 s, completed 2021/09/23 0:54:40

linux/compile
[info] compiling 1 Java source to JWM-dev\linux\target\classes ...
[success] Total time: 1 s, completed 2021/09/23

macos/compile
[info] compiling 2 Java sources to JWM-dev\macos\target\classes ...
[success] Total time: 1 s, completed 2021/09/23 

// clean all target/*
clean
[success] Total time: 0 s, completed 2021/09/23

compile all. note that windows/compile, macos/compile, linux/compile runs paralell.

compile
[info] compiling 39 Java sources to JWM-dev\shared\target\classes .
  | => shared / update 0s                                                                                                         ...
[success] Total time: 3 s, completed 2021/09/23 

compile example application

sbt:metrics> compile
[success] Total time: 0 s, completed 2021/09/23

generate fat jar for example application

> assembly
[success] Total time: 12 s, completed 2021/09/23

package app

packageApp
[info] Wrote 
[info] Windows Installer XML Toolset Compiler version 3.11.2.4516
[info] Copyright (c) .NET Foundation and contributors. All rights reserved.
[info] metrics.wxs
[info] Windows Installer XML Toolset Linker version 3.11.2.4516
[info] Copyright (c) .NET Foundation and contributors. All rights reserved.
use windows/packageBin from sbt-native-packager instead.
save package installer atJWM-dev\examples\metrics/target/windows
[success] Total time: 9 s, completed 2021/09/23

native stuffs

This is internally running the same script as Python.

Running cmake, ninja for the first time in the sbt shell

[info] -- The CXX compiler identification is MSVC 19.29.30133.0
[info] -- Detecting CXX compiler ABI info
[info] -- Detecting CXX compiler ABI info - done
[info] -- Check for working CXX compiler: 
[info] -- Build files have been written to: JWM-dev/windows/build
  | => root / cmake 1s                                                                             [0m
Running ninja
[info] [1/21] Building CXX object JWM-dev\shared
  | => root / ninja 7s                                                                                                            d\src\main\cc\impl\Managed.cc.obj
[info] [2/21] Building CXX object JWM-dev\shared
  | => root / ninja 7s                                                                                                            d\src\main\cc\StringUTF16.cc.obj
[info] [3/21] Building CXX object JWM-dev\shared
  | => root / ninja 7s                                                                                                            d\src\main\cc\impl\RefCounted.cc.obj
[info] [4/21] Building CXX object JWM-dev\shared
  | => root / ninja 7s                                                                                                            d\src\main\cc\Window.cc.obj



[info] [16/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\WindowWin32.cc.obj
                                                                                                                                  J

[info] [17/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Common.cc.obj
  | => root / ninja 7s                                                                                                            0m
[info] [18/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Fence.cc.obj
[info] [19/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Device.cc.obj
  | => root / ninja 7s                                                                                                            0m
[info] [20/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12SwapChain.cc.obj
  | => root / ninja 7s                                                                                                            j
[info] [21/21] Linking CXX shared library jwm_x64.dll
Build JWM-dev\windows\build\jwm_x64.dll
copying artifact under resource directory
[success] Total time: 8 s, completed Sep 23, 2021

Running cmake, ninja from the second time onwards in the sbt shell

[info] [1/2] Building CXX object CMakeFiles\jwm.dir\src\main\cc\AppWin32.cc.obj
[info] [2/2] Linking CXX shared library jwm_x64.dll
Build workspace\JWM-dev\windows\build\jwm_x64.dll
copying artifact under resource directory
[success] Total time: 2 s, completed Sep 23, 2021

other advantages

• sbt lancher will fetch specified version of sbt, which improves reproducibility.

[i10416]

You said that “make it easier to run test and build scripts”. I would like to hear more, what is hard with current Python scripts? Can we maybe address those problems somehow?

General
First, Python scripts are imperative. I prefer declarative style.

Although it is possible to do that via Python script, I think it is intuitive for developers to use build tools and package manager to manage projects.

Build tools have a standard way of doing stuffs and developers should obey the rules build tools impose on them to some extent, which reduces LoC and keeps code clean, but Python scripts does not.

For instance, build tools enable developers to easily specify dependency(local/remote), versions, etc... Many tasks such as managing version, generating fat.jar, releasing app to local repo or remote repo, packaging by jpackage installer, etc... become easier if you use build tools and its ecosystem (as long as you are on the rails).

In fact, this can be written in much more declarative way.

  os.chdir(os.path.dirname(__file__) + "/../" + common.system)
  rev = revision.revision()

  artifact = "jwm-" + common.system + "-" + common.arch

  with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.xml", "r+") as f:
    pomxml = f.read()
    f.seek(0)
    f.write(pomxml.replace("0.0.0-SNAPSHOT", rev))
    f.truncate()

  with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.properties", "r+") as f:
    pomprops = f.read()
    f.seek(0)
    f.write(pomprops.replace("0.0.0-SNAPSHOT", rev))
    f.truncate()

  with open(os.path.join('build', 'jwm.version'), 'w') as f:
    f.write(rev)

...

 # Restore poms
  with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.xml", "w") as f:
    f.write(pomxml)

  with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.properties", "w") as f:
    f.write(pomprops)

ThisBuild / version := "0.0.0-SNAPSHOT"
// edit this and change is reflected to the all modules.

lazy val shared = project
  .in(file("shared"))
  .settings(
    name := "jwm-shared",
  )
lazy val shared = project
  .in(file("windows"))
  .settings(
    name := "jwm-windows-x64",
  )
lazy val shared = project
  .in(file("linux"))
  .settings(
    name := "jwm-linux-x64",
  )
lazy val shared = project
  .in(file("macos"))
  .settings(
    name := "jwm-macos-x64",
  )

This defines the versions for all projects.

sbt:jwm> version
[info] windows / version
[info]      0.0.0-SNAPSHOT
[info] linux / version
[info]      0.0.0-SNAPSHOT
[info] macos / version
[info]      0.0.0-SNAPSHOT
[info] version
[info]      0.0.0-SNAPSHOT

sbt:jwm> projectID
[info] shared / projectID
[info]      org.jetbrains.jwm:jwm-shared:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver)
[info] windows / projectID
[info]      org.jetbrains.jwm:jwm-windows-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver)                                  0J
[info] linux / projectID
[info]      org.jetbrains.jwm:jwm-linux-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver) 
[info] macos / projectID
[info]      org.jetbrains.jwm:jwm-macos-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver) 

Test
With sbt (or any other build tools), developers can easily add and run tests by just adding code at test src directory.

Maintainability
You can take advantage of type and IDE supports when writing scripts.
I do not want to repeatedly run scripts to check its behavior.
Besides, if someone is faced with build task error, in many cases, static typed languages can give more useful information than dynamic ones.

[i10416]

@tonsky

[i10416]

Note: there are some tweaks to be done especially for macos and linux related things.

[i10416]

Performance here (#155 (comment)) is measured on Windows. On linux, it is faster by 2.0-5.0s.

[i10416]

Performance comparison

I changed Log.cc and Example.java:

My environment

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

ninja --version
1.10.0

clang --version
clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

java --version
openjdk 16.0.1 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-24)
OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

Result

ninja build after edit Log.cc: 2s
java sources, javadoc compile and publishLocal : 1s
run example app after edit Example.java: < 1s

steps:

1. enter sbt shell at jwm project root and examples/metrics(omit sbt start-up time)
2. run packArtifact;publishLocal in jwm project sbt shell
3. run run in example app sbt shell

[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/impl/package-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/impl/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/constant-values.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/overview-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/index.html...
[info] javadoc: Building index for all classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/allclasses-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/allpackages-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/index-all.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/overview-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/help-doc.html...
[info] javadoc: 100 warnings
[info] Main Java API documentation successful.
[info] :: delivering :: org.jetbrains.jwm#jwm-shared;0.1.289 :: 0.1.289 :: release :: Thu Sep 23 04:41:50 JST 2021
[info]  delivering ivy file to /home/i10416/workspace/JWM/shared/target/ivy-0.1.289.xml
[info]  published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/poms/jwm-shared.pom
[info]  published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/jars/jwm-shared.jar
[info]  published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/srcs/jwm-shared-sources.jar
[info]  published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/docs/jwm-shared-javadoc.jar
[info]  published ivy to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/ivys/ivy.xml
[info] Wrote /home/i10416/workspace/JWM/linux/target/jwm-linux-x64-0.1.289.pom
[info] Main Java API documentation to /home/i10416/workspace/JWM/linux/target/api...
[info] compiling 1 Java source to /home/i10416/workspace/JWM/linux/target/classes ...
[info] javadoc: Loading source file WindowX11.java...
[info] javadoc: Constructing Javadoc information...
[info] javadoc: Building index for all the packages and classes...
[info] javadoc: Standard Doclet version 16.0.1+9-24
[info] javadoc: Building tree for all the packages and classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/WindowX11.html...

[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/package-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/overview-tree.html...
[info] javadoc: Building index for all classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/allclasses-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/allpackages-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/index-all.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/help-doc.html...
[info] javadoc: 16 warnings
[info] Main Java API documentation successful.
[info] :: delivering :: org.jetbrains.jwm#jwm-linux-x64;0.1.289 :: 0.1.289 :: release :: Thu Sep 23 04:41:50 JST 2021
[info]  delivering ivy file to /home/i10416/workspace/JWM/linux/target/ivy-0.1.289.xml
[info]  published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/poms/jwm-linux-x64.pom
[info]  published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/jars/jwm-linux-x64.jar
[info]  published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/srcs/jwm-linux-x64-sources.jar
[info]  published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/docs/jwm-linux-x64-
[info]  published ivy to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/ivys/ivy.xml
[success] Total time: 3 s, completed Sep 23, 2021

[tonsky]

I agree that using a build tool is an industry standard. You need a really good reason to NOT go with the off-the-shelf tooling.

I also appreciate you building the prototype and making a strong case for it. Under normal circumstances (e.g. standard Java project), I would wholeheartedly agree.

This project is, unfortunately, not a typical project.

First, it’s a 50/50 mix of Java and C++. Most of the build tools and IDEs are great at one language/ecosystem and offer escape hatches for everything else. E.g. in SBT you get Java compilation for free, but C++ is still a call to external tools and you have to write a script to run it.

Second, we also make use of Lombok, which adds an extra step to Java stages of the pipeline. Integrations exists for all major build systems, of course, and it’s great when they work, but it’s very hard to understand when they don’t work.

Third, project is very simple. There are no transitive dependencies, very few classes. Big build tools are great for huge projects, because you pay the overhead once and then reap the benefits. In our case, the overhead is bigger than the benefits. I don’t want to learn all the intricacies of SBT, Gradle or Maven just to make a simple javac or CMake call.

Fourth, our build is fairly non-standard. We are building jars out of native code, which is not a use-case Java build tools were build for. I have plans to merge all platforms in one jar eventually, which would mean merging multiple files built on different machines. It’s not a hard task, but it’s hard to teach e.g. Maven to do it. Build tools are good because they are rigid, they force you into their model. What we need in this project is flexibility.

Finally, I am very performance focused. I love the ability to change something and see the results immediately. Waiting event for a few extra seconds after every change is noticeable for me.

Now, having all those in mind, let’s go over build systems we’ve tried and see why we arrived at Python. Hopefully it’ll make more sense.

Gradle. Pros: flexible. Cons: java-focused, imperative, have to learn new language, slow.

Maven: Pros: declarative. Cons: java-focused, not flexible at all, hard to extend, slow.

Make: Pros: declarative, fast, language-agnostic (C-focused?). Cons: wasn’t able to make per-file recompilation of java files, relies on bash (see below).

Bash: Pros: flexible, language-agnostic, fast. Cons: imperative, not cross-platform, very hard to code.

Clojure (via Babashka): Pros: language-agnostic, cross-platform, fast, flexible, real programming language. Cons: have to learn new language, not enough batteries, imperative.

Python: Pros: flexible, language-agnostic, cross-platform, very popular language, batteries included (no dependencies needed), real programming language, fast. Cons: imperative.

SBT (I haven’t tried it, so it’s just my guesstimate): Pros: declarative, flexible? Cons: have to learn new language, slow, java-focused.

I am mostly happy with Python. I know it’s a non-obvious choice and I know IDE support is lacking, but I am happy with the pros it gives and the simplicity of the whole setup. I like the ability just do what I need directly (e.g. copy this file, call this program with those arguments) instead of guessing what hints I need to give to a build system to convince it to do what I want it to do.

E.g. with Maven, it was very hard to convince it to include some files but don’t include some other files, to NOT to run tests when I didn’t need it to, or generate javadocs on delomboked source files.

I’m sorry, but I have to give it a hard pass. I appreciate you doing the work, but I hope you can see my side of the things and where this decision comes from.

And again, if you have some specific problems with Python scripts, let me know and we could (hopefully) fix them. Including usability/convenience problems.

[i10416]

I like the ability just do what I need directly (e.g. copy this file, call this program with those arguments) instead of guessing what hints I need to give to a build system to convince it to do what I want it to do.

Thank you for elaborate your reasons. Most of them make sense for me. Thus, I will withdraw PR (or feel free to close by yourself).
It is reasonable to run small native c-oriented tools from Python scripts in this project. I feel comfortable using Scala, but Python seems much suitable 😢

I’m sorry, but I have to give it a hard pass. I appreciate you doing the work, but I hope you can see my side of the things and where this decision comes from.

I don't mind 👍

if you have some specific problems with Python scripts,

This may not be about Python scripts and be nits pick, but I think it would be better to separate debug-purpose app from example apps like below. It feels neat if scripts for some remaining tasks such as packaging app or building native-image are managed under that dir.

• examples
  • debug-purpose-app
    • what is currently in examples/*
  • example1
  • example2
  • ...

P.S. If you felt sbt slow when you used it before, try sbt --client option introduced from v1.4.0.
P.S. You can use Scala, real language, in sbt!

SBT (I haven’t tried it, so it’s just my guesstimate): Pros: declarative, flexible?

Thank you for taking time! jpackage task #104 has been almost done in this PR, thus I will rewrite it in Python and make another PR soon.