Disable parallel build by default

Disable the parallel build unless explicitly enabled, as it confuses some
newbies to no end as the compilation semi-deterministically dies out of
lack of memory as gerbil tries to spawn too many gcc's. In the future,
somehow find a good estimate of how many processes to spawn, or better,
learn to dynamically measure and calibrate.

doc/guide/getting-started.md

@@ -104,6 +104,8 @@ the parent directory of the directory in which `gxi` is installed.
 Note that in this final case, the autodetection relies on `gxi` not being a symlink,
 but on your `PATH` pointing to its physical directory or an absolute path being used;
 then again you can define `GERBIL_HOME` or configure with `--prefix` so autodetection isn't needed.
+(Note that incompatibility with a previously defined `GERBIL_HOME` is
+a frequent source of trouble in newbies who make multiple attempts at building from source).
 
 Similarly, Gerbil will use the path specified by the `GERBIL_GSC` environment variable
 to find the Gambit Scheme compiler. If the path is not specified,
@@ -112,6 +114,11 @@ If not specified, then it will assume that the command `gsc` in your `PATH`
 is the version of Gambit you use. It is then important that `gsc` and `gcc` in your `PATH`
 are the same as were used to compile Gerbil and Gambit respectively.
 
+Finally, Gerbil consults the `GERBIL_BUILD_CORES` environment variable
+to determine whether to build its code in parallel, e.g. `export GERBIL_BUILD_CORES=4`.
+This is disabled by default. See details and explanations in
+[the documentation for :std/make](../reference/make.md).
+
 You can see what the default features are and aren't by using `./configure --help`:
 it will offer you options that modify the defaults
 by enabling features that aren't enabled by default,

doc/reference/make.md

@@ -192,11 +192,23 @@ the following keyword arguments, that may configure how your project is built.
     It will still spawn processes for Gambit compilation as such.
     If it's `1`, then Gerbil will spawn a subprocess for the Gerbil compilation steps
     as well as for the Gambit compilation step, but only one process at a time.
-    If it's `#f`, then no parallelism is used, same as `0`.
-    If it's `#t` (the default), then the maximum number of CPUs is used,
-    as detected from the Operating System, or overridden by the value of the environment variable
-    `GERBIL_BUILD_CORES`, if defined and an integer.
-
+    If it's `#t` (the default), then the environment variable `GERBIL_BUILD_CORES` is consulted,
+    and used as above if it's an integer (represented as a decimal string).
+    Otherwise (including if it's `#f`), then no parallelism is used, same as `0`.
+    Note how the default behavior is to compile in parallel according to a specified number of cores,
+    but that the default behavior's default number of cores is `0`, which disables parallelism.
+    An older release of Gerbil (v0.16) did use `(##cpu-count)` by default, but this was disabled
+    and the current behavior is to default to `0` because parallelism has
+    known failure modes with rough edges when you try to compile on a machine with a lot of cores
+    but not a lot of memory, as seems to be frequent enough these days:
+    Gerbil will spawn off a lot of GCC processes via Gambit, and these may try to allocate
+    more memory than is available, and die in bad ways that Gambit and Gerbil don't handle
+    in a very user-friendly way). If you have enough memory
+    (e.g. I am fine with 16GB of memory on 4 cores), you could use:
+    `export GERBIL_BUILD_CORES="$(gsi -e '(display (##cpu-count)) (newline)')"`
+    In the future, we may try to automatically estimate how many cores to use,
+    and/or implement more robust handling of this failure mode.
+    But for now, we disable parallelism. by default.
 
 ## Interface
 

src/std/make.ss

@@ -101,22 +101,31 @@ TODO:
       optimize: (optimize #t) debug: (debug 'env)
       static: (static #t) static-debug: (static-debug #f)
       verbose: (verbose #f) build-deps: (build-deps_ #f)
-      parallelize: (parallelize_ #t))
+      parallelize: (parallelize_ #f))
     (def gerbil-path (getenv "GERBIL_PATH" "~/.gerbil"))
     (def srcdir (or srcdir_ (error "srcdir must be specified")))
     (def libdir (or libdir_ (path-expand "lib" gerbil-path)))
     (def bindir (or bindir_ (path-expand "bin" gerbil-path)))
     (def prefix (or prefix_ (read-package-prefix srcdir)))
     (def libdir-prefix (if prefix (path-expand prefix libdir) libdir))
     (def build-deps (path-expand (or build-deps_ "build-deps") srcdir))
-    (def parallelize (if (eq? parallelize_ #t) (gerbil-build-cores) (or parallelize_ 0)))
+    (def parallelize (gerbil-build-cores parallelize))
     (struct-instance-init!
       self
       srcdir libdir bindir prefix force? optimize debug static static-debug verbose build-deps
       libdir-prefix parallelize)))
 
-(def (gerbil-build-cores)
-  (with-catch (lambda (_) (##cpu-count)) (lambda () (string->number (getenv "GERBIL_BUILD_CORES")))))
+(def (gerbil-build-cores (cpu-count-spec #t))
+  ;; TODO: for the default (catch) case, use something like
+  ;; (min (##cpu-count) (/ (available-memory) (memory-per-compilation-cpu)))
+  ;; Except we need to compute (available-memory) and calibrate (memory-per-compilation-cpu)...
+  ;; Until then, we disable the parallel build as it confuses a lot of beginners.
+  (cond
+   ((real? cpu-count-spec) cpu-count-spec)
+   ((eq? cpu-count-spec #t)
+    (with-catch (lambda (_) 0) ;; if not defined and an integer, default to 0
+                (lambda () (string->number (getenv "GERBIL_BUILD_CORES")))))
+   (else 0)))
 
 (def (settings-verbose>=? settings level)
   (def verbose (settings-verbose settings))