Debugging section of the hyperspec (mighty-gerbils#978)

This fleshes out the debugging section in the development section of the
hyperspec and also makes sure `gerbil build --debug` works as
advertised.

README.md

@@ -68,6 +68,8 @@ that should get you started:
   on Gerbil programming.
 - The [Gerbil Reference Documentation](https://cons.io/reference/) is the reference documentation
   for the Gerbil runtime and standard library.
+- See also the [Developing Software with Gerbil](https://cons.io/reference/dev) section in the
+  hyperspec for general software development practices with Gerbil.
 
 The documentation is automatically rendered online at [cons.io](https://cons.io).
 You can render it locally by running `doc/build.sh`, which will leave

doc/reference/dev/bach.md

@@ -257,9 +257,9 @@ Commands:
  repl                             provides a repl for a running server
  ping                             pings a server or actor in the server
  lookup                           looks up a server by id or role
+ list                             list server state
  shutdown                         shuts down an actor, server, or the entire ensemble including the registry
  admin                            ensemble administrative operations
- list                             list server state
  ca                               ensemble CA operations
  package                          package ensemble state to ship an actor server environment
  help                             display help; help <command> for command help

doc/reference/dev/debug.md

@@ -1,5 +1,133 @@
 # Debugging
 
-(TODO)
+Complex programs usually start as scribbling in a buffer, evaluated in
+the REPL. The scribbling then becomes a module, which later becomes a
+library, and eventually the program takes its more or less final
+shape. In the end, it is compiled as a fully static binary and shipped
+to a production server.
 
-See [debugging libraries](/reference/std/debug.md) for now.
+One thing is certain: in all stages of a program's evolution there
+will be bugs.  Gerbil offers various mechanisms for debugging
+programs, from inception to production.
+
+See also [Testing](test.md).
+
+## Debugging in the REPL
+
+For interactive debugging, you can just load the relevant dynamic
+modules in the `gxi` REPL and utilize its facilities. If the module is
+loaded from source, then you will also get rudimentary stepping
+support.
+
+Here are the debugging commands avaiable in the `gxi` REPL:
+```
+$ gxi
+Gerbil 0.17.0-336-g94ff9728 on Gambit v4.9.5-40-g24201248
+> ,help
+Gambit v4.9.5-40-g24201248
+,?   | ,help    : Summary of comma commands
+,h   | ,(h X)   : Help on procedure of last error or procedure/macro named X
+,q              : Terminate the process
+,qt             : Terminate the current thread
+,t              : Jump to toplevel REPL
+,d              : Jump to enclosing REPL
+,c   | ,(c X)   : Continue the computation with stepping off
+,s   | ,(s X)   : Continue the computation with stepping on (step)
+,l   | ,(l X)   : Continue the computation with stepping on (leap)
+,N              : Move to specific continuation frame (N>=0)
+,N+  | ,N-      : Move forward/backward by N continuation frames (N>=0)
+,+   | ,-       : Like ,1+ and ,1-
+,++  | ,--      : Like ,N+ and ,N- with N = nb. of frames at head of backtrace
+,y              : Display one-line summary of current frame
+,i              : Display procedure attached to current frame
+,b   | ,(b X)   : Display backtrace of current continuation or X (cont/thread)
+,be  | ,(be X)  : Like ,b and ,(b X) but also display environment
+,bed | ,(bed X) : Like ,be and ,(be X) but also display dynamic environment
+,e   | ,(e X)   : Display environment of current frame or X (proc/cont/thread)
+,ed  | ,(ed X)  : Like ,e and ,(e X) but also display dynamic environment
+,st  | ,(st X)  : Display current thread group, or X (thread/thread group)
+,(v X)          : Start a REPL visiting X (proc/cont/thread)
+```
+
+See also the [Interactive Gerbil Shell](/guide/shell.md) page for some
+utilities useful for debugging in the REPL.
+
+## Debugging with Exception Stack Traces
+
+Gerbil goes to a lot of effort to ensure all exceptions have full
+stack traces and a proper error context pointing to the location of
+the source code where the fault condition occured.
+
+If you have an exception stack trace, you can navigate to the
+offending code in `GERBIL_PREFIX/src` and understand what the problem
+is. Once you understand it, you can dynamically load the module where
+the fault originated in the `gxi` REPL to reproduce and fix the
+issue. Once the issue is fixed, you should probably add a regression
+test as well.
+
+## Debugging Hard Crashes
+
+Occasionally, if you are using a `(not safe)` declaration in some
+performance critical piece of code, there can be segfaults. All the
+standard Gerbil modules included in `libgerbil` have debug symbols
+enabled, so you can just run the program in gdb and get the location
+of the crash.
+
+If the crash is inside `libgerbil`, then you will most like have exact
+location pointing to the _expanded_ source of the crash. You can find
+the offending code in `GERBIL_PREFIX/lib/static`.  Note that we
+_could_ have the tracked scheme source point to the original,
+unexpanded source, but we prefer to work with expanded source as we'd
+like to see the actual code that was compiled and also have the
+ability to debug potential compiler bugs (these are exceedingly rare,
+but they can happen).
+
+If the crash is inside your own library or main code and you have an
+input repro, you can just load the executable module of your program
+dynamically with `gerbil :your-executable-module input-param
+...`.
+So within gdb, you can `gdb gerbil` and `r :your-executable-module input-param`.
+This will get you source location, as all dynamically executable
+modules are compiled with debug symbols.
+
+If the crash is non deterministic, you can build your executable with
+`gerbil build --debug`. This will get you debug symbols for the binary
+and you can just run it directly with gdb until the crash occurs.  The
+actual expanded source files will be in your build's
+`GERBIL_PATH/lib/static`.  So if you built your project with `gerbil
+build --debug`, these will be located in `.gerbil/lib/static`.
+
+::: warning Note
+Sometimes you might get a crash that has no usable stack trace.  This
+can occur if you are calling a non procedure in unsafe context or
+because of some very low level bugs.  If that happens, you will need
+to guess a bit where the crash is coming from and disable your `(not
+safe)` declarations in the relevant module so that you get an
+exception with a stack trace instead.
+
+If you determine that such a crash occurs because of some low level
+Gerbil or Gambit bug, please file an [Issue](https://github.com/mighty-gerbils/gerbil/issues)
+and/or get in touch with the Gerbil development team on
+[gitter](https://gitter.im/gerbil-scheme/community).
+:::
+
+
+## Debugging Live Programs
+
+You can use a remote/network REPL to connect to a running program
+and evaluate code that helps you inspect and modify the state of
+the program.
+
+Gerbil offers a couple of different options for this:
+ - The embedded [Network REPL](/reference/std/net/repl.md) is useful for any
+   running program in your development workstation or laptop.
+   You can just use `telnet` to connect, but you have to
+   manually start the repl server inside your program.
+ - Servers in an [Actor Ensemble](/tutorials/ensemble.md) automatically have
+   network repl capabilities for debugging and loading code.
+   You can interact with them using the `gxensemble` tool.
+
+Once you have a REPL to a running program, you can interact with it by
+evaluating code.  Gerbil also provides some useful
+[debugging libraries](/reference/std/debug.md) you can use to inspect the
+state of the program, debug memory usage and leaks, and so on.

doc/reference/dev/test.md

@@ -21,7 +21,7 @@ in a directory recursively you can append `/...` in the directory.
 
 For instance, the following will recursively run all tests in the current directory:
 ```
-gxtest ./...
+gerbil test ./...
 ```
 
 ## Test suites
@@ -62,7 +62,7 @@ $ cat mylib-test.ss
       (check (myadd 1 2) => 3))))
 
 $ gxc -O mylib.ss
-$ gxtest
+$ gerbil test
 === ./mylib-test.ss
 >>> setup
 setting up...

doc/reference/std/net/repl.md

@@ -3,7 +3,9 @@
 Network repl for debugging live programs.
 
 ::: tip usage
+```scheme
 (import :std/net/repl)
+```
 :::
 
 ## start-repl-server!
@@ -46,10 +48,11 @@ Untaints a thread-group, restoring its specific state.
 ## Loading the Expander
 
 By default, once connected, the REPL does not load the Gerbil expander but
-uses the primitive Gambit eval. This allows the REPL to be embedded to
-static binaries without the requirement to embed the expander's environment.
+uses the primitive Gambit eval. This allows the REPL to be embedded in
+binaries without the requirement to embed the expander's environment.
 
-You can load the expander on demand in executables with:
+If you are debugging during development and thus have the expander environment,
+you can load the expander on demand in running executables with:
 ```
 REPL> (gerbil-load-expander!)
 ```

src/gerbil/compiler/driver.ss

@@ -108,14 +108,16 @@ namespace: gxc
         (invoke-gsc? (pgetq invoke-gsc: opts))
         (gsc-options (pgetq gsc-options: opts))
         (keep-scm?   (pgetq keep-scm: opts))
-        (verbosity   (pgetq verbose: opts)))
+        (verbosity   (pgetq verbose: opts))
+        (debug       (pgetq debug: opts)))
     (when outdir
       (with-driver-mutex (create-directory* outdir)))
     (parameterize ((current-compile-output-dir outdir)
                    (current-compile-invoke-gsc invoke-gsc?)
                    (current-compile-gsc-options gsc-options)
                    (current-compile-keep-scm keep-scm?)
                    (current-compile-verbose verbosity)
+                   (current-compile-debug debug)
                    (current-compile-timestamp (compile-timestamp))
                    (current-expander-compiling? #t))
       (verbose "compile exe " srcpath)
@@ -189,7 +191,7 @@ namespace: gxc
            (output_          (string-append (path-strip-extension output-scm) "_"))
            (output_-c        (string-append output_ ".c"))
            (output_-o        (string-append output_ ".o"))
-           (gsc-link-opts    (gsc-link-options #f))
+           (gsc-link-opts    (gsc-link-options))
            (gsc-cc-opts      (get-gsc-cc-opts gerbil-staticdir))
            (output-ld-opts   (gcc-ld-options))
            (libgerbil.a      (path-expand "libgerbil.a" gerbil-libdir))
@@ -342,7 +344,7 @@ namespace: gxc
            (output-o (string-append output-base ".o"))
            (output-c_ (string-append output-base "_.c"))
            (output-o_ (string-append output-base "_.o"))
-           (gsc-link-opts (gsc-link-options #f))
+           (gsc-link-opts (gsc-link-options))
            (gsc-cc-opts (static-include (gsc-cc-options) gerbil-libdir))
            (output-ld-opts (gcc-ld-options))
            (gsc-gx-macros
@@ -726,16 +728,7 @@ namespace: gxc
   (unless (current-compile-keep-scm)
     (delete-file path)))
 
-(def (gsc-debug-options (phi? #f))
-  (defrules not-phi ()
-    ((_ opts)
-     (if phi? [] opts)))
-  (cond
-   ((current-compile-debug)
-    (not-phi ["-debug-source" "-track-scheme" "-cc-options" "-g"]))
-   (else [])))
-
-(def (gsc-link-options phi?)
+(def (gsc-link-options (phi? #f))
   (let lp ((rest (current-compile-gsc-options)) (opts []))
     (match rest
       (["-cc-options" _ . rest]
@@ -745,9 +738,11 @@ namespace: gxc
       ([opt . rest]
        (lp rest (cons opt opts)))
       (else
-       [(gsc-debug-options phi?) ... (reverse opts) ...]))))
+       (if (and (not phi?) (current-compile-debug))
+         ["-debug-source" "-track-scheme" (reverse opts) ...]
+         (reverse opts))))))
 
-(def (gsc-cc-options)
+(def (gsc-cc-options (phi? #f))
     (let lp ((rest (current-compile-gsc-options)) (opts []))
       (match rest
         (["-cc-options" opt . rest]
@@ -757,7 +752,9 @@ namespace: gxc
         ([_ . rest]
          (lp rest opts))
         (else
-         (reverse opts)))))
+         (if (and (not phi?) (current-compile-debug))
+           ["-cc-options" "-g" (reverse opts) ...]
+           (reverse opts))))))
 
 (def (gcc-ld-options)
     (let lp ((rest (current-compile-gsc-options)) (opts []))
@@ -789,7 +786,7 @@ namespace: gxc
          (link-path-c (string-append link-path ".c"))
          (link-path-o (string-append link-path ".o"))
          (gsc-link-opts (gsc-link-options phi?))
-         (gsc-cc-opts (gsc-cc-options))
+         (gsc-cc-opts (gsc-cc-options phi?))
          (gcc-ld-opts (gcc-ld-options)))
     (invoke (gerbil-gsc)
             ["-link" "-flat" "-o" link-path-c gsc-link-opts ...  path]

src/std/build-script.ss

@@ -15,6 +15,8 @@
          (lp rest (cons* build-release: #t options)))
         (["--optimized" . rest]
          (lp rest (cons* build-optimized: #t options)))
+        (["--debug" . rest]
+         (lp rest (cons* debug: #t options)))
         (else
          (error "Unexpected " rest)))))
   (def srcdir

src/std/make.ss

@@ -85,7 +85,7 @@ TODO:
 
 ;;; Settings: see details in doc/reference/make.md
 (defstruct settings
-  (srcdir libdir bindir prefix force optimize debug static-debug verbose build-deps
+  (srcdir libdir bindir prefix force optimize debug verbose build-deps
    libdir-prefix parallelize
    full-program-optimization
    build-release
@@ -96,8 +96,8 @@ TODO:
   (lambda (self
       srcdir: (srcdir_ #f) libdir: (libdir_ #f) bindir: (bindir_ #f)
       prefix: (prefix_ #f) force: (force? #f)
-      optimize: (optimize #t) debug: (debug #f)
-      static: (_ignore-static #t) static-debug: (static-debug #f)
+      optimize: (optimize #t) debug: (debug_ #f)
+      static: (_ignore-static #t)
       verbose: (verbose_ #f) build-deps: (build-deps_ #f)
       parallelize: (parallelize_ #f)
       full-program-optimization: (full-program-optimization #f)
@@ -117,9 +117,14 @@ TODO:
        (verbose_)
        ((getenv "GERBIL_BUILD_VERBOSE" #f) => string->number)
        (else #f)))
+    (def debug
+      (cond
+       (debug_)
+       ((getenv "GERBIL_BUILD_DEBUG" #f) #t)
+       (else #f)))
     (struct-instance-init!
       self
-      srcdir libdir bindir prefix force? optimize debug static-debug verbose build-deps
+      srcdir libdir bindir prefix force? optimize debug verbose build-deps
       libdir-prefix parallelize
       full-program-optimization
       build-release
@@ -794,7 +799,7 @@ TODO:
     [invoke-gsc: #t
      output-file: binpath
      verbose: (settings-verbose>=? settings 9)
-     debug: (settings-static-debug settings)
+     debug: (settings-debug settings)
      full-program-optimization: (settings-full-program-optimization settings)
      (when/list gsc-opts [gsc-options: gsc-opts]) ...])
   (gxc-compile mod gsc-opts settings #f)

src/tools/gxensemble.ss

@@ -374,8 +374,8 @@
   (repl             do-repl)
   (ping             do-ping)
   (lookup           do-lookup)
-  (shutdown         do-shutdown)
   (list             do-list)
+  (shutdown         do-shutdown)
   (admin            do-admin)
   (ca               do-ca)
   (package          do-package))

src/tools/gxpkg.ss

@@ -76,6 +76,7 @@
       local-flag
       (flag 'build-release "-R" "--release" help: "build released (static) executables")
       (flag 'build-optimized "-O" "--optimized" help: "build full program optimized executables")
+      (flag 'build-debug "-g" "--debug" help: "build with debug symbols")
       (rest-arguments 'pkg help: "package to build; all for all packages, omit to build in current directory")))
   (def clean-cmd
     (command 'clean help: "clean compilation artefacts from one or more packages"
@@ -159,7 +160,7 @@
       ((new)
        (pkg-new .package .name .link))
       ((build)
-       (build-pkgs .pkg .?build-release .?build-optimized .?local))
+       (build-pkgs .pkg .?build-release .?build-optimized .?build-debug .?local))
       ((clean)
        (clean-pkgs .pkg .?local))
       ((deps)
@@ -245,13 +246,15 @@
     (set-local-env!))
   (for-each (cut pkg-unlink <> force?) pkgs))
 
-(def (build-pkgs pkgs release? optimized? local?)
+(def (build-pkgs pkgs release? optimized? debug? local?)
   (when local?
     (set-local-env!))
   (when release?
     (setenv "GERBIL_BUILD_RELEASE" "t"))
   (when optimized?
     (setenv "GERBIL_BUILD_OPTIMIZED" "t"))
+  (when debug?
+    (setenv "GERBIL_BUILD_DEBUG" "t"))
   (if (null? pkgs)
     ;; do local build
     (begin
@@ -576,6 +579,9 @@
                       options))
            (options (if (getenv "GERBIL_BUILD_OPTIMIZED" #f)
                       (cons "--optimized" options)
+                      options))
+           (options (if (getenv "GERBIL_BUILD_DEBUG" #f)
+                      (cons "--debug" options)
                       options)))
       options))