Merged changes

README.md

@@ -5,14 +5,11 @@ using the [lparallel] (http://lparallel.org) API.
 
 ### Download
 
-Assuming that you have [Quicklisp](http://www.quicklisp.org/beta/) installed,
+The easiest way to obtain lfarm is through
+[Quicklisp](http://www.quicklisp.org/beta/). Alternatively, one may
+[clone the repository](https://github.com/lmj/lfarm.git).
 
-    $ cd ~/quicklisp/local-projects
-    $ git clone git://github.com/lmj/lfarm.git
-
-lfarm is known to run on Allegro, Clozure, LispWorks, and SBCL.
-
-### Kernel
+### Synopsis
 
 In lparallel a _kernel_ was defined as abstract entity that schedules
 and executes tasks. lparallel implements it with a thread pool, while
@@ -29,11 +26,27 @@ in lfarm it is implemented with a set of servers that execute tasks.
                                               ("127.0.0.1" 22222))))
 
     ;; Use the lparallel API.
-    (let ((channel (lfarm:make-channel)))
-      (lfarm:submit-task channel #'+ 3 4)
-      (lfarm:receive-result channel))
+    (defpackage :example (:use :cl :lfarm))
+    (in-package :example)
+
+    (let ((channel (make-channel)))
+      (submit-task channel #'+ 3 4)
+      (receive-result channel))
+    ;; => 7
+
+    (let ((f (future (+ 3 4))))
+      (force f))
     ;; => 7
 
+    (plet ((x (+ 3 4))
+           (y (+ 5 6)))
+      (+ x y))
+    ;; => 18
+
+    (pmapcar '1+ #(1 2 3))         ; => (2 3 4)
+    (preduce '+ #(1 2 3))          ; => 6
+    (pmap-reduce '1+ '+ #(1 2 3))  ; => 9
+
 Although the servers in this example are local, lfarm servers may run
 in separate Lisp instances on remote machines.
 
@@ -50,9 +63,6 @@ task must be
 recorded. (A Lisp implementation may record a function definition, but
 is not required to do so.)
 
-    (defpackage :example (:use :cl :lfarm))
-    (in-package :example)
-
     (deftask add (x y)
       (+ x y))
 
@@ -64,9 +74,6 @@ is not required to do so.)
 `submit-task` notices that `add` was defined with `deftask` and
 converts it to a named lambda before submitting it to a server.
 
-`deftask*` is a variant of `deftask` which records the function body
-without defining the function.
-
 To define `add` remotely use `broadcast-task`, which executes a given
 task on all servers.
 
@@ -91,12 +98,12 @@ the nickname `lfarm`. It exports the [lparallel kernel
 API](http://lparallel.org/api/kernel) with the following differences.
 
 * tasks have the aforementioned restrictions placed upon them
-* the addition of `deftask` and its non-locally-defining cousin `deftask*`
+* the addition of `deftask`
 * `make-kernel` expects addresses, and lacks the `:context` and
   `:bindings` arguments
 * `task-handler-bind` does not exist
 * `*debug-tasks-p*` and `*kernel-spin-count*` exist but have no effect
-* `submit-task` is a macro that wraps `submit-task*` (explained below)
+* `submit-task` is a macro that wraps `submit-task*` (see the Details section)
 * the addition of `broadcast-task` which similarly wraps `broadcast-task*`
 * `task-execution-error` is signaled when a task fails on a remote
 server, instead of the actual error (which may not have local meaning)
@@ -122,14 +129,14 @@ seconds. Returns true if successful and nil otherwise.
 This only stops new connections from being made. Connections in
 progress are unaffected.
 
-## Security
+### Security
 
 The purpose of an lfarm server is to execute arbitrary code, so it is
 highly advised to enable some form of security. lfarm directly
 supports Kerberos (or Active Directory) authentication. Alternatively,
 SSH tunnels may be used.
 
-### Security with SSH tunneling
+#### Security with SSH tunneling
 
     ;; On the remote machine
     (ql:quickload :lfarm-server)
@@ -150,7 +157,7 @@ Of course there is still local security to consider, as local users on
 both ends have access to the server. If this is a concern then a
 packet filtering tool such as iptables may be used.
 
-### Security with Kerberos/GSSAPI
+#### Security with Kerberos/GSSAPI
 
 The `lfarm-gss` system provides support for GSSAPI authentication. The
 `:auth` argument to `lfarm-server:start-server` and

lfarm-client/closure.lisp

@@ -158,9 +158,7 @@
                  :collect ``(,',var ,',(macroexpand-1 var env))))
         (let (,,@(loop
                     :for var :in lexicals
-                    :collect ``(,',var
-                                (deserialize-buffer
-                                 ,(serialize-to-buffer ,var)))))
+                    :collect ``(,',var ',,var)))
           (,',lambda-type ,@',(unsplice name) ,',lambda-list
             ,@',body)))))
 

lfarm-client/promise.lisp

@@ -71,8 +71,11 @@
 (defun force-future (future)
   (with-slots (result channel) future
     (with-unfulfilled (future)
-      (setf result (receive-result channel)))
-    (values-list result)))
+      (setf result (handler-case (receive-result channel)
+                     (task-execution-error (err) err))))
+    (etypecase result
+      (list (values-list result))
+      (task-execution-error (error result)))))
 
 (defun force (promise)
   (typecase promise

lfarm-common.asd

@@ -30,7 +30,8 @@
 
 (defsystem :lfarm-common
   :description
-  "Common components of lfarm, a library for distributing work across machines."
+  "(private) Common components of lfarm, a library for distributing
+work across machines."
   :long-description "See http://github.com/lmj/lfarm"
   :version "0.1.0"
   :licence "BSD"

lfarm-server.lisp

@@ -79,47 +79,63 @@ closure in which those variables are bound to the captured values."
 
 ;;;; package generator
 
-;;; Allegro doesn't signal PACKAGE-ERROR for an undefined package.
-;;; Work around by parsing the error description string.
-#+allegro
+;;; Allegro and ABCL signal `reader-error' for a missing package
+;;; during `read'. We must parse the report string in order to get the
+;;; package name.
+#+(and (or abcl allegro) lfarm.with-text-serializer)
 (progn
-  (defun match-delimited-seq (delim seq left-match right-match)
-    ;; (match-delimited-seq #\! "hello !want this! world" "hello " " world")
+  (defparameter *match-around* #+abcl '("The package \"" "\" can't be found.")
+                               #+allegro '("Package \"" "\" not found"))
+
+  (defun match-around (seq left right)
+    ;; (match-around "hello !want this! world" "hello !" "! world")
     ;; => "want this"
-    (when-let* ((left-match-pos (search left-match seq))
-                (left-delim (let ((pos (+ left-match-pos (length left-match))))
-                              (and (eql (elt seq pos) delim) pos)))
-                (right-delim (position delim seq :start (1+ left-delim)))
-                (right-match-pos (1+ right-delim))
-                (end-pos (+ right-match-pos (length right-match))))
-      (unless (mismatch right-match seq
-                        :start2 right-match-pos
-                        :end2 (min end-pos (length seq)))
-        (subseq seq (1+ left-delim) right-delim))))
+    (when-let* ((left-pos (search left seq))
+                (match-pos (+ left-pos (length left)))
+                (right-pos (search right seq :start2 match-pos)))
+      (subseq seq match-pos right-pos)))
 
   (defun extract-package-name (err)
-    (let ((desc (princ-to-string err)))
-      (or (match-delimited-seq #\" desc "Package " " not found")
-          (match-delimited-seq #\" desc "" " is not a package")))))
-
-(defun make-package* (pkg)
-  (info "creating package" pkg)
-  (make-package pkg :use nil))
+    (apply #'match-around (princ-to-string err) *match-around*))
 
-(defwith with-package-generator ()
-  (with-tag :retry
-    (handler-bind ((package-error
-                    (lambda (err)
-                      (make-package* (package-error-package err))
-                      (go :retry)))
-                   #+allegro
-                   ((or reader-error type-error)
+  (defwith with-missing-package-handler (action)
+    (handler-bind ((reader-error
                     (lambda (err)
                       (when-let (name (extract-package-name err))
-                        (make-package* name)
-                        (go :retry)))))
+                        (funcall action name)))))
       (call-body))))
 
+;;; Allegro signals `type-error' for a missing package during
+;;; `cl-store:restore'. According to Franz, if package `foo' does not
+;;; exist then `:foo' is not a package designator, which is why
+;;; (intern "BAR" :foo) signals a `type-error'. `cl-store:restore'
+;;; calls `intern' when restoring a symbol.
+#+(and allegro (not lfarm.with-text-serializer))
+(defwith with-missing-package-handler (action)
+  (handler-bind ((type-error
+                  (lambda (err)
+                    (when (eq 'package (type-error-expected-type err))
+                      (funcall action (type-error-datum err))))))
+    (call-body)))
+
+;;; In all other cases `package-error' is signaled for a missing package.
+#-(or (and (or abcl allegro) lfarm.with-text-serializer)
+      (and allegro (not lfarm.with-text-serializer)))
+(defwith with-missing-package-handler (action)
+  (handler-bind ((package-error
+                  (lambda (err)
+                    (funcall action (package-error-package err)))))
+    (call-body)))
+
+(defwith with-package-generator ()
+  (with-tag :retry
+    (flet ((make-package-and-retry (name)
+             (info "creating package" name)
+             (make-package name :use nil)
+             (go :retry)))
+      (with-missing-package-handler (#'make-package-and-retry)
+        (call-body)))))
+
 ;;;; task category tracking
 
 ;;; Vector of task category ids currently running.

lfarm-test.asd

@@ -47,6 +47,7 @@
   :components ((:module "lfarm-test"
                 :serial t
                 :components ((:file "1am")
+                             (:file "base")
                              (:file "kernel-test")
 #+lfarm.with-closures        (:file "closure-test")
                              (:file "promise-test")