Introduction

Py4CL is a bridge between Common Lisp and Python, which enables Common Lisp to interact with Python code. It uses streams to communicate with a separate python process, the approach taken by cl4py. This is different to the CFFI approach used by burgled-batteries, but has the same goal.

Installing

Depends on:

• Currently tested with SBCL, CCL and ECL (after 2016-09-06). CLISP doesn’t (yet) have uiop:launch-program.
• ASDF3 version 3.2.0 (Jan 2017) or later, as uiop:launch-program is used to run and communicate with python asyncronously.
• Trivial-garbage, available through Quicklisp.
• Python 2 or 3
• (optional) The NumPy python library for multidimensional arrays

Clone this repository into ~/quicklisp/local-projects/ or other location where it can be found by ASDF:

$ git clone https://github.com/bendudson/py4cl.git

then load into Lisp with

(ql:quickload :py4cl)

Tests

Tests use clunit, and run on Travis using cl-travis. Most development is done under Arch linux with SBCL and Python3. To run the tests yourself:

(asdf:test-system :py4cl)

or

(ql:quickload :py4cl/tests)
(py4cl/tests:run)

Examples

Py4CL allows python modules to be imported as Lisp packages, python functions to be called from lisp, and lisp functions called from python. In the example below, SciPy’s odeint function is used to integrate ODEs defined by a Lisp function. The result is a Lisp array, which is then plotted using the matplotlib plotting library.

(ql:quickload :py4cl)

(py4cl:import-module "numpy" :as "np")
(py4cl:import-module "scipy.integrate" :as "integrate")

;; Integrate some ODEs
(defparameter *data*
  (integrate:odeint 
   (lambda (y time) 
     (vector (aref y 1)       ; dy[0]/dt = y[1]
             (- (aref y 0)))) ; dy[1]/dt = -y[0]
   #(1.0 0.0)   ; Initial state
   (np:linspace 0.0 (* 2 pi) 20)))  ; Vector of times

; (array-dimensions *data*) => (20 2)

;; Make a plot, save and show it in a window
(py4cl:import-module "matplotlib.pyplot" :as "plt")

(plt:plot *data*)
(plt:xlabel "Time")
(plt:savefig "result.pdf")
(plt:show)

More detailed examples of using python packages using py4cl:

• Numpy arrays
• Matplotlib plotting
• Scipy scientific library

Reference

Direct evaluation of python code: python-eval, python-exec

For direct access to the python subprocess, python-eval evaluates an expression, converting the result to a suitable lisp type. Note that there are nicer, more lispy wrappers around this function, described below, but they are mostly built on top of python-eval.

(asdf:load-system "py4cl")

(py4cl:python-eval "[i**2 for i in range(5)]") ; => #(0 1 4 9 16)

(py4cl:python-eval "{'hello':'world', 'answer':42}") ; => #<HASH-TABLE :TEST EQUAL :COUNT 2>

Data is passed between python and lisp as text. The python function lispify converts values to a form which can be read by the lisp reader; the lisp function pythonize outputs strings which can be eval‘d in python. The following type conversions are done:

Lisp type	Python type
NIL	None
integer	int
ratio	fractions.Fraction
real	float
complex	complex float
string	str
hash map	dict
list	tuple
vector	list
array	NumPy array
symbol	Symbol class
function	function

Note that python does not have all the numerical types which lisp has, for example complex integers.

Because python-eval and python-exec evaluate strings as python expressions, strings passed to them are not escaped or converted as other types are. To pass a string to python as an argument, call py4cl::pythonize

(let ((my-str "testing"))
  (py4cl:python-eval "len(" (py4cl::pythonize my-str) ")" ))

Note that this escaping is done automatically by higher-level interfaces like python-call and chain:

(let ((my-str "testing"))
  (py4cl:python-call "len" my-str))

(let ((my-str "testing"))
  (py4cl:chain (len my-str)))

If python objects cannot be converted into a lisp value, then they are stored and a handle is returned to lisp. This handle can be used to manipulate the object, and when it is garbage collected the python object is also deleted (using the trivial-garbage package).

(destructuring-bind (fig ax) (plt:subplots)
  ;; fig is #S(PY4CL::PYTHON-OBJECT :TYPE "<class 'matplotlib.figure.Figure'>" :HANDLE 6)
  (py4cl:python-eval ax ".plot(" #(0 1 0 1) ")")
  (plt:show)) 

The interface to python objects is nicer using chain (see below):

(destructuring-bind (fig ax) (plt:subplots)
  (py4cl:chain ax (plot #(0 1 0 1)))
  (plt:show)) 

The python process can be explicitly started and stopped using python-start and python-stop, but py4cl functions start python automatically if needed by calling python-start-if-not-alive.

Calling python functions: python-call

python-call can be used to pass arguments to any python callable, such as a function in a module:

(py4cl:python-exec "import math")
(py4cl:python-call "math.sqrt" 42)

or a lambda function:

(py4cl:python-call "lambda x: 2*x" 21)

Keywords are translated, with the symbol made lowercase:

(py4cl:python-call "lambda a=0, b=1: a-b" :b 2 :a 1)

Calling python methods: python-method

Python methods on objects can be called by using the python-method function. The first argument is the object (including strings, arrays, tuples); the second argument is either a string or a symbol specifying the method, followed by any arguments:

(py4cl:python-method "hello {0}" 'format "world") ; => "hello world"

(py4cl:python-method '(1 2 3) '__len__) ; => 3

Chaining python methods: chain

In python it is quite common to apply a chain of method calls, data member access, and indexing operations to an object. To make this work smoothly in Lisp, there is the chain macro (Thanks to @kat-co and parenscript for the inspiration). This consists of a target object, followed by a chain of operations to apply. For example

(py4cl:chain "hello {0}" (format "world") (capitalize)) ; => "Hello world"

which is converted to python

return "hello {0}".format("world").capitalize()

The only things which are treated specially by this macro are lists and symbols at the top level. The first element of lists are treated as python method names, top-level symbols are treated as data members. Everything else is evaluated as lisp before being converted to a python value.

If the first argument is a list, then it is assumed to be a python function to be called; otherwise it is evaluated before converting to a python value. For example

(py4cl:chain (slice 3) stop)

is converted to the python:

return slice(3).stop

Symbols as first argument, or arguments to python methods, are evaluated, so the following works:

(let ((format-str "hello {0}")
      (argument "world"))
 (py4cl:chain format-str (format argument))) ; => "hello world"

Arguments to methods are lisp, since only the top level forms in chain are treated specially:

(py4cl:chain "result: {0}" (format (+ 1 2))) ; => "result: 3"

Indexing with [] brackets is commonly used in python, which calls the __getitem__ method. This method can be called like any other method

(py4cl:chain "hello" (__getitem__ 4)) ; => "o"

but since this is a common method an alias [] is supported:

(py4cl:chain "hello" ([] 4)) ; => "o"

which is converted to the python

return "hello"[4]

For simple cases where the index is a value like a number or string (not a symbol or a list), the brackets can be omitted:

(py4cl:chain "hello" 4) ; => "o"

Slicing can be done by calling the python slice function:

(py4cl:chain "hello" ([] (py4cl:python-call "slice" 2 4)))  ; => "ll"

which could be imported as a lisp function (see below):

(py4cl:import-function "slice")
(py4cl:chain "hello" ([] (slice 2 4))) ; => "ll"

This of course also works with multidimensional arrays:

(py4cl:chain #2A((1 2 3) (4 5 6))  ([] 1 (slice 0 2)))  ;=> #(4 5)

Sometimes the python functions or methods may contain upper case characters; class names often start with a capital letter. All symbols are converted to lower case, but the case can be controlled by passing a string rather than a symbol as the first element:

;; Define a class
(py4cl:python-exec
   "class TestClass:
      def doThing(self, value = 42):
        return value")

;; Create an object and call the method
(py4cl:chain ("TestClass") ("doThing" :value 31))  ; => 31

Note that the keyword is converted, converting to lower case.

Printing from python

Since standard output is used for communication between lisp and python, this is redirected (to a StringIO buffer) while user python code is running. The output from python functions is then sent to lisp, to be printed to *standard-output*. This means that anything printed by the python process may only appear in chunks, as it is sent to lisp. The following does however work as expected:

(py4cl:chain (print "hello world")) 
; => prints "hello world", returns NIL

In python print_function is imported from __future__, so should be available as a function in python 2.6+, as well as in version 3+.

Asynchronous python functions: python-call-async

One of the advantages of using streams to communicate with a separate python process, is that the python and lisp processes can run at the same time. python-call-async calls python but returns a closure immediately. The python process continues running, and the result can be retrieved by calling the returned closure.

(defparameter thunk (py4cl:python-call-async "lambda x: 2*x" 21))

(funcall thunk)  ; => 42

If the function call requires callbacks to lisp, then these will only be serviced when a py4cl function is called. In that case the python function may not be able to finish until the thunk is called. This should not result in deadlocks, because all py4cl functions can service callbacks while waiting for a result.

Importing functions: import-function

Python functions can be made available in Lisp by using import-function. By default this makes a function which can take any number of arguments, and then translates these into a call to the python function.

(asdf:load-system "py4cl")

(py4cl:python-exec "import math")
(py4cl:import-function "math.sqrt")
(math.sqrt 42) ; => 6.4807405

If a different symbol is needed in Lisp then the :as keyword can be used with either a string or symbol:

(py4cl:import-function "sum" :as "pysum")
(pysum '(1 2 3))  ; => 6

This is implemented as a macro which defines a function which in turn calls python-call.

Importing modules: import-module

Python modules can be imported as lisp packages using import-module. For example, to import the matplotlib plotting library, and make its functions available in the package PLT from within Lisp:

(asdf:load-system "py4cl")
(py4cl:import-module "matplotlib.pyplot" :as "plt") ; Creates PLT package

This will also import it into the python process as the module plt, so that python-call or python-eval can also make use of the plt module.

Like python-exec, python-call and other similar functions, import-module starts python if it is not already running, so that the available functions can be discovered.

The python docstrings are made available as Lisp function docstrings, so we can see them using describe:

(describe 'plt:plot)

Functions in the PLT package can be used to make simple plots:

(plt:plot #(1 2 3 2 1) :color "r")
(plt:show)

Notes:

• import-module should be used as a top-level form, to ensure that the package is defined before it is used.

• If using import-module within org-mode babel then the import should be done in a separate code block to the first use of the imported package, or a condition will be raised like “Package NP does not exist.”

Exporting a function to python: export-function

Lisp functions can be passed as arguments to python-call or imported functions:

(py4cl:python-exec "from scipy.integrate import romberg")

(py4cl:python-call "romberg" 
                   (lambda (x) (/ (exp (- (* x x)))
                                  (sqrt pi)))
                   0.0 1.0) ; Range of integration

Lisp functions can be made available to python code using export-function:

(py4cl:python-exec "from scipy.integrate import romberg")

(py4cl:export-function (lambda (x) (/ (exp (- (* x x)))
                                      (sqrt pi))) "gaussian")

(py4cl:python-eval "romberg(gaussian, 0.0, 1.0)") ; => 0.4213504

Manipulating objects remotely: remote-objects

If a sequence of python functions and methods are being used to manipulate data, then data may be passed between python and lisp. This is fine for small amounts of data, but inefficient for large datasets.

The remote-objects and remote-objects* macros provide unwind-protect environments in which all python functions return handles rather than values to lisp. This enables python functions to be combined without transferring much data.

The difference between these macros is remote-objects returns a handle, but remote-objects* evaluates the result, and so will return a value if possible.

(py4cl:remote-objects (py4cl:python-eval "1+2")) ; => #S(PY4CL::PYTHON-OBJECT :TYPE "<class 'int'>" :HANDLE 0)

(py4cl:remote-objects* (py4cl:python-eval "1+2")) ; => 3

The advantage comes when dealing with large arrays or other datasets:

(time (np:sum (np:arange 1000000)))
; => 3.672 seconds of real time
;    390,958,896 bytes consed

(time (py4cl:remote-objects* (np:sum (np:arange 1000000))))
; => 0.025 seconds of real time
;    32,544 bytes consed

setf-able places

The python-eval function is setf-able, so that python objects can be assigned to by using setf. Since chain uses python-eval, it is also setf-able. This can be used to set elements in an array, entries in a dict/hash-table, or object data members, for example:

(py4cl:import-module "numpy" :as "np")

(py4cl:remote-objects*
  (let ((array (np:zeros '(2 2))))
    (setf (py4cl:chain array ([] 0 1)) 1.0
          (py4cl:chain array ([] 1 0)) -1.0)
    array)) 
; => #2A((0.0 1.0)
;        (-1.0 0.0))

Note that this modifies the value in python, so the above example only works because array is a handle to a python object, rather than an array which is stored in lisp. The following therefore does not work:

(let ((array (np:zeros '(2 2))))
  (setf (py4cl:chain array ([] 0 1)) 1.0
        (py4cl:chain array ([] 1 0)) -1.0)
  array)
; => #2A((0.0 0.0)
;        (0.0 0.0))

The np:zeros function returned an array to lisp; the array was then sent to python and modified in python. The modified array is not returned, since this would mean transferring the whole array. If the value is in lisp then just use the lisp functions:

(let ((array (np:zeros '(2 2))))
  (setf (aref array 0 1) 1.0
        (aref array 1 0) -1.0)
  array)
; => #2A((0.0 1.0)
;        (-1.0 0.0))

Passing lisp objects to python: python-getattr

Lisp structs and class objects can be passed to python, put into data structures and returned:

(py4cl:import-function "dict") ; Makes python dictionaries

(defstruct test-struct 
    x y)

(let ((map (dict :key (make-test-struct :x 1 :y 2))))  ; Make a dictionary, return as hash-map
  ;; Get the struct from the hash-map, and get the Y slot
  (test-struct-y
    (py4cl:chain map "key")))  ; => 2

In python this is handled using an object of class UnknownLispObject, which contains a handle. The lisp object is stored in a hash map *lisp-objects*. When the python object is deleted, a message is sent to remove the object from the hash map.

To enable python to access slots, or call methods on a struct or class, a handler function needs to be registered. This is done by providing a method for generic function python-getattr. This function will be called when a python function attempts to access attributes of an object (__getattr__ method).

;; Define a class with some slots
(defclass test-class ()
  ((value :initarg :value)))

;; Define a method to handle calls from python
(defmethod py4cl:python-getattr ((object test-class) slot-name)
  (cond
    ((string= slot-name "value") ; data member
      (slot-value object 'value))
    ((string= slot-name "func")  ; method, return a function
      (lambda (arg) (* 2 arg)))
    (t (call-next-method)))) ; Otherwise go to next method

(let ((instance (make-instance 'test-class :value 21))) 
  ;; Get the value from the slot, call the method
  ;; python: instance.func(instance.value)
  (py4cl:chain instance (func (py4cl:chain instance value))))  ; => 42

Inheritance then works as usual with CLOS methods:

;; Class inheriting from test-class
(defclass child-class (test-class)
  ((other :initarg :other)))

;; Define method which passes to the next method if slot not recognised
(defmethod py4cl:python-getattr ((object child-class) slot-name)
  (cond
    ((string= slot-name "other")
     (slot-value object 'other))
    (t (call-next-method))))

(let ((object (make-instance 'child-class :value 42 :other 3)))
  (list 
    (py4cl:chain object value) ; Call TEST-CLASS getattr method via CALL-NEXT-METHOD
    (py4cl:chain object other))) ;=> (42 3)