forked from clj-python/libpython-clj
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathembedded.html
174 lines (170 loc) · 12.8 KB
/
embedded.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
<!DOCTYPE html PUBLIC ""
"">
<html><head><meta charset="UTF-8" /><title>Embedding Clojure In Python</title><script async="true" src="https://www.googletagmanager.com/gtag/js?id=G-LN7PG6FJ2D"></script><script>window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-LN7PG6FJ2D');</script><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name">libpython-clj</span> <span class="project-version">2.024</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="Usage.html"><div class="inner"><span>LibPython-CLJ Usage</span></div></a></li><li class="depth-1 current"><a href="embedded.html"><div class="inner"><span>Embedding Clojure In Python</span></div></a></li><li class="depth-1 "><a href="environments.html"><div class="inner"><span>Python Environments</span></div></a></li><li class="depth-1 "><a href="new-to-clojure.html"><div class="inner"><span>So Many Parenthesis!</span></div></a></li><li class="depth-1 "><a href="scopes-and-gc.html"><div class="inner"><span>Scopes And Garbage Collection</span></div></a></li><li class="depth-1 "><a href="slicing.html"><div class="inner"><span>Slicing And Slices</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>libpython-clj2</span></div></div></li><li class="depth-2 branch"><a href="libpython-clj2.codegen.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>codegen</span></div></a></li><li class="depth-2 branch"><a href="libpython-clj2.embedded.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>embedded</span></div></a></li><li class="depth-2 branch"><a href="libpython-clj2.java-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>java-api</span></div></a></li><li class="depth-2"><a href="libpython-clj2.python.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>python</span></div></a></li><li class="depth-3 branch"><a href="libpython-clj2.python.class.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>class</span></div></a></li><li class="depth-3"><a href="libpython-clj2.python.np-array.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>np-array</span></div></a></li><li class="depth-2"><a href="libpython-clj2.require.html"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>require</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1>Embedding Clojure In Python</h1>
<p>The initial development push for <code>libpython-clj</code> was simply to embed Python in
Clojure allowing Clojure developers to use Python modules simply transparently.
This approach relied on <code>libpython-clj</code> being able to find the Python shared library
and having some capability to setup various Python system variables before loading
any modules. While this works great most of the time there are several reasons for
which this approach is less than ideal:</p>
<ol>
<li>In some cases a mainly Python team wants to use some Clojure for a small part of
their work. Telling them to host Python from Clojure is for them a potentially
very disruptive change.</li>
<li>The python ecosystem is moving away from the shared library and towards
compiling a statically linked executable. This can never work with
libpython-clj's default pathway.</li>
<li>Embedded Python cannot use all of Python functionality available due to the fact
that the host process isn't <code>python</code>. Specifically the multithreading module
relies on forking the host process and thus produces a hang if the JVM is the
main underlying process.</li>
<li>Replicating the exact python environment is error prone especially when Python
environment managers such as <code>pyenv</code> and <code>Conda</code> are taken into account.</li>
</ol>
<p>Due to the above reasons there is a solid argument for, if possible, embedding
Clojure into Python allowing the Python executable to be the host process.</p>
<h2>Enter: cljbridge</h2>
<p>Python already had a nascent system for embedding Java in Python - the
<a href="https://pypi.org/project/javabridge/">javabridge module</a>.</p>
<p>We went a step further and provide <code>cljbridge</code> python module.</p>
<p>In order to compile <code>javabridge</code>
a JDK is required and <strong>not just the JRE</strong>. <a href="https://github.com/tristanstraub/">tristanstraub</a>
had found a way to use this in order to work with <a href="https://github.com/tristanstraub/blender-clj/">Blender</a>.
We took a bit more time and worked out ways to smooth out these interactions
and make sure they were supported throughout the system.</p>
<h2>From the Python REPL</h2>
<p>The next step involves starting a python repl.</p>
<p>This requires a python library <code>cljbridge</code>,
which can be installed via</p>
<pre><code>export JAVA_HOME=<--YOUR JAVA HOME-->
python3 -m pip install cljbridge
</code></pre>
<p>This will install and eventually compile <code>javabridge</code> as well.</p>
<p>If the installation cannot find 'jni.h' then most likely you have the Java runtime
(JRE) installed as opposed to the Java development kit (JDK).</p>
<p>So we start by importing
that script:</p>
<pre><code class="language-python">Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from clojurebridge import cljbridge
>>> test_var=10
>>> cljbridge.init_jvm(start_repl=True)
Mar 11, 2021 9:08:47 AM clojure.tools.logging$eval3186$fn__3189 invoke
INFO: nREPL server started on port 40241 on host localhost - nrepl://localhost:40241
</code></pre>
<p>At this point we do not get control back; we have released the GIL and java
is blocking this thread to allow the Clojure REPL systems access to the GIL. We have
two important libraries for clojure loaded, nrepl and cider which allow a rich,
interactive development experience so let's now connect to that port with our favorite
Clojure editor - emacs of course ;-).</p>
<h3>Passing JVM arguments</h3>
<p>If you want to specify arbitrary arguments for the JVM to be started by Python,
you can use the environment variable <code>JDK_JAVA_OPTIONS</code> to do so. It will be picked up by
the JVM when starting.</p>
<p>Since clojurebridge 0.0.8, you can as well specify a list of aliases, which get resolved
from the <code>deps.edn</code> file. This allows as well to specify JVM arguments and JVM properties.</p>
<p>Example:</p>
<p>Starting Clojure embedded from python via</p>
<pre><code class="language-python">cljbridge.init_jvm(aliases=["jdk-17","fastcall"],start_repl=True)
</code></pre>
<p>and a <code>deps.edn</code> with</p>
<pre><code class="language-clojure">:aliases {
:fastcall
{:jvm-opts ["-Dlibpython_clj.manual_gil=true"]}
:jdk-17
{:jvm-opts ["--add-modules=jdk.incubator.foreign"
"--enable-native-access=ALL-UNNAMED"]}}
</code></pre>
<p>would add then the appropriate JVM options.</p>
<h2>From the Clojure REPL</h2>
<p>From emacs, I run the command 'cider-connect' which allows me to specify a host
and port to connect to. Once connected, I get a minimal repl environment:</p>
<pre><code class="language-clojure">;; M-x cider-connect ...localhost...40241
;; I am not sure why but to initialize the user namespace I have to eval ns user
user> (eval '(ns user))
nil
user> (require '[libpython-clj2.python :as py])
nil
;; Python has been initialized and libpython-clj can detect this
user> (py/initialize!)
:already-initialized
user> ;;We can share data via the main module
user> (def main-mod (py/add-module "__main__"))
#'user/main-mod
user> (def mod-dict (py/module-dict main-mod))
#'user/mod-dict
user> (keys mod-dict)
("__name__"
"__doc__"
"__package__"
"__loader__"
"__spec__"
"__annotations__"
"__builtins__"
"cljbridge"
"test_var")
user> (get mod-dict "test_var")
10
user> (.put mod-dict "clj_fn" (fn [& args] (println "Printing from Clojure: " (vec args))))
nil
user> ;;Now if we stop the repl server we can access our python environment again
user> (require '[libpython-clj2.embedded :as embedded])
nil
user> (embedded/stop-repl!)
</code></pre>
<h2>And Back to Python!!</h2>
<p>Shutting down the repl always gives us an exception; something perhaps to work on.
But the important thing is that we can access variables and data that we set
in the main module -</p>
<pre><code class="language-python">>>> Exception in thread "nREPL-session-d684061e-f21c-4265-a9a2-828b99dcaf42" java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at nrepl.transport$bencode$fn__7714.invoke(transport.clj:121)
at nrepl.transport.FnTransport.send(transport.clj:28)
at nrepl.middleware.print$send_streamed.invokeStatic(print.clj:136)
at nrepl.middleware.print$send_streamed.invoke(print.clj:122)
at nrepl.middleware.print$printing_transport$reify__8149.send(print.clj:173)
at cider.nrepl.middleware.track_state$make_transport$reify__17923.send(track_state.clj:228)
at nrepl.middleware.caught$caught_transport$reify__8184.send(caught.clj:58)
at nrepl.middleware.interruptible_eval$evaluate$fn__8250.invoke(interruptible_eval.clj:132)
at clojure.main$repl$fn__9121.invoke(main.clj:460)
at clojure.main$repl.invokeStatic(main.clj:458)
at clojure.main$repl.doInvoke(main.clj:368)
at clojure.lang.RestFn.invoke(RestFn.java:1523)
at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:84)
at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:56)
at nrepl.middleware.interruptible_eval$interruptible_eval$fn__8258$fn__8262.invoke(interruptible_eval.clj:152)
at clojure.lang.AFn.run(AFn.java:22)
at nrepl.middleware.session$session_exec$main_loop__8326$fn__8330.invoke(session.clj:202)
at nrepl.middleware.session$session_exec$main_loop__8326.invoke(session.clj:201)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:748)
>>> # So let's call our new clojure fn
>>> clj_fn(1, 2, 3, 4, "Embedded Clojure FTW!!")
Printing from Clojure: [1 2 3 4 Embedded Clojure FTW!!]
>>>
</code></pre>
<h2>Loading and running a Clojure file in embedded mode</h2>
<p>We can runs as well a .clj file in embedded mode.
The following does this without an interactive pytho shell, it just runs the provided clj file with
<code>clojure.core/load-file</code></p>
<pre><code class="language-bash">python3 -c 'import cljbridge;cljbridge.load_clojure_file(clj_file="my-file.clj")'
</code></pre>
<h2>Are You Not Entertained???</h2>
<p>So there you have it, embedding a Clojure repl in a Python process and passing data
in between these two systems. This sidesteps a <em>ton</em> of issues with embedding Python
and provides another interesting set of possibilities, essentially extending existing
Python systems with some of the greatest tech the JVM has to offer :-).</p>
<ul>
<li><a href="https://github.com/LeeKamentsky/python-javabridge">javabridge github</a></li>
<li><a href="https://github.com/clj-python/libpython-clj/blob/94c72ca0ac94b210a9b126805cd4112024ad0b96/cljbridge.py">libpython-clj cljbridge.py</a></li>
<li><a href="https://clj-python.github.io/libpython-clj/libpython-clj2.embedded.html">libpython-clj embedded docs</a></li>
<li><a href="https://github.com/tristanstraub/blender-clj/">blender-clj - the inspiration</a></li>
</ul>
</div></div></div></body></html>