Skip to content

Commit 5847c85

Browse files
tkftkf
committed
Copy README contents to docs/source/*.md
1 parent 7f63b00 commit 5847c85

File tree

7 files changed

+409
-0
lines changed

7 files changed

+409
-0
lines changed

‎docs/source/how_it_works.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
How it works
2+
------------
3+
PyJulia loads the `libjulia` library and executes the statements therein.
4+
To convert the variables, the `PyCall` package is used. Python references
5+
to Julia objects are reference counted by Python, and retained in the
6+
`PyCall.pycall_gc` mapping on the Julia side (the mapping is removed
7+
when reference count drops to zero, so that the Julia object may be freed).

‎docs/source/index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,12 @@ PyJulia is tested against Python versions 2.7, 3.5, 3.6, and 3.7.
2020
:maxdepth: 2
2121
:caption: Contents:
2222
23+
installation
24+
usage
25+
troubleshooting
2326
api
27+
how_it_works
28+
testing
2429
```
2530

2631
Indices and tables

‎docs/source/installation.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
Installation
2+
------------
3+
4+
**Note:** If you are using Python installed with Ubuntu or `conda`,
5+
PyJulia may not work with Julia ≥ 0.7. For workarounds, see
6+
[Troubleshooting](troubleshooting.md). Same caution applies to
7+
any Debian-based and possibly other GNU/Linux distributions.
8+
9+
You will need to install PyCall in your existing Julia installation
10+
11+
```julia
12+
julia> using Pkg # for julia ≥ 0.7
13+
julia> Pkg.add("PyCall")
14+
```
15+
16+
Your python installation must be able to call command line program
17+
`julia`. If your installer does not add the Julia binary directory to
18+
your `PATH`, you will have to add it. _An alias will not work._
19+
20+
Then finally you have to install PyJulia.
21+
22+
**Note:** If you are not familiar with `pip` and have some troubles
23+
with the following installation steps, we recommend going through the
24+
[Tutorials in Python Packaging User Guide](https://packaging.python.org/tutorials/).
25+
26+
To get released versions you can use:
27+
28+
```console
29+
$ python3 -m pip install --user julia
30+
$ python2 -m pip install --user julia # If you need Python 2
31+
```
32+
33+
where `--user` should be omitted if you are using virtual environment
34+
(`virtualenv`, `venv`, `conda`, etc.).
35+
36+
If you are interested in using the development version, you can
37+
install PyJulia directly from GitHub:
38+
39+
```console
40+
$ python3 -m pip install --user 'https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia'
41+
```
42+
43+
You may clone it directly to (say) your home directory.
44+
45+
```console
46+
$ git clone https://github.com/JuliaPy/pyjulia
47+
```
48+
49+
then inside the `pyjulia` directory you need to run the python setup file
50+
51+
```console
52+
$ cd pyjulia
53+
$ python3 -m pip install --user .
54+
$ python3 -m pip install --user -e . # If you want "development install"
55+
```
56+
57+
The `-e` flag makes a development install, meaning that any change to PyJulia
58+
source tree will take effect at next python interpreter restart without having
59+
to reissue an install command.
60+
61+
See [Testing](testing.md) for how to run tests.

‎docs/source/limitations.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
Limitations
2+
------------
3+
4+
### Mismatch in valid set of identifiers
5+
6+
Not all valid Julia identifiers are valid Python identifiers. Unicode
7+
identifiers are invalid in Python 2.7 and so PyJulia cannot call or
8+
access Julia methods/variables with names that are not ASCII only.
9+
Although Python 3 allows Unicode identifiers, they are more
10+
aggressively normalized than Julia. For example, `ϵ` (GREEK LUNATE
11+
EPSILON SYMBOL) and `ε` (GREEK SMALL LETTER EPSILON) are identical in
12+
Python 3 but different in Julia. Additionally, it is a common idiom
13+
in Julia to append a `!` character to methods which mutate their
14+
arguments. These method names are invalid Python identifers.
15+
PyJulia renames these methods by subsituting `!` with `_b`. For
16+
example, the Julia method `sum!` can be called in PyJulia using
17+
`sum_b(...)`.
18+
19+
### Pre-compilation mechanism in Julia 1.0
20+
21+
There was a major overhaul in the module loading system between Julia
22+
0.6 and 1.0. As a result, the "hack" supporting the PyJulia to load
23+
PyCall stopped working. For the implementation detail of the hack,
24+
see: https://github.com/JuliaPy/pyjulia/tree/master/julia/fake-julia
25+
26+
For the update on this problem, see:
27+
https://github.com/JuliaLang/julia/issues/28518
28+
29+
### <kbd>Ctrl-C</kbd> does not work / terminates the whole Python process
30+
31+
Currently, initializing PyJulia (e.g., by `from julia import Main`)
32+
disables `KeyboardInterrupt` handling in the Python process. If you
33+
are using normal `python` interpreter, it means that canceling the
34+
input by <kbd>Ctrl-C</kbd> does not work and repeatedly providing
35+
<kbd>Ctrl-C</kbd> terminates the whole Python process with the error
36+
message `WARNING: Force throwing a SIGINT`. Using IPython 7.0 or
37+
above is recommended to avoid such accidental shutdown.
38+
39+
It also means that there is no safe way to cancel long-running
40+
computations or I/O at the moment. Sending SIGINT with
41+
<kbd>Ctrl-C</kbd> will terminate the whole Python process.
42+
43+
For the update on this problem, see:
44+
https://github.com/JuliaPy/pyjulia/issues/211
45+
46+
### No threading support
47+
48+
PyJulia cannot be used in different threads since libjulia is not
49+
thread safe. However, you can
50+
[use multiple threads within Julia](https://docs.julialang.org/en/v1.0/manual/parallel-computing/#Multi-Threading-(Experimental)-1).
51+
For example, start IPython by `JULIA_NUM_THREADS=4 ipython` and then
52+
run:
53+
54+
```julia
55+
In [1]: %load_ext julia.magic
56+
Initializing Julia interpreter. This may take some time...
57+
58+
In [2]: %%julia
59+
...: a = zeros(10)
60+
...: Threads.@threads for i = 1:10
61+
...: a[i] = Threads.threadid()
62+
...: end
63+
...: a
64+
Out[3]: array([1., 1., 1., 2., 2., 2., 3., 3., 4., 4.])
65+
```
66+
67+
### PyJulia does not release GIL
68+
69+
PyJulia does not release the Global Interpreter Lock (GIL) while
70+
calling Julia functions since PyCall expects the GIL to be acquired
71+
always. It means that Python code and Julia code cannot run in
72+
parallel.

‎docs/source/testing.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
Testing
2+
-------
3+
4+
PyJulia can be tested by simply running [`tox`](http://tox.readthedocs.io):
5+
6+
```console
7+
$ tox
8+
```
9+
10+
The full syntax for invoking `tox` is
11+
12+
```shell
13+
[PYJULIA_TEST_REBUILD=yes] [PYJULIA_TEST_RUNTIME=<julia>] tox [options] [-- pytest options]
14+
```
15+
16+
* `PYJULIA_TEST_REBUILD`: *Be careful using this environment
17+
variable!* When it is set to `yes`, your `PyCall.jl` installation
18+
will be rebuilt using the Python interpreter used for testing. The
19+
test suite tries to build back to the original configuration but the
20+
precompilation would be in the stale state after the test. Note
21+
also that it does not work if you unconditionally set `PYTHON`
22+
environment variable in your Julia startup file.
23+
24+
* `PYJULIA_TEST_RUNTIME`: `julia` executable to be used for testing.
25+
26+
* Positional arguments after `--` are passed to `pytest`.
27+
28+
For example,
29+
30+
```console
31+
$ PYJULIA_TEST_REBUILD=yes PYJULIA_TEST_RUNTIME=~/julia/julia tox -e py37 -- -s
32+
```
33+
34+
means to execute tests with
35+
36+
* PyJulia in shared-cache mode
37+
* `julia` executable at `~/julia/julia`
38+
* Python 3.7
39+
* `pytest`'s capturing mode turned off

‎docs/source/troubleshooting.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
Troubleshooting
2+
---------------
3+
4+
### Your Python interpreter is statically linked to libpython
5+
6+
If you use Python installed with Debian-based Linux distribution such
7+
as Ubuntu or install Python by `conda`, you might have noticed that
8+
PyJulia cannot be initialized properly with Julia ≥ 0.7. This is
9+
because those Python executables are statically linked to libpython.
10+
(See [Limitations](limitations.md) for why that's a problem.)
11+
12+
If you are unsure if your `python` has this problem, you can quickly
13+
check it by:
14+
15+
```console
16+
$ ldd /usr/bin/python
17+
linux-vdso.so.1 (0x00007ffd73f7c000)
18+
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f10ef84e000)
19+
libc.so.6 => /usr/lib/libc.so.6 (0x00007f10ef68a000)
20+
libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f10ef116000)
21+
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f10efaa4000)
22+
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f10ef111000)
23+
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f10ef10c000)
24+
libm.so.6 => /usr/lib/libm.so.6 (0x00007f10eef87000)
25+
```
26+
27+
in Linux where `/usr/bin/python` should be replaced with the path to
28+
your `python` command (use `which python` to find it out). In macOS,
29+
use `otool -L` instead of `ldd`. If it does not print the path to
30+
libpython like `/usr/lib/libpython3.7m.so.1.0` in above example, you
31+
need to use one of the workaround below.
32+
33+
The easiest workaround is to use the `python-jl` command bundled in
34+
PyJulia. This can be used instead of normal `python` command for
35+
basic use-cases such as:
36+
37+
```console
38+
$ python-jl your_script.py
39+
$ python-jl -c 'from julia.Base import banner; banner()'
40+
$ python-jl -m IPython
41+
```
42+
43+
See `python-jl --help` for more information.
44+
45+
Note that `python-jl` works by launching Python interpreter inside
46+
Julia. Importantly, it means that PyJulia has to be installed in the
47+
Python environment with which PyCall is configured. That is to say,
48+
following commands must work for `python-jl` to be usable:
49+
50+
```julia
51+
julia> using PyCall
52+
53+
julia> pyimport("julia")
54+
PyObject <module 'julia' from '/.../julia/__init__.py'>
55+
```
56+
57+
In fact, you can simply use PyJulia inside the Julia REPL, if you are
58+
comfortable with working in it:
59+
60+
```julia
61+
julia> using PyCall
62+
63+
julia> py"""
64+
from julia import Julia
65+
Julia(init_julia=False)
66+
67+
from your_module_using_pyjulia import function
68+
function()
69+
"""
70+
```
71+
72+
Alternatively, you can use [pyenv](https://github.com/pyenv/pyenv) to
73+
build Python with
74+
[`--enable-shared` option](https://github.com/pyenv/pyenv/wiki#how-to-build-cpython-with---enable-shared).
75+
Of course, manually building from Python source distribution with the
76+
same configuration also works.
77+
78+
```console
79+
$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6
80+
Downloading Python-3.6.6.tar.xz...
81+
-> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
82+
Installing Python-3.6.6...
83+
Installed Python-3.6.6 to /home/USER/.pyenv/versions/3.6.6
84+
85+
$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6 | grep libpython
86+
libpython3.6m.so.1.0 => /home/USER/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000)
87+
```
88+
89+
For more discussion, see:
90+
https://github.com/JuliaPy/pyjulia/issues/185
91+
92+
### Segmentation fault in IPython
93+
94+
You may experience segmentation fault when using PyJulia in old
95+
versions of IPython. You can avoid this issue by updating IPython to
96+
7.0 or above. Alternatively, you can use IPython via Jupyter (e.g.,
97+
`jupyter console`) to workaround the problem.
98+
99+
### Error due to `libstdc++` version
100+
101+
When you use PyJulia with another Python extension, you may see an
102+
error like ``version `GLIBCXX_3.4.22' not found`` (Linux) or `The
103+
procedure entry point ... could not be located in the dynamic link
104+
library libstdc++6.dll` (Windows). In this case, you might have
105+
observed that initializing PyJulia first fixes the problem. This is
106+
because Julia (or likely its dependencies like LLVM) requires a recent
107+
version of `libstdc++`.
108+
109+
Possible fixes:
110+
111+
* Initialize PyJulia (e.g., by `from julia import Main`) as early as
112+
possible. Note that just importing PyJulia (`import julia`) does
113+
not work.
114+
* Load `libstdc++.so.6` first by setting environment variable
115+
`LD_PRELOAD` (Linux) to
116+
`/PATH/TO/JULIA/DIR/lib/julia/libstdc++.so.6` where
117+
`/PATH/TO/JULIA/DIR/lib` is the directory which has `libjulia.so`.
118+
macOS and Windows likely to have similar mechanisms (untested).
119+
* Similarly, set environment variable `LD_LIBRARY_PATH` (Linux) to
120+
`/PATH/TO/JULIA/DIR/lib/julia` directory. Using `DYLD_LIBRARY_PATH`
121+
on macOS and `PATH` on Windows may work (untested).
122+
123+
See:
124+
https://github.com/JuliaPy/pyjulia/issues/180,
125+
https://github.com/JuliaPy/pyjulia/issues/223

0 commit comments

Comments
 (0)