Home

Awesome

py4cl2-cffi - a CFFI approach to python interfacing in Common Lisp

Despite being featureful, I won't recommend using py4cl2-cffi in production or in long-term projects yet. Numpy version 2 is unsupported atleast until 2025.

Previous Common Lisp attempts: burgled-batteries3 and cl-python.

New Common Lisp / SBCL attempt with support for calling Lisp from Python: lang.

Non Common Lisp approaches

See this publication for the broad design.

Table of Contents

<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->

Table of Contents

<!-- markdown-toc end -->

Why

py4cl2 has gotten the work done for the past few years. But it has the overhead of (i) stream-based inter-process-communication (ii) eval. That's as worse as one could get.

However, when capable, the CFFI approach can be a 50 times faster than py4cl2.

CL-USER> (py4cl2-cffi:raw-pyexec "def foo(): return str(1)")
0
CL-USER> (time (dotimes (i 10000)
                 (py4cl2-cffi:pycall "foo")))
Evaluation took:
  0.080 seconds of real time
  0.079740 seconds of total run time (0.079740 user, 0.000000 system)
  100.00% CPU
  174,443,654 processor cycles
  3,045,440 bytes consed

NIL
CL-USER> (py4cl2:raw-pyexec "def foo(): return str(1)")
; No value
CL-USER> (time (dotimes (i 10000)
                 (py4cl2:pycall "foo")))
Evaluation took:
  1.051 seconds of real time
  0.482699 seconds of total run time (0.304186 user, 0.178513 system)
  45.96% CPU
  20,000 forms interpreted
  2,327,364,348 processor cycles
  5,760,832 bytes consed

NIL

Limitations

Goals are less ambitious than burgled-batteries. We aim to get "most" libraries working, with a special focus on functional python.

Tested only on Ubuntu 20.04 (CI) and Ubuntu 18.04 (personal machine). Porting to Windows does not look trivial, but someone could prove me wrong (at least provide some pointers!).

Caveats

Unlike py4cl and py4cl2, py4cl2-cffi can only use one python version in a running lisp image. In addition, while the author has been successful in running the py4cl2-cffi-tests without segmentation faults, the project is still in beta stage, so be prepared to run into segmentation faults especially while importing and using python modules.

The project is being tested on

Status

... and much more ...

Tutorial

Installation

Lisp Package Managers

Use ultralisp or ocicl.

Barebones + quicklisp

Locally clone the repository and quickload against the latest quicklisp dist.

git clone https://github.com/digikar99/py4cl2-cffi

Configuration

CL-USER> (ql:quickload "py4cl2-cffi/config")
To load "py4cl2-cffi/config":
  Load 1 ASDF system:
    py4cl2-cffi/config
; Loading "py4cl2-cffi/config"
[package py4cl2-cffi/config]
("py4cl2-cffi/config")

Loading py4cl2-cffi/config sets the following variables of interest in the py4cl2-cffi/config package:

In addition, py4cl2-cffi/config also exports the following useful symbols:

For the most part, configuration happens automatically while loading py4cl2-cffi/config. This requires that python3 and python3-config point to the right programs in the environment in which the lisp is run. Configuration in global or conda environments should be automatic then.

However, pyvenv environments require that users set *python-executable-path* manually, before loading py4cl2-cffi.

Basic Usage

(defpackage :python-lisp-user
  (:use :cl :py4cl2-cffi))

(in-package :python-lisp-user)

Start the embedded python by (pystart). If successful, (python-alive-p) should return T.

At the heart of py4cl2-cffi are pyvalue and pycall. This is in contrast to py4cl2. In py4cl2, raw-pyeval and raw-pyexec were the heart. There, pycall ultimately resulted in calls to raw-pyeval. However, py4cl2-cffi has the opposite behavior. Here, raw-pyeval and raw-pyexec ultimately depend on pyvalue and pycall. raw-pyeval and raw-pyexec are not only slow but also ugly in their implementations. Thus, whereever possible, you should try to use pyvalue, (setf pyvalue), and pycall.

pyvalue lets you obtain the value corresponding to a python name. It takes a string denoting a python name and returns a value corresponding to that name. For example,

(pyvalue "sys")
;=> #<PYOBJECT-WRAPPER :type <class 'module'>
;     <module 'sys' (built-in)>
;    {1004AAA1C3}>

An attempt will be made to convert the return values to lisp objects.

(pyvalue "answer_to_everything")
; Evaluation aborted on #<PY4CL2-CFFI:PYERROR {100514C153}>.
(setf (pyvalue "answer_to_everything") 42)
;=> 42
(pyvalue "answer_to_everything")
;=> 42

But if such a conversion is not possible (see define-lispifier and pythonize), an instance of pyobject-wrapper will be returned. Note that pyvalue also has a (setf pyvalue) which can be used to set the value of a python name, as we did in the above example.

The printing of a pyobject-wrapper-instance can be made less ugly by setting *print-pyobject-wrapper-identity* to NIL.

(pyvalue "sys")
;=> #<PYOBJECT-WRAPPER :type <class 'module'>
;    <module 'sys' (built-in)>
;   {1004AAA1C3}>
(setf *print-pyobject-wrapper-identity* nil)
;=> NIL
(pyvalue "sys")
;=> <module 'sys' (built-in)>

A python callable, that is, a python function, class, or anything that can be called, can be called using pycall. The first argument of pycall is usually the name of a python callable.

(pyvalue "str")
;=> <class 'str'>
(pycall "str" 42)
;=> "42"

But it can also be

Like pyvalue, pycall too will attempt to convert the return value to a lisp object. But if that is not possible, it will return a pyobject-wrapper-instance.

Python modules can be imported using the lightweight import-module. The values from the module can then be accessed using pyvalue, and callables can be called using pycall.

(import-module "math") ;=> T
(pyvalue "math.pi") ;=> 3.141592653589793d0
(pycall "math.hypot" 3 4) ;=> 5.0d0

A particular name from a particular python module can be imported using import-function.

(import-function "hypot" "math") ;=> T
(pycall "hypot" 12 5) ;=> 13.0d0

defpyfun and defpymodule are heavyweight alternatives to import-function and import-module. defpyfun defines a lisp function that calls the python callable. defpymodule defines lisp package(s) that holds certain symbols. These symbols are fbound to lisp functions which call corresponding python callables. For example, the below (defpymodule "math") form defines a lisp-package with name MATH.

(defpymodule "math")
; Defining MATH for accessing python package math..
;=> T

The lisp package MATH contains lisp functions that call corresponding python callables. For example the lisp function math:hypot below calls the python callable math.hypot.

(math:hypot 7 24) ;=> 25.0d0

Passing arrays by reference:

PYTHON-LISP-USER> (ql:quickload "array-operations")
To load "array-operations":
  Load 1 ASDF system:
    array-operations
; Loading "array-operations"

("array-operations")
PYTHON-LISP-USER> (let ((a (aops:rand* 'single-float 10))
                        (b (aops:rand* 'single-float 10)))
                    (print a)
                    (print b)
                    (pycall "numpy.add" a b :out a)
                    a)

#(0.5093733 0.615062 0.5520501 0.4115485 0.35940528 0.0056368113 0.31019592
  0.4214077 0.32522345 0.2879219)
#(0.23799527 0.9120656 0.99672806 0.54783416 0.91948783 0.14750922 0.68077135
  0.75351477 0.17053545 0.6163509)
#(0.7473686 1.5271276 1.5487782 0.95938265 1.2788931 0.15314603 0.9909673
1.1749225 0.4957589 0.9042728)

PYTHON-LISP-USER> (let ((a (aops:rand* 'double-float '(3 3))))
                    (print a)
                    (pycall "numpy.linalg.svd" a))
#2A((0.8441753387451172d0 0.3109557628631592d0 0.34773027896881104d0)
    (0.3423733711242676d0 0.6038261651992798d0 0.41209208965301514d0)
    (0.5945597887039185d0 0.06366562843322754d0 0.6331008672714233d0))
SVDResult(U=array([[-0.65892971,  0.44979069, -0.60290959],
       [-0.65302311, -0.73986678,  0.16173425],
       [-0.37332622,  0.50028539,  0.78124392]]), S=array([1.41737039, 0.5804025 , 0.01089903]), Vh=array([[-0.53068899, -0.75714703, -0.38091676],
       [-0.6309116 ,  0.65299712, -0.41898128],
       [ 0.56596798,  0.01797605, -0.82423122]]))

Callbacks

PYTHON-LISP-USER> (raw-pyexec "def foo(fn, *args, **kwargs): return fn(*args, **kwargs)")

; No value
PYTHON-LISP-USER> (pycall "foo" (lambda (d e &rest args &key a b &allow-other-keys)
                                  (declare (ignore a b))
                                  (list* d e args))
                          8 9 :a 2 :b 3 :d 5)
(8 9 "d" 5 "b" 3 "a" 2)

Numpy

PYTHON-LISP-USER> (defpymodule "numpy" t :silent t)
T
PYTHON-LISP-USER> (numpy.random:random '(2 3 4))
#3A(((0.9556724994386294d0 0.9207667929741092d0 0.38080996781642207d0
      0.36058417847643864d0)
     (0.1939761803809288d0 0.052707969761970785d0 0.5641774015926598d0
      0.34218703751890367d0)
     (0.663085466238284d0 0.8208948328437302d0 0.768715035806218d0
      0.8225795094037658d0))
    ((0.9523448513613038d0 0.8293149376922084d0 0.6616993552816121d0
      0.560839589292125d0)
     (0.004265522613073891d0 0.8874616779694773d0 0.45500882951834853d0
      0.34081255137211874d0)
     (0.3041085477740366d0 0.4351811902627044d0 0.031589664841209175d0
      0.6375274178283377d0)))
PYTHON-LISP-USER> (numpy:sum * :axis '(0 2))
#(5.622032172332849d0 2.8405971707274817d0 4.483681664998286d0)
PYTHON-LISP-USER> (with-lispifiers ((array (lambda (o)
                                             (magicl:from-array o (array-dimensions o)))))
                    (numpy.random:random '(3 4)))
#<MAGICL:MATRIX/DOUBLE-FLOAT (3x4):
   0.814     0.278     0.330     0.782
   0.858     0.342     0.282     0.225
   0.806     0.144     0.543     0.215>

Plotting

See https://github.com/digikar99/py4cl2-cffi/issues/11

PYTHON-LISP-USER> (import-module "matplotlib.pyplot" :as "plt")
T
PYTHON-LISP-USER> (pycall "plt.plot"
                     (alexandria:iota 10)
                     (mapcar (lambda (x) (* x x))
                             (alexandria:iota 10)))
#(#<PYTHON-OBJECT :type <class 'matplotlib.lines.Line2D'>
  Line2D(_child0)
 {1006670F83}>)
PYTHON-LISP-USER> (float-features:with-float-traps-masked t
                    (pycall "plt.show"))
#<PYTHON-OBJECT :type <class 'NoneType'>
  None
 {1006672273}>

<img margin="auto" width="75%" src="./plt-example.png"></img>

Optimization

Out of the box, py4cl2-cffi is about 10-15 times slower than native CPython.

PYTHON-LISP-USER> (import-module "math")
NIL
PYTHON-LISP-USER> (time
                   (loop for i below 100000
                         do (pycall "math.sin" i)))
Evaluation took:
  1.112 seconds of real time
  1.109842 seconds of total run time (1.105747 user, 0.004095 system)
  [ Real times consist of 0.052 seconds GC time, and 1.060 seconds non-GC time. ]
  [ Run times consist of 0.052 seconds GC time, and 1.058 seconds non-GC time. ]
  99.82% CPU
  2,448,781,258 processor cycles
  41,682,272 bytes consed

NIL

A simple way to speed up is to surround the block of code in (with-python-gil ...) or (pygil-ensure ...) and (pygil-release). However, lisp implementations usually perform garbage collection from separate thread. So, if pyobject-wrapper-objects are generated, there finalizers will never run until the GIL is released from the main thread.

PYTHON-LISP-USER> (time
                   (with-python-gil
                     (loop for i below 100000
                           do (pycall "math.sin" i))))
Evaluation took:
  0.792 seconds of real time
  0.794365 seconds of total run time (0.793760 user, 0.000605 system)
  100.25% CPU
  1,754,116,832 processor cycles
  30,403,504 bytes consed

NIL

A slightly more involved method involves setting py4cl2-cffi/config:*disable-pystop* to non-NIL before py4cl2-cffi is loaded. You may optionally need to compile py4cl2-cffi again. Thus -

;;; On a fresh lisp image
(asdf:load-system "py4cl2-cffi/config" :silent t)
(setf py4cl2-cffi/config:*disable-pystop* t)
(asdf:load-system "py4cl2-cffi" :force t)

(defpackage :python-lisp-user
  (:use :cl :py4cl2-cffi))

(in-package :python-lisp-user)

This will allow the compiler macro of pycall to dump the pointer to the callable into the compiled code. For example, see the SB-SYS:INT-SAP below.

PYTHON-LISP-USER> (disassemble
                   (compile nil `(lambda (x)
                                   (pycall "math.sin" x))))
; disassembly for (LAMBDA (X))
; Size: 35 bytes. Origin: #x557DD5CF                          ; (LAMBDA (X))
; CF:       498B4510         MOV RAX, [R13+16]                ; thread.binding-stack-pointer
; D3:       488945F8         MOV [RBP-8], RAX
; D7:       488B15BAFFFFFF   MOV RDX, [RIP-70]                ; #.(SB-SYS:INT-SAP #X753F7547FD80)
; DE:       488BFE           MOV RDI, RSI
; E1:       B904000000       MOV ECX, 4
; E6:       FF7508           PUSH QWORD PTR [RBP+8]
; E9:       B8828FB550       MOV EAX, #x50B58F82              ; #<FDEFN PY4CL2-CFFI::%PYCALL>
; EE:       FFE0             JMP RAX
; F0:       CC10             INT3 16                          ; Invalid argument count trap
NIL

The impact on performance:

PYTHON-LISP-USER> (time
                   (loop for i below 100000
                         do (pycall "math.sin" i)))
Evaluation took:
  0.520 seconds of real time
  0.521538 seconds of total run time (0.521249 user, 0.000289 system)
  100.38% CPU
  1,151,647,842 processor cycles
  17,628,400 bytes consed

NIL
PYTHON-LISP-USER> (time
                   (with-python-gil
                     (loop for i below 100000
                           do (pycall "math.sin" i))))
Evaluation took:
  0.324 seconds of real time
  0.322341 seconds of total run time (0.322341 user, 0.000000 system)
  99.38% CPU
  711,752,998 processor cycles
  9,600,240 bytes consed

NIL

defpymodule and defpyfun add an (import-module ...) form to the lisp function bodies when :safety is non-NIL (default). With this, the function can work correctly even if it was called after a (pystop). With *disable-pystop* set to NIL (default), (pystop) clears the global namespace. The (import-module ...) before the actual pycall can then import the module back to the global namespace. However, this incurs a performance penalty.

PYTHON-LISP-USER> (defpyfun "sin" "math" :lisp-fun-name "PYSIN")
PYSIN
PYTHON-LISP-USER> (time
                   (loop for i below 10000
                         do (pysin i)))
Evaluation took:
  2.872 seconds of real time
  2.928811 seconds of total run time (2.508456 user, 0.420355 system)
  101.98% CPU
  6,349,788,686 processor cycles
  26,563,104 bytes consed

NIL

By passing :safety nil, the (import-module ...) form can be avoided.

PYTHON-LISP-USER> (defpyfun "sin" "math" :lisp-fun-name "PYSIN"
                    :safety nil)
WARNING: redefining PYTHON-LISP-USER::PYSIN in DEFUN
PYSIN
PYTHON-LISP-USER> (time
                   (loop for i below 10000
                         do (pysin i)))
Evaluation took:
  0.060 seconds of real time
  0.060613 seconds of total run time (0.060607 user, 0.000006 system)
  101.67% CPU
  133,806,218 processor cycles
  2,424,544 bytes consed

NIL

Overall, these measures allow py4cl2-cffi to be within a factor of 5 of CPython. See ./perf-compare/.

Developer Thoughts on Garbage Collection

If you are working with raw pointers, then all bets are off about handling garbage collection.

Thus, the only time garbage collection should "work correctly" aka - without (i) segmentation faults (ii) memory leaks - is when you are not working with raw pointers. In other words, functions that return lisp values must perform garbage collection.

An exhaustive list of functions that return lisp values include:

Even amongst these, GC should not take place until the top level call has done its processing. To be more intrincate seems to require a merge of the python interpreter with the lisp interpreter.

The single place which decides what to not collect is the function "lispify" when it returns a python-object struct-wrapper around the pyobject. In these cases, we PYUNTRACK the pointers.

During DecRef-ing through an object finalizer, one needs to hold the GIL, because at least on SBCL, the finalizer may be called through any thread. DecRef-ing without holding the GIL results in segmentation faults.

API Reference

*additional-init-codes*

Variable
Default Value: NIL

A list of strings each of which should be python code. All the code will be executed by pystart.

*defpymodule-silent-p*

Variable
Default Value: NIL

defpymodule avoids printing progress if this is T.

*internal-features*

Variable
Default Value: (:TYPED-ARRAYS :WITH-PYTHON-OUTPUT)

A list of PY4CL2 features available on the system. (Support for :ARRAYS requires numpy and is only determinable after python process has started.)

The list can include one or more of:

:with-python-output :TYPED-ARRAYS

*lispifiers*

Variable
Default Value: NIL

Each entry in the alist LISPIFIERS maps from a lisp-type to a single-argument lisp function. This function takes as input the "default" lisp objects and is expected to appropriately parse it to the corresponding lisp object.

NOTE: This is a new feature and hence unstable; recommended to avoid in production code.

*print-pyobject*

Variable
Default Value: T

If non-NIL, python's 'str' is called on the python-object before printing.

*print-pyobject-wrapper-identity*

Variable
Default Value: T

If non-NIL, print's the lisp type and identity of the pyobject-wrapper.

*pygc-threshold*

Variable
Default Value: 1000

Number of references in PYTHON-NEW-REFERENCES after which PYGC manipulates reference counts.

*pythonizers*

Variable
Default Value: NIL

Each entry in the alist PYTHONIZERS maps from a lisp-type to a single-argument PYTHON-FUNCTION-DESIGNATOR. This python function takes as input the "default" python objects and is expected to appropriately convert it to the corresponding python object.

NOTE: This is a new feature and hence unstable; recommended to avoid in production code.

+disable-pystop+

Constant: NIL

+py-empty-tuple+

No documentation found for +py-empty-tuple+

+py-empty-tuple-pointer+

No documentation found for +py-empty-tuple-pointer+

+py-none+

No documentation found for +py-none+

+py-none-pointer+

No documentation found for +py-none-pointer+

chain

Macro: (chain &rest chain)

This is inspired by PARENSCRIPT:CHAIN, discussed in this issue: https://github.com/bendudson/py4cl/issues/4

In python it is quite common to apply a chain of method calls, data member access, and indexing operations to an object. To make this work smoothly in Lisp, there is the chain macro (Thanks to @kat-co and parenscript for the inspiration). This consists of a target object, followed by a chain of operations to apply. For example

(chain "hello {0}" (format "world") (capitalize)) ; => "Hello world"

which is interpreted as: "hello {0}".format("world").capitalize().

Or -

(chain "hello {0}" (format "world") (capitalize) (aref 1)) ; => "e"

which is interpreted as: "hello {0}".format("world").capitalize()[1].

chain has two variants: chain is a macro, with the elements of chain unevaluated, while chain* is a function with its elements (arguments) evaluated according to a normal lisp function call..

Some more examples are as follows:

(chain (slice 3) stop) ; => 3
(let ((format-str "hello {0}")
      (argument "world"))
 (chain* format-str `(format ,argument))) ; => "hello world"

Arguments to methods are lisp, since only the top level forms in chain are treated specially:

CL-USER> (chain (slice 3) stop)
3
CL-USER> (let ((format-str "hello {0}")
               (argument "world"))
           (chain* format-str `(format ,argument)))
"hello world"
CL-USER> (chain* "result: {0}" `(format ,(+ 1 2)))
"result: 3"
CL-USER> (chain (aref "hello" 4))
"o"
CL-USER> (chain (aref "hello" (slice 2 4)))
"ll"
CL-USER> (chain (aref #2A((1 2 3) (4 5 6)) (slice 0 2)))
#2A((1 2 3) (4 5 6))
CL-USER> (chain (aref #2A((1 2 3) (4 5 6)) 1 (slice 0 2))) ; array[1, 0:2]
#(4 5)
CL-USER> (pyexec "class TestClass:
    def doThing(self, value = 42):
        return value")
CL-USER> (chain ("TestClass") ("doThing" :value 31))
31

There is also (SETF chain). However, this is more experimental. It requires that the python object remains as a wrapper. with-remote-objects can ensure this. with-remote-objects* additionally lispifies the return value:

CL-USER> (with-remote-objects*
           (let ((lst (pycall "list" '(1 2 3))))
             (setf (chain* `(aref ,lst 0)) 4)
             lst))
#(4 2 3)

Note that this modifies the value in python, so the above example only works because lst is a pyobject-wrapper, rather than a lisp object. The following therefore does not work:

CL-USER> (let ((lst (pycall "list" '(1 2 3))))
           (setf (chain* `(aref ,lst 0)) 4)
           lst)
#(1 2 3)

chain*

Function: (chain* &rest chain)

define-lispifier

Macro: (define-lispifier name (pyobject-var) &body body)

defpyfun

Macro: (defpyfun fun-name &optional pymodule-name &key (as fun-name) (cache t)
        (lisp-fun-name (lispify-name as)) (lisp-package *package*) (safety t))

Defines a function which calls python

Example

(py4cl:pyexec "import math")
(py4cl:defpyfun "math.sqrt")
(math.sqrt 42) -> 6.4807405

Arguments:

defpymodule

Macro: (defpymodule pymodule-name &optional (submodules NIL) &key (cache t)
        (continue-ignoring-errors t)
        (lisp-package (lispify-name pymodule-name)) (reload t)
        (recompile-on-change NIL) (safety t) (silent *defpymodule-silent-p*))

Import a python module (and its submodules) as a lisp-package(s).

Example:

(py4cl:defpymodule "math" :lisp-package "M")
(m:sqrt 4)   ; => 2.0

Arguments:

disable-pygc

Macro: (disable-pygc)

enable-pygc

Macro: (enable-pygc)

export-function

Function: (export-function function python-name)

Makes a lisp function available in python process as python-name

import-function

Function: (import-function name from &key (as NIL))

import-module

Function: (import-module name &key (as NIL))

pycall

Function: (pycall python-callable &rest args)

If python-callable is a string or symbol, it is treated as the name of a python callable, which is then retrieved using PYVALUE*

pyerror

Condition

A lisp error to indicate all kinds of python error.

pyeval

Function: (pyeval &rest args)

pyexec

Function: (pyexec &rest args)

pygenerator

Function: (pygenerator function stop-value)

pyhelp

Function: (pyhelp string-or-python-callable)

pymethod

Function: (pymethod object method-name &rest args)

pymethod-list

Function: (pymethod-list pyobject &key (as-vector NIL))

pyobject-wrapper

Structure

A wrapper around a pointer to a python object. LOAD-FORM is used if the pyobject-wrapper is dumped into a compiled lisp file.

pyobject-wrapper-eq

Function: (pyobject-wrapper-eq o1 o2)

Returns T if o1 and o2 are both pyobject-wrapper with the same pointer, or the same lisp objects which are EQ to each other. Returns NIL in all other cases.

pyobject-wrapper-eq*

Function: (pyobject-wrapper-eq* o1 o2)

Like pyobject-wrapper-eq but assumes that o1 and o2 are pyobject-wrapper each.

pyref

Function: (pyref object &rest indices)

Wrapper around python's getitem along with support for slicing.

Example

(pyref "hello" 1) ;=> "e"
; NOTE: Python does not make a distinction between strings and characters

(pyref #(1 2 3) 1) ;=> 2
(pyref #(1 2 3 4 5) '(slice 3 5)) ;=> #(4 5)

(import-module "numpy")
(with-remote-objects*
  (pyref (chain ("numpy.arange" 12) ("reshape" (3 4)))
         `(slice ,+py-none+)
         0))

pyslot-list

Function: (pyslot-list pyobject &key (as-vector NIL))

pyslot-value

Function: (pyslot-value object slot-name)

pystart

Function: (pystart)

pystop

Function: (pystop)

python-alive-p

Function: (python-alive-p)

python-getattr

Generic Function: (python-getattr object slot-name)

Called when python accesses an object's slot (getattr)

python-setattr

Generic Function: (python-setattr object slot-name value)

Called when python sets an object's slot (setattr)

python-start-if-not-alive

Function: (python-start-if-not-alive)

pythonize

Generic Function: (pythonize lisp-value-or-object)

Given a lisp object, return a CFFI:FOREIGN-POINTER pointing to the python object corresponding to the given lisp object.

The implemented methods are expected to return a new (strong) reference to the python object. The method is also expected to call PYTRACK to notify the PYGC functionality to delete the reference once the object is no longer needed.

See the documentation for PYGC to understand when reference deletion takes place.

pyvalue

Function: (pyvalue python-name-or-variable)

Get the value of a python-name-or-variable.

Example:

(pyvalue "sys") ;=> <module 'sys' (built-in)>
(pyvalue "sys.path")
;=>
  #("/home/user/miniconda3/lib/python310.zip"
    "/home/user/miniconda3/lib/python3.10"
    "/home/user/miniconda3/lib/python3.10/lib-dynload"
    "/home/user/miniconda3/lib/python3.10/site-packages")

pyversion-info

Function: (pyversion-info)

Return a list, using the result of python's sys.version_info.

raw-pyeval

Function: (raw-pyeval &rest code-strings)

Unlike PY4CL or PY4CL2, the use of RAW-PY, raw-pyeval and raw-pyexec, pyeval, pyexec should be avoided unless necessary. Instead, use pycall, pyvalue, (SETF pyvalue), pyslot-value, (SETF pyslot-value), and pymethod.

RAW-PY, raw-pyeval, raw-pyexec are only provided for backward compatibility.

raw-pyexec

Function: (raw-pyexec &rest code-strings)

Unlike PY4CL or PY4CL2, the use of RAW-PY, raw-pyeval and raw-pyexec, pyeval, pyexec should be avoided unless necessary. Instead, use pycall, pyvalue, (SETF pyvalue), pyslot-value, (SETF pyslot-value), and pymethod.

RAW-PY, raw-pyeval, raw-pyexec are only provided for backward compatibility.

simple-pyerror

Condition

A specialization of PYERROR to hold the python error type.

<u>Direct Slots</u>

type

Initargs: :TYPE

with-lispifiers

Macro: (with-lispifiers (&rest overriding-lispifiers) &body body)

Each entry of overriding-lispifiers is a two-element list of the form

(TYPE LISPIFIER)

Here, TYPE is unevaluated, while LISPIFIER will be evaluated; the LISPIFIER is expected to take a default-lispified object (see lisp-python types translation table in docs) and return the appropriate object user expects.

For example,

(raw-pyeval "[1, 2, 3]") ;=> #(1 2 3) ; the default lispified object
(with-lispifiers ((vector (lambda (x) (coerce x 'list))))
  (print (raw-pyeval "[1,2,3]"))
  (print (raw-pyeval "5")))
; #(1 2 3) ; default lispified object
; (1 2 3)  ; coerced to LIST by the lispifier
; 5        ; lispifier uncalled for non-VECTOR
5

NOTE: This is a new feature and hence unstable; recommended to avoid in production code.

with-pygc

Macro: (with-pygc &body body)

Code surrounded by with-pygc performs garbage collection only after executing all of body.

with-python-error-output

Macro: (with-python-error-output &body forms-decl)

Gets the output of the python program executed in forms-decl in the form a string.

with-python-gil

Macro: (with-python-gil &body body)

with-python-output

Macro: (with-python-output &body forms-decl)

Gets the output of the python program executed in forms-decl in the form a string.

with-pythonizers

Macro: (with-pythonizers (&rest overriding-pythonizers) &body body)

Each entry of overriding-pythonizers is a two-element list of the form

(TYPE PYTHONIZER)

Here, TYPE is unevaluated, while PYTHONIZER will be evaluated; the PYTHONIZER is expected to take a default-pythonized object (see lisp-python types translation table in docs) and return the appropriate object user expects.

For example,

; A convenience function
(defun pyprint (object)
  (pycall "print" object)
  (pycall "sys.stdout.flush")
  (values))

(pyprint #(1 2 3)) ; prints [1, 2, 3] ; the default object
(with-pythonizers ((vector "tuple"))
  (pyprint #(1 2 3))
  (pyprint 5))
; (1, 2, 3) ; coerced to tuple by the pythonizer
; 5         ; pythonizer uncalled for non-VECTOR
5

NOTE: This is a new feature and hence unstable; recommended to avoid in production code.

with-remote-objects

Macro: (with-remote-objects &body body)

Ensures that all values returned by python functions and methods are kept in python, and only pointers are returned to lisp. This is useful if performing operations on large datasets.

with-remote-objects*

Macro: (with-remote-objects* &body body)

Ensures that all values returned by python functions and methods are kept in python, and only handles returned to lisp. This is useful if performing operations on large datasets. Unlike with-remote-objects, evaluates the last result and returns not just a handle.