julia functions as C callbacks

[vtjnash]

--- update ---
the original issue is pretty much covered by cfunction. the remaining issues are:

• support closures
• adding conversion wrapper in case of inexact matches

---- original text ---
To enable certain function calls to C code, it would be helpful to have a function callback API in Julia (this is partly inspired by the libuv interface, which ended up handling most of these issues by creating a tight coupling between julila function and types and libuv callbacks and types)

The two issues that I currently see would need to be handled are:

1. Since pointers are leaving control of Julia, the garbage collector needs to know not to clean-up anything that may be used in the future (and notified when it can)
2. Since C doesn't natively have type information, it requires some sort of reverse ccall interface

To handle 1, I think it may be safest to create a garbage collector allocation pool. Objects can be pushed to and popped from it from within Julia (using ref counting?). Then possibly add a default behavior that parameters to a ccall containing a Callback object are automatically saved until all callbacks are manually deleted. (since additional and previous calls to C functions may also hold onto object pointers, it may be useful to have this as a separately available function)

To handle 2, it will be necessary to indicate type information to the ccall. One possibility is to make a type that contains the information type Callback{ params <: Tuple } end and is passed to the ccall in the third argument tuple. A generic function name would then be passed in the parameters list. A second possibility is to make Callback a simple type that contains all of that information type Callback; params <: Tuple; f :: Function; end. A third (similar) possibility is to do the exact function lookup in Julia and require something like the value returned by getmethods and printed by whicht. In any case, I envision that ccall would JIT an intermediary function that converts from raw bits types to Julia types. This prompts the last possibility (and possibly my favorite) of having make_callback return a raw pointer to a newly JIT'd intermediary C function:

callback_fcn = make_callback(fcn::Function, arg_types::Tuple)
arg2 = "test"
cpreserve(arg2); cpreserve(fcn) # if args are not automatically preserved
ccall(:cfcn, Void, (Ptr, String), callback_fcn, arg2)
...
crelease(arg2) # if args are not automatically preserved
crelease(fcn) # if args are automatically preserved, arg2 won't be freed till after this point

A final question is whether pointer_to_array is the best way to grab a c array? Or is there some way to indicate this in the argument type?

Thoughts and alternatives? Am I missing any challenging callback argument types?

[dioptre]

Hi mate,

It might be worthwhile looking at:
http://www.erlang.org/doc/apps/erts/driver.html#id82769

That's how we do it in erlang. You may want to look into the argument definitions.

One of the reasons for me checking out Julia today is to see if I can work with it and Erlang. I'd really like to be able to call Julia functions from Erlang and visa-versa. The issue with using callbacks means there will probably end up being a huge marshalling overhead. It would be great if we could produce a "shared nothing" middle man, extending the idea of libuv with common structs/object definitions for direct access and an independent "middle man" garbage tool for synchronizing the shared memory.

I'm looking to do this over clusters of machines, which Erlang is fantastic for - but I don't know if using callbacks will cut the mustard. Ideally, I'd like to directly call Julia, and manage its resources externally (with an api, static link etc.) but potentially getting the two systems to call each other through callbacks via a complex queue and resource monitoring process including garbage collection might be the only way for now. What do you think?

It might be good to do something that would work for everything. Let me know if you are interested.

Cheers
A

[StefanKarpinski]

Having a good interop story (for Python/NumPy in particular, but not exclusively) is very important to us, so this is a helpful design point to consider. Although we can't quite achieve it right now, we will eventually have C-callable Julia functions so that there is no function call overhead if you can do C-style function calls. Do you know if Erlang can do that?

[dioptre]

Yes it can with NIFs and Port Drivers,

Take a look at them here:
http://www.erlang.org/doc/tutorial/nif.html

Erlang (scale) and Julia (speed) would be a really great partnership - it would be great to get these two working together.

If you have anything that you've started on sharing julia as a library, I'd like to see/help.

Cheers
A

[dioptre]

I haven't used the Julia ZMQ implementation, but that may be a good alternative with little overhead in the meantime. What do you think?

Cheers
A

[timholy]

I've got Matlab successfully interoperating with Julia, so in principle it works very well. The main issue is the serializer. I targeted Julia's native serializer, but several folks have pointed out that we might need a more "standardized" serializer for ZMQ interop.

[timholy]

@vtjnash, to me your plans look great. I don't have anything to add, other than my enthusiasm for this development!

I don't know of a way to indicate an array in the arguments to ccall, but you're right that it would be more elegant.

[vtjnash]

I don't see an explanation on those pages for how to pass references to arbitrary Erlang functions and call them from C, which was the original point of this post. If I've missed it, can you help point me in the right direction? An example function of this is calling the system qsort which has declaration: void qsort(void *base, size_t nmemb, size_t size, int(*compar)(const void *, const void *)) (I know Julia has a built-in qsort, this is just an example). Julia already has a great FFI, with the function ccall, that requires no special c code interface for many typical functions. I would like to extend this to allow C code to use Julia functions as native callbacks (with as minimal overhead as possible).

I second the comment that using ZMQ for interop on new programs is probably a really good alternative, with low overhead and more flexible compatibility. I was really excited to see that get added to the Julia library. This proposal is more targeting existing API's that require C function style callbacks.

Using Julia as a shared library is a slightly different issue and would not be covered by this proposal. Although, it could be pretty straightforward to code an interface library that would register callbacks. Although it must be observed that Julia is single-threaded and must not be called from a different thread.

[dioptre]

This is my third day using Julia, so excuse the stab in the dark...

I'm not sure of a way to directly access erlang functions without exporting them (much like c), using callbacks or direct calls (both in the example provided) or using messaging. For the erlang system to be particularly useful, it can use multi-threaded asynchronous callbacks. If we only have a single thread, I'd imagine the call would be blocking? If this is the case why would callbacks be used at all?

If Julia is single threaded, how can we manage a non trivial message queue (or a multithreaded equivalent). I can't see anything in the Julia parrallel computing documentation that describes events, job prioority, or the ability to monitor (and fix) it's health. How does it manage multiple connections if it has only 1 thread? What if it stalls? Is there internal message passing? Can we use this if it exists? How does julia stop blocking? Does it?

I hope your response will be of use to more people interested in integrating with Julia too!

Thanks
A

[pao]

@dioptre, that is a topic well suited for discussion on -dev.

[dioptre]

@pao I've pushed the questions to https://groups.google.com/forum/?hl=en-GB&fromgroups#!topic/julia-dev/Nk-AWe2netA

I discovered this https://github.com/dioptre/julia-nlopt/blob/master/src/nlopt/nlopt_wrapper.c which may be a start.

Anyhow - would still like to hear from anyone how they recommend managing many connections/events/etc.

Thanks
A

[JeffBezanson]

Callbacks are now supported for top-level generic functions whose signatures can be made to match a given C signature. Example:

julia> f(x) = 2x+1

julia> p = cfunction(f, Float64, (Float64,))
Ptr{Void} @0x00007f4df81f8b88

julia> ccall(p, Float64, (Float64,), 2.1)
5.2

Still need to add support for closures, and possibly adding conversions in case of inexact matches.

[StefanKarpinski]

This is such a big deal in and of itself, but also as a step in the direction of being able to generate binaries and shared libraries.