Create/Document API to call Julia functions with keywords from C

[ronisbr]

Hi guys!

I have asked on Discourse about how can I call from C a Julia function with keywords [1]. After some researching, the only way I found was to create a dummy function without keywords that call the original one with keywords.

Since nobody answered me on Discourse, I am not sure if this is not possible or if this is not documented.

---

[1] https://discourse.julialang.org/t/call-julia-function-with-keywords-from-c/20981/2

[ronisbr]

Looking at the documentation, I found a way using the Core.kwfunc. Hence, the current way I am using to call a function with keywords is:

#include <iostream>

extern "C" {
#include <julia.h>
}

int main(int argv, char* argc[])
{
    jl_init();

    jl_function_t* teste = jl_eval_string("function teste(a;c = 1.0) \
            a+c \
            end");

    jl_function_t* kwsorter = jl_eval_string("Core.kwfunc");
    jl_function_t* testek = jl_call1(kwsorter,teste);

    jl_value_t* a = nullptr;
    jl_value_t* b = nullptr;
    jl_value_t* c = nullptr;

    JL_GC_PUSH3(a,b,c);

    a = (jl_value_t*)jl_box_float64(1.0);
    b = (jl_value_t*)jl_box_float64(1.0);
    c = (jl_value_t*)jl_eval_string("(c = 3.0,)");

    jl_value_t* ret = (jl_value_t*)jl_call3(testek,c,teste,a);

    std::cout << jl_unbox_float64(ret) << std::endl;

    JL_GC_POP();

    jl_atexit_hook(0);

    return 0;
}

Now, I just need to find a way to create NamedTuples in C.

[cnuernber]

@mkitti - This is where I am stuck as well. How to call keyword functions from C. That seems to getting into how to create tuples and named tuples from C as the original poster mentioned.

[ronisbr]

@cnuernber in my case I have never succeeded in doing so :(

What I did was run a Julia code inside C that creates a dummy function to avoid the keywords. Something like this:

my_function(a, b, c) = other_function(a = a, b = b, c = c)

[mkitti]

I'm digging, but the easiest way may be to use https://github.com/JeffreySarnoff/NamedTupleTools.jl

[cnuernber]

OK, both of these are valid pathways forward. From the C interface there is jl_apply_tuple_v but no jl_apply_named_tuple_v which to me seems a natural extension.

Thanks both of you, this is totally sufficient to move forward.

[mkitti]

Let me preface by saying that anything outside of the embedding documentation may be an unstable API. The official line is to make ample use of jl_eval_string.

The relevant history is that NamedTuple was added to Julia in 0.7 in #22194, which is rather late. Julia v0.7 was just before 1.0. This is a hint that NamedTuple may not be a low level type. Rather it is a parameterized struct.

Let's explore in Julia first:

julia> @which NamedTuple # NamedTuple is defined in the Core module
Core

julia> typeof(NamedTuple) # NamedTuple is of type UnionAll
UnionAll

julia> isstructtype(NamedTuple) # NamedTuple is basically just a parameterized struct
true

julia> Base.unwrap_unionall(NamedTuple)
NamedTuple{names,T<:Tuple}

julia> NamedTuple.body
NamedTuple{names,T} where T<:Tuple

julia> typeof(NamedTuple.body)
UnionAll

julia> NamedTuple.body.body # unwrap_unionall just iterates through body
NamedTuple{names,T<:Tuple}

julia> typeof(NamedTuple.body.body)
DataType

Let's fill in the two parameters for a NamedTuple:

julia> nt_datatype_ints = NamedTuple{(:a,:b),Tuple{Int,Int}} # Create a DataType, jl_apply_type2
NamedTuple{(:a, :b),Tuple{Int64,Int64}}

julia> nt_datatype_ints((5,3)) # Instantiate the type, jl_new_struct
(a = 5, b = 3)

julia> NamedTuple{(:a,:b)} # We can curry parameters, jl_apply_type1
NamedTuple{(:a, :b),T} where T<:Tuple

julia> NamedTuple{(:a,:b)}{Tuple{Int,Int}} # jl_apply_type1 (applied twice)
NamedTuple{(:a, :b),Tuple{Int64,Int64}}

julia> NamedTuple{(:a,:b)}{Tuple{Int,Int}}((0,1)) # jl_apply_type1 (applied twice), jl_new_struct
(a = 0, b = 1)

julia> names = (:a,:b)
(:a, :b)

julia> values = (5,3)
(5, 3)

julia> NamedTuple{names,typeof(values)}
NamedTuple{(:a, :b),Tuple{Int64,Int64}}

julia> NamedTuple{names,typeof(values)}(values)
(a = 5, b = 3)

Getting to C, first a reference to the NamedTuple is needed. jl_namedtuple_type is exported here:

julia/src/julia.h

Line 686 in ab94776

extern JL_DLLEXPORT jl_unionall_t *jl_namedtuple_type JL_GLOBALLY_ROOTED;

That is defined here:

julia/src/jltypes.c

Lines 2437 to 2444 in ab94776

	jl_tvar_t *ntval_var = jl_new_typevar(jl_symbol("T"), (jl_value_t*)jl_bottom_type,
	(jl_value_t*)jl_anytuple_type);
	tv = jl_svec2(tvar("names"), ntval_var);
	jl_datatype_t *ntt = jl_new_datatype(jl_symbol("NamedTuple"), core, jl_any_type, tv,
	jl_emptysvec, jl_emptysvec, 0, 0, 0);
	jl_namedtuple_type = (jl_unionall_t*)ntt->name->wrapper;
	((jl_datatype_t*)jl_unwrap_unionall((jl_value_t*)jl_namedtuple_type))->layout = NULL;
	jl_namedtuple_typename = ntt->name;

The psuedocode is thus:

1. Obtain a jl_datatype_t * by using jl_apply_type on jl_namedtuple_type to pass the parameters names and T
   i. names is a kind of Tuple usually of type Tuple{Symbol,Symbol, ... }
   ii. T is the type of the values. Use jl_typeof(values). This should be something like Tuple{Int,Int}
2. Pass the jl_datatype_t * to jl_new_struct with the values as a Tuple.

I have not tested this yet.

The main point is that there is nothing special about a NamedTuple. It is just a normal composite datatype.

[ronisbr]

@mkitti awesome! Your suggestion worked perfectly:

#include <iostream>

extern "C" {
#include <julia.h>

}

int main(int argv, char* argc[])
{
    jl_init();
    jl_gc_enable(0);

    jl_function_t* display = jl_eval_string("dump");

    jl_function_t* teste = jl_eval_string("function teste(a;c = 1.0, b = 2.0) \
            a+b+c \
            end");

    jl_function_t* kwsorter = jl_eval_string("Core.kwfunc");
    jl_function_t* testek = jl_call1(kwsorter,teste);

    jl_value_t* a = nullptr;
    jl_value_t* b = nullptr;
    jl_value_t* c = nullptr;

    a = (jl_value_t*)jl_box_float64(1.0);
    b = (jl_value_t*)jl_box_float64(5.0);
    c = (jl_value_t*)jl_box_float64(8.0);

    // Named tuple
    // =========================================================================

    // Tuple type with the types of the values.
    jl_tupletype_t* tvalues_type =
        jl_apply_tuple_type(jl_svec2(jl_typeof(b), jl_typeof(c)));

    // Tuple with the names of the keyword arguments.
    jl_tupletype_t* tnames_type =
        jl_apply_tuple_type(jl_svec2(jl_symbol_type, jl_symbol_type));

    jl_value_t* tnames = jl_new_struct(tnames_type,
                                       jl_symbol("b"),
                                       jl_symbol("c"));

    // Create the type of the named tuple.
    jl_datatype_t* nt_type =
        (jl_datatype_t*)jl_apply_type2((jl_value_t*)jl_namedtuple_type,
                                       (jl_value_t*)tnames,
                                       (jl_value_t*)tvalues_type);

    // Create the named tuple.
    jl_value_t* kwargs = jl_new_struct(nt_type, b, c);

    // Call function
    // =========================================================================

     jl_value_t* ret = (jl_value_t*)jl_call3(testek, kwargs, teste, a);

    std::cout << jl_unbox_float64(ret) << std::endl;

    jl_atexit_hook(0);

    return 0;

}

NOTE: this code should not be use in production, we need a log of JL_GC_PUSH. I just disabled the garbage collector to avoid segmentation faults...

$ ./a.out
14

[ronisbr]

When I have some spare time, I will improve this code, wait for someone to review it, and then submit a PR to update the documentation stating how this can be achieved.

[cnuernber]

Wow, yep the C pathway above works for me:

(defn named-tuple
  "Create a julia named tuple from a map of values."
  [value-map]
  (let [generic-nt-type (lookup-julia-type :jl-namedtuple-type)
        [jl-values nt-type]
        (julia-jna/with-disabled-julia-gc
          (let [item-keys (apply tuple (keys value-map))
                map-vals (vals value-map)
                jl-values (base/jvm-args->julia map-vals)
                item-type-tuple (apply apply-tuple-type (map julia-jna/jl_typeof jl-values))
                nt-type (julia-jna/jl_apply_type2 generic-nt-type item-keys
                                                  item-type-tuple)]
            [jl-values nt-type]))]
    (base/check-last-error)
    ;;And now, with gc (potentially) enabled, attempt to create the struct
    (apply struct nt-type jl-values)))



libjulia-clj.julia> (def test-fn (eval-string "function teste(a;c = 1.0, b = 2.0) 
    a+b+c 
end"))
Nov 29, 2020 8:58:16 AM clojure.tools.logging$eval6526$fn__6529 invoke
INFO: Rooting address  0x00007F3BDA988010
#'libjulia-clj.julia/test-fn
libjulia-clj.julia> (def kwfunc (eval-string "Core.kwfunc"))
Nov 29, 2020 8:58:37 AM clojure.tools.logging$eval6526$fn__6529 invoke
INFO: Rooting address  0x00007F3BE0540840
#'libjulia-clj.julia/kwfunc
libjulia-clj.julia> (def test-kwf (kwfunc test-fn))
Nov 29, 2020 8:59:14 AM clojure.tools.logging$eval6526$fn__6529 invoke
INFO: Rooting address  0x00007F3BDA988020
#'libjulia-clj.julia/ttest-kwf
libjulia-clj.julia> (def a 1.0)
#'libjulia-clj.julia/a
libjulia-clj.julia> (def b 5.0)
#'libjulia-clj.julia/b
libjulia-clj.julia> (def c 8.0)
#'libjulia-clj.julia/c
libjulia-clj.julia> (def arg-tuple (named-tuple {:b 10 :c 20}))
Nov 29, 2020 9:00:10 AM clojure.tools.logging$eval6526$fn__6529 invoke
INFO: Rooting address  0x00007F3BDA98D190
#'libjulia-clj.julia/arg-tuple
libjulia-clj.julia> arg-tuple
(b = 10, c = 20)
libjulia-clj.julia> (test-kwf arg-tuple test-fn a)
31.0

[cnuernber]

Here is the partial specialization pathway Mark initially showed us above:

user> (def nt-type (julia/eval-string "NamedTuple"))
#'user/nt-type
user> (def partial-specialized (julia/apply-type nt-type (julia/tuple :a :b)))
#'user/partial-specialized
user> partial-specialized
NamedTuple{(:a, :b),T} where T<:Tuple
user> (def int-tuple-type (julia/apply-tuple-type 
                           (julia/lookup-julia-type :int64)
                           (julia/lookup-julia-type :int64)))
#'user/int-tuple-type
user> int-tuple-type
Tuple{Int64,Int64}
user> (julia/apply-type partial-specialized int-tuple-type)
NamedTuple{(:a, :b),Tuple{Int64,Int64}}
user> (def full-type *1)
#'user/full-type
user> (julia/struct full-type 1 2)
(a = 1, b = 2)
user> (full-type (julia/tuple 1 2))
(a = 1, b = 2)
user> (full-type (julia/tuple 1 2.0))
(a = 1, b = 2)