Is it possible to mix imported and exported resources?

[Sintrastes]

Hello!

First off, I am not sure if this is the proper place for this issue, as I don't know if this issue is a problem with the component model itself, or if the issue is just with cargo-component. If this is not an issue with the component model / WIT itself, I am happy to move this issue to the appropriate repo.

Since I understand that stream<T> is still a work in progress, I decided to try to roll my own WIT worlds / interfaces for reactive flows. Since WIT does not currently have first-class functions / callback types, I tried to emulate this via resource types. I did something like:

interface callbacks {
    resource s32-update-handler {
        on-update: func(value: s32);
    }
}

interface reactive-values {
    resource s32-reactive-value {
        current-value: func() -> s32;
        register-update-handler: func(handler: s32-update-handler);
        unregister-update-handler: func(handler: s32-update-handler);
    }

    new-s32-reactive-value: func(initial: s32) -> tuple<s32-update-handler, s32-reactive-value>;
}

world my-world {
    export callbacks;
    import reactive-values;
}

The callbacks are exported so that the client component can implement the callbacks natively, and the reactive-values are imported, because this is something that should be implemented by the host / another wasm component (to keep the implementation of the reactive values decoupled from the client component).

However, when I try to use this WIT to implement a component with cargo-component, with the following snippet:

let (set_x, x) = new_s32_reactive_value(0);
let (set_y, y) = new_s32_reactive_value(0);

let (set_x_y_sum, x_y_sum) = new_string_reactive_value("0");

x.register_update_handler(S32UpdateHandler::new(IntCallback(Box::new(|x| {
    let y = y.current_value();
    let sum = (x + y).to_string();
    set_x_y_sum.on_update(sum);
}))));

y.register_update_handler(S32UpdateHandler::new(IntCallback(Box::new(|y| {
    let x = x.current_value();
    let sum = (x + y).to_string();
    set_x_y_sum.on_update(&sum);
}))));

I get the error:

mismatched types
`S32UpdateHandler` and `S32UpdateHandler` have similar names, but are actually distinct typesrustc[Click for full compiler diagnostic](rust-analyzer-diagnostics-view:/diagnostic%20message%20[5]?5#file:///Users/nathan/Code/wasm-rust-gui-example/src/lib.rs)
lib.rs(32, 11): arguments to this method are incorrect
bindings.rs(5222, 17): `S32UpdateHandler` is defined in module `crate::bindings::exports::component::wasm_rust_gui_example::update_handlers` of the current crate
bindings.rs(502, 13): `S32UpdateHandler` is defined in module `crate::bindings::component::wasm_rust_gui_example::update_handlers` of the current crate
bindings.rs(1254, 24): method defined here

So it seems to me like cargo-component is generating two separate types for S32UpdateHandler for imported versions of the type, and exported versions of the type, which makes sense to me -- because obviously the implementation is going to be different depending on whether the resource is coming from your component, or from an external component.

However, what I really want is (to make up some syntax for an "imported" and "exported" modality for types:

imported resource s32-reactive-value {
      current-value: func() -> s32;
      register-update-handler: func(handler: exported s32-update-handler);
      unregister-update-handler: func(handler: exported s32-update-handler);
}

In other words, to be able to use callbacks (update handlers) that have been exported from the client within an imported resource type. But since there isn't a way of expressing the "import / export" modality (for lack of a better term) of the types in such a granular way, I am not sure how to accomplish this.

Is it possible to do what I am trying to do here in WIT today and maybe I am just doing things wrong? Do the semantics of imported / exported resources need to be refined / clarified in order to support this kind of use-case?

Note: I can provide a trimmed down minimally reproducible example if needed.

[Sintrastes]

Doing a bit more digging, I found the explanation and workaround in #223, which I think is basically what I'm running into. I am wondering if there's any way we can make these sort of issues / limitations clearer in the docs somehow, as it took me a while to dig this up.

For what I want, I'm not sure if stream would end up being sufficient or not. Right now I am not even able to try it out since including a stream type in WIT seems to break the codegen in cargo component. (I will file a separate bug report for this)

What I'd really like here with the reactive-values interface is something that can interop with "traditional" FRP frameworks such as reflex (or the one I am building in Kotlin inspired by it -- yafrl) -- which are not asynchronous by default. Think of them instead like cells in a spreadsheet -- there is no asynchronous behavior -- everything updates synchronously on changes.

An important feature of these kinds of frameworks is determinism, so I am wary of WIT's asynchronous constructs as the docs currently say that there is at least some inherent non-determinism involved. So e.x. I am not sure if a stream backed by a yafrl Event in a Kotlin component would end up inheriting all of the good features (determinism) of Event or not, of if I'd need to devise an entirely different synchronous API for it, even though otherwise conceptually stream and Event are very similar (the former is an asynchronous reactive stream, the latter a synchronous one).

Edit: For traceability, I created the cargo-component bug report here: bytecodealliance/cargo-component#405

[Sintrastes]

Also, strangely enough, I seem to be having this issue even if I export both reactive-values and callbacks.

[lukewagner]

Unfortunately, the limitation where imports are not allowed to refer to exported types is somewhat important. The root issue is ensuring acyclicy in the types and definitions of components and avoiding the significant complexity increase that comes with recursive module types. Also, at an intuitive level: imported definitions need to have been created and already exist before the component that wants to import them is instantiated; exported resource types are created fresh for each component instance (which makes sense: that component instance likely contains the linear memory implementing its exported resource type) and thus exported resource types can't exist before their defining component is instantiated and thus can't be used as imports.

The callback use cases are import though and all the workarounds in WIT today are pretty painful. async, stream and future address many of these use cases, but not all, such as those discussed in #223. Thus, I'd very much like to add a callback type (with some careful consideration given to lifetimes to avoid the normal cyclic leaks you get in cross-language scenarios involving closures) before declaring a "1.0". I hope this should address all the remaining issues people have hit with this imports-can't-use-exports limitation.

[Sintrastes]

Interesting. Is this behavior mentioned anywhere in the spec? Maybe I missed it.

Also, at an intuitive level: imported definitions need to have been created and already exist before the component that wants to import them is instantiated; exported resource types are created fresh for each component instance (which makes sense: that component instance likely contains the linear memory implementing its exported resource type) and thus exported resource types can't exist before their defining component is instantiated and thus can't be used as imports.

When you put it that way it makes a lot of sense to me. I think maybe where the confusion came is was because I was trying to use resource types as higher-order functions -- so the reasoning went "Why can't I create a S32UpdateHandler as an exported type? After all, regardless of the language of the host / guest, it's always going to be some kind of function pointer, right? I should at least be able to convert between the imported and exported resources somehow."

This does sort of lead me to another question though: Are there valid use-cases where a component can have both "imported" and "exported" versions of the same resource type, or is the mere presence of this duplication a sign of some kind of semantic error that should be forbidden by the spec? (i.e. "Cannot use s32-update-handler as an exported type, as it is also being used as an imported type")

Maybe this is only an edge case that arises when you try to use a workaround to define a callback like I was attempting to do and so won't be as relevant of a concern in the future, but I will say from an end-user perspective, trying to wrap my head around why the bindings included both an imported and exported variant of the same type was quite confusing.

[Sintrastes]

Side-note -- and I'm not sure how useful this would be for anything but my own understanding -- but I wonder if the semantics of these imported / exported types could be clarified by means of some sort of graded category. You could model "imported" types as objects which exist in "previous" times, and enforce a morphism grading that ensures that definitions are stratified and that things in the "past" don't depend on things in the "future".

This is something I did an MS Thesis on in another life (before the siren song of industry called my name).

Sadly it has since been paywalled. I really need to upload another copy of it to ArXiV.

[syntactically]

Side-note -- and I'm not sure how useful this would be for anything but my own understanding -- but I wonder if the semantics of these imported / exported types could be clarified by means of some sort of graded category. You could model "imported" types as objects which exist in "previous" times, and enforce a morphism grading that ensures that definitions are stratified and that things in the "past" don't depend on things in the "future".

For what it's worth, on the syntactic side of things, resource type imports/exports turn into quantifiers on the component types (inspired by F-ing modules), and so the inability of imports to refer to exported types is enforced by scoping: a component type is essentially forall <imported resource types>. <imports> -> exists <exported resource types>. <exports>, so the exported resource types are not in scope for the imports. I haven't put much thought to categorical semantics, but the denotational-operational model should just be able to use a standard closing substitution.

[lukewagner]

Are there valid use-cases where a component can have both "imported" and "exported" versions of the same resource type, or is the mere presence of this duplication a sign of some kind of semantic error that should be forbidden by the spec?

Based on the F-ing Modules semantics @syntactically mentioned, there's no problem doing this and it's even actively done today (for virtualization purposes). E.g., a component can import a "base" resource type with label r (along with functions using the imported r) and then use these imports to define-and-export a new enriched resource type r (along with functions using the exported r), and it's even possible for a 3rd component to import both the base and enriched types and their operations (with the type system giving the base and enriched types distinct local type indices with no equality relationship).