rutabaga_gfx roadmap/discussion/collaboration

[gurchetansingh]

Hello!

With rutabaga finally deleted from crosvm and this repo being the new source of truth, this is a good time to discuss needs/desires from the project going forward.

This is motivated by a thread on fd.o where I proposed greater collaboration among virtio-gpu enthusiasts to reach our respective goals. A big part is sharing thinking and ideas, so the development process is truly open.  

Goals: First things first, the question I want to answer is: what is the what is rutabaga? rutabaga is a set of crates that enable cross-platform, open-source, Rust-based GPU/display para-virtualization.  It exists to serve its needs of whichever VMMs  use it -- most notably crosvm, QEMU, rust-vmm, others -- which then enable other larger projects (Android virtualization and Linux virtualization).

It has dependencies on gfxstream, virglrenderer and increasingly Rust-based crates housed in Mesa3D itself.

Open-source is the most important one, since it benefits all projects and makes sure the project is not reliant on a single corporation.  For example, although rutabaga has origins in crosvm, it does not require a corporate Google CLA like other Chromium-based projects (browser, CrOS, others).  This is because we've noticed corporate CLAs cause unnecessary hassles to partners and open-source developers alike, with little tangible benefit since everyone relies on FOSS projects anyways. For rutabaga, older code will keep the BSD-3-License for legacy reasons, but newer code is encouraged to use the MIT (functionally, the difference is negligible between BSD-3 and MIT. MIT is just simpler).

Cross-platform is another important one, since for example 70% of the market and developer base still uses Windows.  We are also interested in running ANY guest OS in a VM with accelerated graphics -- including Windows, Haiku, Fuchsia.  Cross-platform desires make rutabaga goals quite ambitious, since it showcases the true power of VIRTIO.

What about HW-accelerated virtualization and SR-IOV?  When SR-IOV and GPU assignment is available from major vendors on major guest OSes, I'll be the first to sign-up. Until then, VIRTIO is the best we have.

Communication: just file issues on Github or fd.o gitlab, and tag relevant devs.  Chat rooms would be useful, but everyone probably uses something different (irc/discord/google chat), maybe we can agree to congregate on an existing one??    

 With that introduction out of the one, here are potential work items we can concretely collaborate on:

Magma

We would like to land Magma.  There are multiple possibilities in this space.  Right, there's some support for (msm, i915/Xe, amdgpu, d3dkmt), plans for Fuchsia backends under ONE library interface/protocol.  The memory management is looking good, but we still need to cover all use cases. Command buffer submission (including VM_BIND varieties) needs to be taken care of.  Use extensible structs if you need to and focus on the Rust API. FFI/protocol/encode/decode should be autogen'ed.

apigen-xml

We would like to generate a client <--> server protocol, and a host <--> guest protocol too. Hand-writing is a pain, so we started the apigen-xml project to auto generate these protocols. Many concepts are modern-day reinterpretations of gfxstream.  This is useful for both Magma and other protocols.  This is exactly what virtio-gpu is: a way to route to guest/host. 

ntx-rs

Some have suggested writing the nctx in Rust rather than waiting for Magma.  Make sense, Mesa already supports nctx.  But I suggest use of Rutabaga, since many people like to pair nctx/cross-domain and we can possibly even implement nctx-on-Magma, helping both the cross-platform and Linux-specific initiatives.

upstream GBM instead of minigbm

With the ChromeOS LTS branch created, it is no longer necessary to use minigbm for rutabaga gralloc. This is useful for sharing cross-domain Wayland passthrough directly with the display without memfd. Will work with linear dma-bufs. This is also useful for Android guests when enabling the next-generation cross-domain allocator.

Package virtwl-proxy in Debian and fence passing?

So Sommelier is essentially dead.  Nobody works on it and it's buggy. It seems the most maintained solution is virtwl-proxy. Also interested in X11 compatibility issues and possible solutions.

Also some people were interested in fence-passing to the host compositor. We should get that working, eventually.

gfxstream guest + windows Kumquat + gfxstream backend

So I got it building. To land it, it could be better if we actually got it building and running. Need to finish stubs for Windows in mesa3d_util.

meson improvements to support rlib versioning

rutabaga wants to depend on Mesa3D crates.  But right now Meson requires using subprojects to integrate Rust crates.  It would be better to use for some package-config based handsake to exist, so one Rust project can discover another. See discussion here. The meson devs are open to it, and it would help ALL Rust crates.

Anyways, these are just ideas we can POTENTIALLY collaborate on, possibly via Github projects. If you have other ideas, feel free to share :-)

CC: @valpackett @DemiMarie @alyssais @talex5 @digetx @robclark @slp @dorindabassey @mtjhrc @stsquad @epilys @lygstate @elmarco

[epilys]

meson improvements to support rlib versioning

rutabaga wants to depend on Mesa3D crates. But right now Meson requires using subprojects to integrate Rust crates. It would be better to use for some package-config based handsake to exist, so one Rust project can discover another. See mesonbuild/meson#14639 (comment). The meson devs are open to it, and it would help ALL Rust crates.

This is also something we want for QEMU. Cc @bonzini

[alyssais]

This is all great stuff, thank you!

Package virtwl-proxy in Debian and fence passing?

So Sommelier is essentially dead. Nobody works on it and it's buggy. It seems the most maintained solution is virtwl-proxy. Also interested in X11 compatibility issues and possible solutions.

Also some people were interested in fence-passing to the host compositor. We should get that working, eventually.

At the moment wayland-proxy-virtwl development seems a bit fragmented. talex5/wayland-proxy-virtwl#91 has been sitting unreviewed for a long time (I suspect the issue is just that that PR is very big, and has a lot of stuff in it that should be split into smaller PRs), and the recent developments from @valpackett haven't been offered to upstream so far. It would be good to hear from everyone working on it what we could do to unstick things there, and get everybody working with a common source where changes don't end up stuck out of tree for a long time. cc @talex5 @DemiMarie

Anyways, these are just ideas we can POTENTIALLY collaborate on, possibly via Github projects. If you have other ideas, feel free to share :-)

Something I'd love to see is a vhost-user GPU device in rust-vmm, that uses the crosvm-style of having everything in the backend, rather than the QEMU vhost-user-gpu way where the VMM process is also responsible for drawing host windows etc. as a historical artifact of having already had the code for doing so. Combined with a generic vhost-user device (as implemented in QEMU for testing purposes and proposed for Cloud Hypervisor in cloud-hypervisor/cloud-hypervisor#7221, this will enable using virtio-gpu devices with VMMs without requiring the VMM to have any GPU-specific knowledge. Currently, we use the crosvm vhost-user-gpu device, but crosvm isn't ideal as an upstream for the reasons you've identified. For a long time, the path to this has been impeded by crosvm using non-standard vhost-user messages for its GPU device, but equivalent messages have now finally been basically accepted for the standard, and should hopefully be available in the rust-vmm vhost crate soon: rust-vmm/vhost#251 (comment)

Communication: just file issues on Github or fd.o gitlab, and tag relevant devs. Chat rooms would be useful, but everyone probably uses something different (irc/discord/google chat), maybe we can agree to congregate on an existing one??

Mesa/DRI, virglrenderer and QEMU use OFTC IRC, and crosvm, Spectrum, clan and Qubes OS use Matrix. With a single Matrix account it's therefore possible to participate in the chats for all these projects, so to me Matrix or OFTC IRC would be the natural choice for a new project working at the intersection of these. Proprietary chat platforms tend to see quite a lot of pushback when used for desktop free software projects (much more so than proprietary git forges, as far as I can see — perhaps because the proprietary chat platforms are less entrenched than GitHub).

[elmarco]

minor comment:

rather than the QEMU vhost-user-gpu way where the VMM process is also responsible for drawing host windows

This isn't true if you use -display dbus (or spice/vnc). However, the display updates are dispatched through QEMU atm. The qemu dbus display protocol was designed so that the display updates could bypass QEMU roundtrip eventually.

[bonzini]

It would be better to use for some package-config based handsake to exist, so one Rust project can discover another

The problem is that installing rlibs does not work (some may say cannot work). The solution could be to add registry support to Meson and use a private registry, but other solutions have been proposed within Mesa and they could span the whole spectrum:

• make periodic releases of crates on every Mesa release
• make independent releases of the crates with their own semver-based version numbers, but keep them in the Mesa repository for convenience; rutabaga consumes them via crates.io
• split the crates out of Mesa, with a completely independent release schedule, and add them as subprojects to both Mesa and rutabaga

[slp]

It's great to see that rutabaga_gfx has become an independent. We'd like to remove the vendorized crate we have in libkrun and consume this one, but that would require upstreaming changes such as X11 support for cross-domain. Would that be accepted here?

Also, I'm curious about gfxstream future plans. I know it works fine with Android, but are there any plans to extend both the host and the guest sides (in mesa) to support rendering (through cross-domain or with a complete VK_surface emulation on top of drm) on Linux? It'd be really nice to be able to rely on gfxstream for Vulkan rendering on the guest, specially now that we have compositors with vulkan backends.

[gurchetansingh]

It would be good to hear from everyone working on it what we could do to unstick things there, and get everybody working with a common source where changes don't end up stuck out of tree for a long time.

+1, that CL has the potential to clear up the Sommelier issue.

The problem is that installing rlibs does not work (some may say cannot work).

I'll try to come up with a more rigorous proposal and post as a RFC on mesonbuild. I do think with proper pkg-config metadata, it could work.

Would that be accepted here?

Yes, X11 channel types are fine.

We can probably delete it anyways after virtwl-proxies are good enough (some distros are phasing out X11 support). For virtualization, I think the issues are with Sommelier, not necessarily with X11/Wayland.

I know it works fine with Android, but are there any plans to extend both the host and the guest sides (in mesa) to support rendering (through cross-domain or with a complete VK_surface emulation on top of drm) on Linux?

I think the next step for Linux guests/Linux host is to verify cross-domain pass-through + gfxstream, but that probably depends on the aforementation virtwl proxy fixes. For Linux guests/MacOS, there are Zink issues on MacOS (only gets GL1.2 on KosmicKrisp independent of virtualization), so that needs to be chased down. For Zink/Windows, that's somewhat buggy last time depending on the driver (I tested on NV), again independent of virtualization. So I think there needs to be some work in the Mesa Zink layer before we can reliably go VK only: Linux guests/Linux host is doable now though.

[valpackett]

re: virtwl-proxy: sure, I'll start upstreaming what I can. upd: oop, if none of the dmabuf PR has landed, that's tough..

equivalent messages have now finally been basically accepted for the standard, and should hopefully be available in the rust-vmm vhost crate soon

nice!

writing the nctx in Rust rather than waiting for Magma. Make sense, Mesa already supports nctx. But I suggest use of Rutabaga

Of course it would be a rendering backend for Rutabaga, as an alternative to virglrenderer.

X11 channel types are fine

IMO the dream is to have a generic channel type that could forward any unix socket based protocol that fd-passes dmabufs, memfds, eventfds, pipes, and sockets. Get rid of the need for sommelier/virtwl-proxy, and have a protocol-unaware universal proxy that could do Wayland, X11, PipeWire, D-Bus, PipeWire-passed-over-D-Bus (for portals), …

Looks like the biggest obstacle for that is the shared memory situation (copying from guest memfd to memory allocated on the host). Sounds like a very tricky problem to solve in a non-hacky way. None of the protocols were written with cross-domain in mind so local shm (memfd) being allocated by the client is very common.

[gurchetansingh]

IMO the dream is to have a generic channel type that could forward any unix socket based protocol that fd-passes dmabufs, memfds, eventfds, pipes, and sockets.

Some people are working on it: look up virtio-msg. BUT that's going to take forever to upstream to the Linux kernel and get to users, a very tough challenge technically.

EDIT: realized you just wanted generic channel types + generic proxy, not necessarily virtio-msg. If you can make that work, sgtm.

[valpackett]

huh, that virtio-msg I've been vaguely aware of but it's a new underlying transport for virtio, not a device over virtio..

[bonzini]

I'll try to come up with a more rigorous proposal and post as a RFC on mesonbuild. I do think with proper pkg-config metadata, it could work.

The problem is more the incompatibility across rustc versions than the structure of the metadata.

[stsquad]

huh, that virtio-msg I've been vaguely aware of but it's a new underlying transport for virtio, not a device over virtio..

It's probably slightly muddled as one of the first things we've used virtio-msg for is carrying virtio-vsock and looking at ways to transition memory buffers between domains (in this case a Linux guest and Trusty guest). This does involve teaching the vsock backend about how it can translate those memory references though.

[dorindabassey]

I wanted to share our experience with vhost-device-gpu that might be relevant to this collaboration discussion.
We're currently refactoring vhost-device-gpu (see PR rust-vmm/vhost-device#890 ) to use virglrenderer-rs directly instead of going through rutabaga_gfx. This is a significant architectural change that simplifies our stack. while the gfxstream backend still uses rutabaga_gfx.
For virglrenderer, I found out that:

1. We didn't need the additional abstractions that rutabaga provides
2. direct bindings reduced our dependency chain
3. Backend-specific features are more readily accessible
4. virglrenderer-rs provides a more focused, Rust-idiomatic API

For gfxstream, rutabaga_gfx continues to work well for our needs.

Now that Both rutabaga_gfx and virglrenderer-rs provide Rust access to libvirglrenderer, could rutabaga_gfx use virglrenderer-rs internally instead of wrapping the C library directly? This could reduce duplicate FFI binding code across the ecosystem.

[valpackett]

@dorindabassey uhh.. is it just me or does that sound like you're throwing away cross-domain support? Which is by far the most valuable part of rutabaga, and something that we definitely require from the Qubes side (and same for Spectrum).

[dorindabassey]

@dorindabassey uhh.. is it just me or does that sound like you're throwing away cross-domain support? Which is by far the most valuable part of rutabaga, and something that we definitely require from the Qubes side (and same for Spectrum).

Hi @valpackett , To clarify - we're not throwing away cross-domain support. Our refactoring is specifically for vhost-device-gpu, which doesn't currently use cross-domain features.
What I was asking about is could rutabaga_gfx's internal virglrenderer FFI layer use virglrenderer-rs under the hood, while still providing all the cross-domain and other features on top? This would just reduce duplicate FFI binding maintenance, not remove any functionality.

[gurchetansingh]

We're currently refactoring vhost-device-gpu (see PR rust-vmm/vhost-device#890 ) to use virglrenderer-rs directly instead of going through rutabaga_gfx. This is a significant architectural change that simplifies our stack. while the gfxstream backend still uses rutabaga_gfx.

In that CL, I do recommend the "gfxstream" backend be renamed "rutabaga" backend.  This is because we do want to use Wayland passthrough (Linux guests) and cross-domain minigbm backend (Android guests) with gfxstream.  Also Magma/nctx-rs backends: so there might be a future where gfxstream isn't involved at all for rendering.

Both rutabaga_gfx and virglrenderer-rs provide Rust access to libvirglrenderer, could rutabaga_gfx use virglrenderer-rs internally instead of wrapping the C library directly.

We do like to minimize dependencies to the extent possible.  Internally, there's 3 downstreams of rutabaga_gfx, and one would have to go through an approval process with each downstream to get new dependencies in.  That's why with the Mesa3D deps, they are put into "third_party".  Given this cost/benefit tradeoff, I think it makes sense to avoid the additional dep on virglrenderer-rs, since it uses many constructs already in rutabaga (VirglHandle versus RutabagaHandle/MesaHandle, for example). 

The virgl bindings inside rutabaga can be improved, for example by deleting the python generate scripts and just doing bindgen.