User provided mesh binding / per-instance mesh data

[tychedelia]

What problem does this solve or what need does it fill?

A common pattern in creative coding is to use the instance index in a shader in order to sample from something like a texture or read from a SSBO. For example, the shader might use one texture in order to offset the position of the mesh and a second texture in order to determine the vertex color.

Currently, batch rendered entities (i.e. entities that share the same mesh and material) are instanced in a random order relative to how they were spawned. This is due a variety of causes:

• Instability in ECS queries.
• GPU preprocessing compute shader.
• Order of rendering bins.

Consequently, because the instance index is "random", it's not possible in a custom shader to know where to sample from an input texture.

See the following pseudo UV maps to see what this looks like with different rendering features enabled:

What solution would you like?

The user should be able to provide a custom binding representing per-instance mesh data, i.e. a SSBO that they can index into with the instance index to retrieve additional data required for their vertex shader.

The difficult part here is that in order to write into the user data buffer, it needs to be plumbed everywhere the MeshUniform is created, including our new preprocessing compute shader. In order not to pollute all our rendering code, this means we need some kind of generic plugin for instance data that gets type erased in our rendering code so that it can optionally injected everywhere it needs to go.

What alternative(s) have you considered?

We could, alternatively, track some kind of total "spawn order" and add this as a new index to MeshUniform. The user could then bind their buffer to their material and index into that instead. This has a few disadvantages:

• Adding a u32 to every mesh uniform that is likely unnecessary for most users.
• Requiring all possible data to be input to the material even though many of the instances may be culled.
• Difficult to track when the spawn order should be when meshes are despawned and respawned.

Additional context

There's prior art here I'm happy to reference in terms of how this typically works in creative coding applications.

[tychedelia]

@pcwalton Tagging you because you suggested this as a solution, and it's a great idea that would really help advance using bevy for art :). Also, in reviewing the feasibility of implementing this, the most gnarly part passes through the preprocessing compute shader bits which you wrote.

My approach is basically going to be this:

1. Add a new UserMeshInstancePlugin<T> that the user can add once per application, where T ensures proper alignment, etc.
2. In order not to pollute all the rendering code with a new type parameter, this is going to be type erased as RawBufferVec<u32>.
3. Pass this into the compute shader etc as an optional buffer that enables a shaderdef.
4. In the compute shader, bind input, output, and a push constant (or whatever) that indicates the size of the data.
5. After culling, write input to output at the same time as the mesh data.
6. In the user's vertex shader, they are thus able to do @group(0) @binding(6) var<storage> my_data: array<MyData>; and use the instance index to look up their per-mesh data.

Does this general approach make sense? Do you have any other more general advice for how to approach this?

[tychedelia]

One of the problems here is that we have a lot of nice wrappers around things like gpu buffers, creating an "erased" variant for each one is a bit unfortunate. If we permitted the pollution of lots of the rendering code with a user supplied generic type, it would significantly help the implementation, but I'm not sure how to implement the default case where there is no user supplied mesh instance data without specialization (which would still make the code much uglier).