Shader to directly render to screen pixels

[alice-i-cecile]

What problem does this solve or what need does it fill?

"Pixel-perfect rendering" is a hotly requested feature for retro games, or for more unusual use cases where very-direct control is desired.

What solution would you like?

We create a shader that does this, which is fed out of a resource that stores a screen_width x screen_height array of pixel data.

What alternative(s) have you considered?

None, but I don't know graphics well.

Additional context

bevy_retro by @zicklag has this functionality, but the license is very much incompatible with Bevy itself. Guidance would be lovely if you're willing to chime in <3

[zicklag]

Sure, I can give my thoughts. 🙂

Bevy Retro actually doesn't give you a way to directly write to the screen. It works similarly to how Bevy works, actually, and whenever you create entities with Handle<Image>, Sprite, WorldPosition, and Visible components, it will make sure they get rendered on the screen by the rendering system. The only difference is the way that it handles positions ( integers, not floats ) and the camera and framebuffer resolution so that the pixels always come out perfectly clean.

I think that having to send pixels directly to the screen through a shader like you're mentioning here may not be the nicest way to handle it from a developer experience perspective. I think that that can actually already be done in vanilla bevy today by manually creating Texture assets and passing that into a ColorMaterial and a SpriteBundle. You could just feed the data of a Texture with the bytes from an image created using the image crate and I think you could do that without any changes to Bevy.

Again, though, I'm not sure that's what most people want to do. Updating all those pixels manually is probably not what a typical retro game progammer wants to do because they will want to specify sprites and positions just like they would in normal Bevy, which is the approach Bevy Retro takes.

If you wanted to do retro rendering without doing a different renderer like Bevy Retro does, a thought that comes to mind would be to create a special RetroSurface component that would be like a "screen" of sorts that is rendered using a strategy similar to how Bevy Retro does it: you would create sprites as a child of the RetroSurface and that would cause them to be rendered, not by the normal Bevy camera, but on the flat RetroSurface. Then you could place the RetroSurface anywhere in the Bevy scene and render it with the Bevy camera. Including possibly just putting the RetroSurface full-screen right in front of the camera.

[alice-i-cecile]

Awesome, that's incredibly useful. I agree with your concerns on developer experience here too.

I'll defer to the rendering experts on the best approach here though :p

[Frizi]

This might be only semi-related, but i guess it's worth throwing it out there. When speaking about rendering low-resolution pixel-art on modern displays, it is important to use proper texture sampling that takes the "squareness of the pixel" into account and antialiases it properly at texel edges.
See this shadertoy https://www.shadertoy.com/view/MlB3D3
explanation and math breakdown on handmade hero: https://hero.handmade.network/episode/chat/chat018/

I believe it is very important to have that antialiased pixel-art rendering in retro renderer as a default, instead of the usual nearest-neighbor "filtering".

[zicklag]

Ah... So there is a way to handle that. That's not something I do in Bevy Retro. I'll have to look into those links. It didn't look bad or anything, but I wondered if there was any way to handle trying to keep pixels perfect squares at different resolutions.

[CptPotato]

Just thought I'd chime in and share my experiences on how this can be implemented since I've done some work on this myself (unrelated to bevy).

Generally there's two approaches for this kind of 2d low-res rendering:

1. The first one is to render the scene to a low resolution buffer (offscreen rendertarget) where the pixels translate 1:1. Then, in a post process step the buffer is upscaled and drawn to the screen. This is the solution I personally prefer because everything adheres to the low resolution pixel grid. It gives a very consistent look (nothing can "break" the alignment), it's fast and generally doesn't cause much issues while easy to implement. The downside is that subpixel movement is not possible, which means slow moving objects can look a bit jitter-y. (Smooth camera movement can be worked around with some trickery in the upscaling step).

2. The alternative is to draw at full resolution directly and apply a scaling factor to basically everything. This one makes it a bit harder to archieve a consistent, good look. Objects that have a different scale or ones that are rotated stick out like a sore thumb (to me at least). One benefit, though, is that objects can be placed "inbetween pixels" which allows sprites to move a bit more smoothly in some cases.
   The only game I can think of which pulled this one off corretly is Shovel Knight: All backgrounds and stationary sprites are prefectly aligned while dynamic sprites can move freely. Rotations are also locked to 90° steps which never breaks the pixel grid pattern and scaling is never used.

As for the shading, the first option would simply use nearest neighbour sampling for drawing each sprite (or linear filtering + rounding the sprite/vertex positions, both have their own benefits). The upscaling step (which may use non-integer scales) can be more sophisticated as outlined by @Frizi (also see ¹ below). Other post processing effects can be done before or after upscaling, depending on the type of effect.

Option 2 requires the scaling to be done directly when drawing each sprite making that part a bit more tricky. When combined with the smooth filtering mentioned above, every sprite is effectively "transparent" at the edges and needs depth sorting as a result. It's also a bit slower but I think that's not really an issue in these type of games.

---

This might be only semi-related, but i guess it's worth throwing it out there. When speaking about rendering low-resolution pixel-art on modern displays, it is important to use proper texture sampling that takes the "squareness of the pixel" into account and antialiases it properly at texel edges.
See this shadertoy https://www.shadertoy.com/view/MlB3D3
explanation and math breakdown on handmade hero: https://hero.handmade.network/episode/chat/chat018/

I believe it is very important to have that antialiased pixel-art rendering in retro renderer as a default, instead of the usual nearest-neighbor "filtering".

¹ I did some work on this aswell 😅. One thing to note is that the solution from the shadertoy link can interfere with mipmapping (since it remaps the uv coordinates). This can be worked around by calculating the derivatives manually and using textureLod()/textureGrad() for sampling. Then again, I don't think mipmapping is required unless you use this type of filtering in 3D.

My shader code for this is available over here - for anyone working on this, feel free to salvage what you need. For 2D I'd recommend the function that doesn't use dFdx()/dFdy().

(oof, sorry for the wall of text)

[zicklag]

Just wanted to comment with my recent experiences here, that might shed some light on what features Bevy actually needs to be good at "pixel perfect rendering".

For Bevy Retrograde we recently decided to switch to using Bevy's built-in renderer instead of our own. The reasoning was mostly for portability, but also influencing the decision was the realization that most of the design decisions initially made in Bevy Retrograde were fundamentally unhelpful ( at least in my experience so far ). For instance:

• Integer coordinates: These are not helpful because it makes it much harder to move a sprite slower than 1 pixel-per-frame. Because you literaly can't move less than one pixel at a time if you want to move slower than one pixel at a time, you have to move every-other frame, or every-x-frames, etc. This makes the math for moving things much trickier than what it is with floating points where you can just let speed = 0.3; and then multiply the speed by your movement vector.
• Low res framebuffer: This is what @CptPotato mentioned above as rendering option 1, which Bevy Retrograde initially used, but then moved away from. While rendering to a low-res framebuffer is great for the super-authentic retro feel because it's, again, impossible to break the pixel alignment, me and my team ended up deciding that it was better for the player to have smooth movement and precise gameplay. That is obviously a game-dependent decision, but it allows the most choices on the part of the game designer, so I think it's probably the better option for Bevy.

I haven't gotten into the advanced filtering to make the pixels look as correct as possible yet, but now with Bevy Retrograde moving to Bevy's renderer, it essentially just became a Bevy plugin pack providing map loading, etc. Nothing special on top of Bevy.

What would be necessary to make Bevy more pixel-perfect? I'm not positive on the answer, but it seems like it's mostly a matter of how we handle the pixel filtering and the camera, and maybe also a way to make it eaiser for people to snap sprites to the pixel grid.

Camera & Filtering

For Bevy Retrograde our thoughts were to have three different pixel filtering modes ( see katharostech/bevy_retrograde#51 ):

Mode	Crisp Edges	Perfect Squares	Any Zoom Level	Doesn't Require Camera Borders
Crisp Scalable	✔️	❌	✔️	✔️
Smooth Scalable	❌	✔️	✔️	✔️
Crisp Fixed Scale	✔️	✔️	❌	❌

"Snapping" Coordnates

For people who want moving objects to snap to the pixel grid, we could create a new GridTransform or something similar, that users will use instead of Transform, that has all of the same fields as Transform. There will be a GridTransformSync system that will run before the transform propagation system that will simply copy the GridTransform to the entities Transform, but it will round the position, rotation, and scale, to enforce the pixel alignment.

This will make it easy for anybody to pixel-snapped objects, but still with floating point coordinates, without any extra work.

---

Anyway, that's my current brain-dump on pixel-perfect rendering in Bevy. I might have more thoughts later after Bevy Retrograde is fully migrated to Bevy's renderer. 😀

[gbip]

Hey, I'd like to add my use-case to this discussion 😄

I'm trying to perform agent simulation just like in this video and direct access to the framebuffer would at least allow me to first write the code in a sequential way before moving to a shader :)

[brianguertin]

Hey, I just want to chime in and mention another use case for the above mentioned strategy: "render the scene to a low resolution buffer (offscreen rendertarget) where the pixels translate 1:1. Then, in a post process step the buffer is upscaled and drawn to the screen."

Performance.

In some cases (windowed mode, web browser) the user may have a very large physical resolution (e.g. 4k, Hi-dpi, etc.) that I can't control. So, pixel art or not, I might want to render at a lower-than-native resolution.

If my pixel art game was based on 240p and the user has a 4k screen. There are more than just two options (render at 240p or render at 4k) I might also want to render at 480p or 720p or 1080p to get varying degrees of sub-pixel movement mentioned above, but still not render at the full native resolution for performance reasons.

On the web, this is actually pretty easy by setting the canvas width/height to the desired pixel resolution, than using CSS to stretch it to fill the viewport. But I'm not sure how to accomplish this within Bevy itself.

[alice-i-cecile]

This ecosystem crate looks like a great place to start: https://github.com/drakmaniso/bevy_pixel_camera