Add parallax mapping to bevy PBR (#5928)

# Objective

Add a [parallax mapping] shader to bevy. Please note that
this is a 3d technique, NOT a 2d sidescroller feature.

## Solution

- Add related fields to `StandardMaterial`
- update the pbr shader
- Add an example taking advantage of parallax mapping

A pre-existing implementation exists at:
https://github.com/nicopap/bevy_mod_paramap/

The implementation is derived from:

https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28

Further discussion on literature is found in the `bevy_mod_paramap`
README.

### Limitations

- The mesh silhouette isn't affected by the depth map.
- The depth of the pixel does not reflect its visual position, resulting
  in artifacts for depth-dependent features such as fog or SSAO
- GLTF does not define a height map texture, so somehow the user will
  always need to work around this limitation, though [an extension is in
  the works][gltf]

### Future work

- It's possible to update the depth in the depth buffer to follow the
  parallaxed texture. This would enable interop with depth-based
  visual effects, it also allows `discard`ing pixels of materials when
  computed depth is higher than the one in depth buffer
- Cheap lower quality single-sample method using [offset limiting]
- Add distance fading, to disable parallaxing (relatively expensive)
  on distant objects
- GLTF extension to allow defining height maps. Or a workaround
  implemented through a blender plugin to the GLTF exporter that
  uses the `extras` field to add height map.
- [Quadratic surface vertex attributes][oliveira_3] to enable parallax
  mapping on bending surfaces and allow clean silhouetting.
- noise based sampling, to limit the pancake artifacts.
- Cone mapping ([GPU gems], [Simcity (2013)][simcity]). Requires
  preprocessing, increase depth map size, reduces sample count greatly.
- [Quadtree parallax mapping][qpm] (also requires preprocessing)
- Self-shadowing of parallax-mapped surfaces by modifying the shadow map
- Generate depth map from normal map [link to slides], [blender
question]


https://user-images.githubusercontent.com/26321040/223563792-dffcc6ab-70e8-4ff9-90d1-b36c338695ad.mp4

[blender question]:
https://blender.stackexchange.com/questions/89278/how-to-get-a-smooth-curvature-map-from-a-normal-map
[link to slides]:
https://developer.download.nvidia.com/assets/gamedev/docs/nmap2displacement.pdf
[oliveira_3]:
https://www.inf.ufrgs.br/~oliveira/pubs_files/Oliveira_Policarpo_RP-351_Jan_2005.pdf
[GPU gems]:
https://developer.nvidia.com/gpugems/gpugems3/part-iii-rendering/chapter-18-relaxed-cone-stepping-relief-mapping
[simcity]:
https://community.simtropolis.com/omnibus/other-games/building-and-rendering-simcity-2013-r247/
[offset limiting]:
https://raw.githubusercontent.com/marcusstenbeck/tncg14-parallax-mapping/master/documents/Parallax%20Mapping%20with%20Offset%20Limiting%20-%20A%20Per-Pixel%20Approximation%20of%20Uneven%20Surfaces.pdf
[gltf]: KhronosGroup/glTF#2196
[qpm]:
https://www.gamedevs.org/uploads/quadtree-displacement-mapping-with-height-blending.pdf

---

## Changelog

- Add a `depth_map` field to the `StandardMaterial`, it is a grayscale
  image where white represents bottom and black the top. If `depth_map`
  is set, bevy's pbr shader will use it to do [parallax mapping] to
  give an increased feel of depth to the material. This is similar to a
  displacement map, but with infinite precision at fairly low cost.
- The fields `parallax_mapping_method`, `parallax_depth_scale` and
  `max_parallax_layer_count` allow finer grained control over the
  behavior of the parallax shader.
- Add the `parallax_mapping` example to show off the effect.

[parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping

---------

Co-authored-by: Robert Swain <[email protected]>

Author: nicopap

Cargo.toml

@@ -561,6 +561,16 @@ description = "Demonstrates use of Physically Based Rendering (PBR) properties"
 category = "3D Rendering"
 wasm = true
 
+[[example]]
+name = "parallax_mapping"
+path = "examples/3d/parallax_mapping.rs"
+
+[package.metadata.example.parallax_mapping]
+name = "Parallax Mapping"
+description = "Demonstrates use of a normal map and depth map for parallax mapping"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "render_to_texture"
 path = "examples/3d/render_to_texture.rs"

assets/textures/parallax_example/cube_color.png

@@ -8,6 +8,7 @@ mod environment_map;
 mod fog;
 mod light;
 mod material;
+mod parallax;
 mod pbr_material;
 mod prepass;
 mod render;
@@ -18,6 +19,7 @@ pub use environment_map::EnvironmentMapLight;
 pub use fog::*;
 pub use light::*;
 pub use material::*;
+pub use parallax::*;
 pub use pbr_material::*;
 pub use prepass::*;
 pub use render::*;
@@ -34,6 +36,7 @@ pub mod prelude {
         fog::{FogFalloff, FogSettings},
         light::{AmbientLight, DirectionalLight, PointLight, SpotLight},
         material::{Material, MaterialPlugin},
+        parallax::ParallaxMappingMethod,
         pbr_material::StandardMaterial,
     };
 }
@@ -82,6 +85,8 @@ pub const PBR_FUNCTIONS_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 16550102964439850292);
 pub const PBR_AMBIENT_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 2441520459096337034);
+pub const PARALLAX_MAPPING_SHADER_HANDLE: HandleUntyped =
+    HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 17035894873630133905);
 
 /// Sets up the entire PBR infrastructure of bevy.
 pub struct PbrPlugin {
@@ -150,6 +155,12 @@ impl Plugin for PbrPlugin {
             "render/pbr_prepass.wgsl",
             Shader::from_wgsl
         );
+        load_internal_asset!(
+            app,
+            PARALLAX_MAPPING_SHADER_HANDLE,
+            "render/parallax_mapping.wgsl",
+            Shader::from_wgsl
+        );
 
         app.register_asset_reflect::<StandardMaterial>()
             .register_type::<AmbientLight>()

assets/textures/parallax_example/cube_depth.png

@@ -0,0 +1,45 @@
+use bevy_reflect::{FromReflect, Reflect};
+
+/// The [parallax mapping] method to use to compute depth based on the
+/// material's [`depth_map`].
+///
+/// Parallax Mapping uses a depth map texture to give the illusion of depth
+/// variation on a mesh surface that is geometrically flat.
+///
+/// See the `parallax_mapping.wgsl` shader code for implementation details
+/// and explanation of the methods used.
+///
+/// [`depth_map`]: crate::StandardMaterial::depth_map
+/// [parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Default, Reflect, FromReflect)]
+pub enum ParallaxMappingMethod {
+    /// A simple linear interpolation, using a single texture sample.
+    ///
+    /// This method is named "Parallax Occlusion Mapping".
+    ///
+    /// Unlike [`ParallaxMappingMethod::Relief`], only requires a single lookup,
+    /// but may skip small details and result in writhing material artifacts.
+    #[default]
+    Occlusion,
+    /// Discovers the best depth value based on binary search.
+    ///
+    /// Each iteration incurs a texture sample.
+    /// The result has fewer visual artifacts than [`ParallaxMappingMethod::Occlusion`].
+    ///
+    /// This method is named "Relief Mapping".
+    Relief {
+        /// How many additional steps to use at most to find the depth value.
+        max_steps: u32,
+    },
+}
+impl ParallaxMappingMethod {
+    /// [`ParallaxMappingMethod::Relief`] with a 5 steps, a reasonable default.
+    pub const DEFAULT_RELIEF_MAPPING: Self = ParallaxMappingMethod::Relief { max_steps: 5 };
+
+    pub(crate) fn max_steps(&self) -> u32 {
+        match self {
+            ParallaxMappingMethod::Occlusion => 0,
+            ParallaxMappingMethod::Relief { max_steps } => *max_steps,
+        }
+    }
+}

assets/textures/parallax_example/cube_normal.png

@@ -1,6 +1,6 @@
 use crate::{
-    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, PBR_PREPASS_SHADER_HANDLE,
-    PBR_SHADER_HANDLE,
+    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, ParallaxMappingMethod,
+    PBR_PREPASS_SHADER_HANDLE, PBR_SHADER_HANDLE,
 };
 use bevy_asset::Handle;
 use bevy_math::Vec4;
@@ -231,6 +231,84 @@ pub struct StandardMaterial {
     ///
     /// [z-fighting]: https://en.wikipedia.org/wiki/Z-fighting
     pub depth_bias: f32,
+
+    /// The depth map used for [parallax mapping].
+    ///
+    /// It is a greyscale image where white represents bottom and black the top.
+    /// If this field is set, bevy will apply [parallax mapping].
+    /// Parallax mapping, unlike simple normal maps, will move the texture
+    /// coordinate according to the current perspective,
+    /// giving actual depth to the texture.
+    ///
+    /// The visual result is similar to a displacement map,
+    /// but does not require additional geometry.
+    ///
+    /// Use the [`parallax_depth_scale`] field to control the depth of the parallax.
+    ///
+    /// ## Limitations
+    ///
+    /// - It will look weird on bent/non-planar surfaces.
+    /// - The depth of the pixel does not reflect its visual position, resulting
+    ///   in artifacts for depth-dependent features such as fog or SSAO.
+    /// - For the same reason, the the geometry silhouette will always be
+    ///   the one of the actual geometry, not the parallaxed version, resulting
+    ///   in awkward looks on intersecting parallaxed surfaces.
+    ///
+    /// ## Performance
+    ///
+    /// Parallax mapping requires multiple texture lookups, proportional to
+    /// [`max_parallax_layer_count`], which might be costly.
+    ///
+    /// Use the [`parallax_mapping_method`] and [`max_parallax_layer_count`] fields
+    /// to tweak the shader, trading graphical quality for performance.
+    ///
+    /// To improve performance, set your `depth_map`'s [`Image::sampler_descriptor`]
+    /// filter mode to `FilterMode::Nearest`, as [this paper] indicates, it improves
+    /// performance a bit.
+    ///
+    /// To reduce artifacts, avoid steep changes in depth, blurring the depth
+    /// map helps with this.
+    ///
+    /// Larger depth maps haves a disproportionate performance impact.
+    ///
+    /// [this paper]: https://www.diva-portal.org/smash/get/diva2:831762/FULLTEXT01.pdf
+    /// [parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
+    /// [`parallax_depth_scale`]: StandardMaterial::parallax_depth_scale
+    /// [`parallax_mapping_method`]: StandardMaterial::parallax_mapping_method
+    /// [`max_parallax_layer_count`]: StandardMaterial::max_parallax_layer_count
+    #[texture(11)]
+    #[sampler(12)]
+    pub depth_map: Option<Handle<Image>>,
+
+    /// How deep the offset introduced by the depth map should be.
+    ///
+    /// Default is `0.1`, anything over that value may look distorted.
+    /// Lower values lessen the effect.
+    ///
+    /// The depth is relative to texture size. This means that if your texture
+    /// occupies a surface of `1` world unit, and `parallax_depth_scale` is `0.1`, then
+    /// the in-world depth will be of `0.1` world units.
+    /// If the texture stretches for `10` world units, then the final depth
+    /// will be of `1` world unit.
+    pub parallax_depth_scale: f32,
+
+    /// Which parallax mapping method to use.
+    ///
+    /// We recommend that all objects use the same [`ParallaxMappingMethod`], to avoid
+    /// duplicating and running two shaders.
+    pub parallax_mapping_method: ParallaxMappingMethod,
+
+    /// In how many layers to split the depth maps for parallax mapping.
+    ///
+    /// If you are seeing jaggy edges, increase this value.
+    /// However, this incurs a performance cost.
+    ///
+    /// Dependent on the situation, switching to [`ParallaxMappingMethod::Relief`]
+    /// and keeping this value low might have better performance than increasing the
+    /// layer count while using [`ParallaxMappingMethod::Occlusion`].
+    ///
+    /// Default is `16.0`.
+    pub max_parallax_layer_count: f32,
 }
 
 impl Default for StandardMaterial {
@@ -260,6 +338,10 @@ impl Default for StandardMaterial {
             fog_enabled: true,
             alpha_mode: AlphaMode::Opaque,
             depth_bias: 0.0,
+            depth_map: None,
+            parallax_depth_scale: 0.1,
+            max_parallax_layer_count: 16.0,
+            parallax_mapping_method: ParallaxMappingMethod::Occlusion,
         }
     }
 }
@@ -302,6 +384,7 @@ bitflags::bitflags! {
         const TWO_COMPONENT_NORMAL_MAP   = (1 << 6);
         const FLIP_NORMAL_MAP_Y          = (1 << 7);
         const FOG_ENABLED                = (1 << 8);
+        const DEPTH_MAP                  = (1 << 9); // Used for parallax mapping
         const ALPHA_MODE_RESERVED_BITS   = (Self::ALPHA_MODE_MASK_BITS << Self::ALPHA_MODE_SHIFT_BITS); // ← Bitmask reserving bits for the `AlphaMode`
         const ALPHA_MODE_OPAQUE          = (0 << Self::ALPHA_MODE_SHIFT_BITS);                          // ← Values are just sequential values bitshifted into
         const ALPHA_MODE_MASK            = (1 << Self::ALPHA_MODE_SHIFT_BITS);                          //   the bitmask, and can range from 0 to 7.
@@ -341,6 +424,16 @@ pub struct StandardMaterialUniform {
     /// When the alpha mode mask flag is set, any base color alpha above this cutoff means fully opaque,
     /// and any below means fully transparent.
     pub alpha_cutoff: f32,
+    /// The depth of the [`StandardMaterial::depth_map`] to apply.
+    pub parallax_depth_scale: f32,
+    /// In how many layers to split the depth maps for Steep parallax mapping.
+    ///
+    /// If your `parallax_depth_scale` is >0.1 and you are seeing jaggy edges,
+    /// increase this value. However, this incurs a performance cost.
+    pub max_parallax_layer_count: f32,
+    /// Using [`ParallaxMappingMethod::Relief`], how many additional
+    /// steps to use at most to find the depth value.
+    pub max_relief_mapping_search_steps: u32,
 }
 
 impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
@@ -367,6 +460,9 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
         if self.fog_enabled {
             flags |= StandardMaterialFlags::FOG_ENABLED;
         }
+        if self.depth_map.is_some() {
+            flags |= StandardMaterialFlags::DEPTH_MAP;
+        }
         let has_normal_map = self.normal_map_texture.is_some();
         if has_normal_map {
             if let Some(texture) = images.get(self.normal_map_texture.as_ref().unwrap()) {
@@ -407,15 +503,20 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
             reflectance: self.reflectance,
             flags: flags.bits(),
             alpha_cutoff,
+            parallax_depth_scale: self.parallax_depth_scale,
+            max_parallax_layer_count: self.max_parallax_layer_count,
+            max_relief_mapping_search_steps: self.parallax_mapping_method.max_steps(),
         }
     }
 }
 
+/// The pipeline key for [`StandardMaterial`].
 #[derive(Clone, PartialEq, Eq, Hash)]
 pub struct StandardMaterialKey {
     normal_map: bool,
     cull_mode: Option<Face>,
     depth_bias: i32,
+    relief_mapping: bool,
 }
 
 impl From<&StandardMaterial> for StandardMaterialKey {
@@ -424,6 +525,10 @@ impl From<&StandardMaterial> for StandardMaterialKey {
             normal_map: material.normal_map_texture.is_some(),
             cull_mode: material.cull_mode,
             depth_bias: material.depth_bias as i32,
+            relief_mapping: matches!(
+                material.parallax_mapping_method,
+                ParallaxMappingMethod::Relief { .. }
+            ),
         }
     }
 }
@@ -435,11 +540,14 @@ impl Material for StandardMaterial {
         _layout: &MeshVertexBufferLayout,
         key: MaterialPipelineKey<Self>,
     ) -> Result<(), SpecializedMeshPipelineError> {
-        if key.bind_group_data.normal_map {
-            if let Some(fragment) = descriptor.fragment.as_mut() {
-                fragment
-                    .shader_defs
-                    .push("STANDARDMATERIAL_NORMAL_MAP".into());
+        if let Some(fragment) = descriptor.fragment.as_mut() {
+            let shader_defs = &mut fragment.shader_defs;
+
+            if key.bind_group_data.normal_map {
+                shader_defs.push("STANDARDMATERIAL_NORMAL_MAP".into());
+            }
+            if key.bind_group_data.relief_mapping {
+                shader_defs.push("RELIEF_MAPPING".into());
             }
         }
         descriptor.primitive.cull_mode = key.bind_group_data.cull_mode;