XR Volume Rendering

[tbirdso]

Need

WebXR examples currently only encompass surface rendering. It would be useful to add an example demonstrating volume rendering as well.

Feature

Two changes are needed:

• An AR button should be added to the VolumeViewer example
• Any observed issues with volume rendering should be debugged

AR volume rendering was previously observed to work in the WebXR Emulator but the vtk.js scene did not appear when viewing on mobile.

[tbirdso]

Investigation results:

• AR volume rendering in the VolumeViewer example works as expected in WebXR emulator browser extension
• VolumeViewer AR rendering works on mobile device, however the far clipping plane seems to be set very close to the camera. The volume is not displayed until the device is moved forward to be a few inches away from the virtual object.
• Same behavior for various sizes of image data. Smaller image data renders much faster and is recommended for testing on mobile hardware. Created a small tube of random intensities scaled to the intensity range of VolumeViewer sample data so as to match the opacity/color functions: https://data.kitware.com/api/v1/file/624320e74acac99f42254a25/download
• Current theory is that the behavior may be related to default camera parameters in VolumeViewer. Unfortunately I won't have time to dig deeper than this for a few days.

[tbirdso]

@floryst Reminder that this exists, no particular rush though. Would be helpful to have 1) any console printouts from volume rendering on Android and 2) your thoughts on parameters that might affect the clipping plane, which seems to be set to the default (far plane 1000.0)

[floryst]

I got the "Start AR" button to work with some CSS modification, and got a volume to show up in AR on the ar-apps branch.

Touching the screen did throw the following error (index.js:437):

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'buttons')
    at index.js:437:62
    at XRInputSourceArray.forEach (<anonymous>)
    at Object.publicAPI.updateXRGamepads (index.js:422:28)
    at _callee3$ (index.js:418:8)
    at tryCatch (polyfill.js:5901:40)
    at Generator.invoke [as _invoke] (polyfill.js:6135:22)
    at Generator.prototype.<computed> [as next] (polyfill.js:5953:21)
    at asyncGeneratorStep (Volume.js:11:1)
    at _next (Volume.js:11:1)
    at Volume.js:11:1

I'm also experiencing the clipping plane issue. The range should be much wider in order, otherwise it's difficult to use.

[floryst]

If I invoke pipeline.renderer.getActiveCamera().setClippingRange(10, 1200), the visualization is far more acceptable. The standard resetCamera() works well for 2D views, but not so well with AR. One idea is to get the scene's bounding box, inflate it a bunch, and then pass it into resetCamera(bounds).

[floryst]

FYI @tbirdso to fix the clipping issue, we can add conditionals for the VR version of resetCamera and resetCameraClippingRange, as seen here: https://github.com/Kitware/VTK/blob/master/Rendering/VR/vtkVRRenderer.cxx

[tbirdso]

I experimented a bit with resetting clipping planes for AR last Friday. Increasing the far clip plane helped significantly and seemed to correspond neatly to real-world distances. However, the near clipping plane to anything farther than 0.1 or so seemed to reduce the far clip distance rather than moving the near clip plane outward, which was unexpected. Unclear if this is an issue specific to WebXR, need to experiment further.

Do we have an example for adjusting a camera's clipping planes without XR that I could use as a baseline?

[floryst]

There isn't any example that adjusts the clipping range IIRC. You can take the volume rendering example and adjust the camera clipping range manually via devtools, or add a small slider UI for near/far.

[floryst]

I'm not familiar with any behavior that would force the far clipping range to move closer if you reduce the near clipping plane to below 0.1. I would be interested to know if that occurs with the regular renderer.

[tbirdso]

I was not able to reproduce the clipping issue by playing around with the Cone example, the near and far clip planes were adjustable with setClippingRange as expected. That is a bit of a relief.

I will investigate whether something strange might be happening with our physical-to-view transformation in XR rendering.

EDIT: I also get expected near clipping behavior when experimenting with data in VolumeViewer without XR. I continue to get strange behavior in XR where I cannot adjust the near clipping plane to clip through volume data. Have not been able to pin down any underlying behavior that would lead to this.

[tbirdso]

@floryst I am getting a bit confused on this one. From what I can tell getClippingRange is being called in VolumeMapper using parameters from openGLCamera. Those camera parameters do not seem to reflect the view-to-physical and physical-to-world adjustments made in Core/Camera. As far as I can tell OpenGLCamera and Camera are being treated as completely separate objects.

Could you please comment on how OpenGLCamera is being used in VolumeMapper? Am I missing something obvious as to how OpenGLCamera should be composed with / inherit from Camera?

[floryst]

model.openGLCamera.getRenderable() should return a vtkCamera. The OpenGLCamera represents the opengl scene-graph node for the camera, so getRenderable() returns the vtk.js abstraction for the camera with which we interact.

[tbirdso]

Thanks, I think that makes sense. Maybe the problem does not lie there then. Will keep digging.

[tbirdso]

Collecting thoughts so far:

• The issue at hand is that apparent near/far clip planes do not correspond to user settings when performing volume rendering for a scene to overlay on an AR device with WebXR
• The near clip plane appears to always be close to 0, so that the camera can move right up to the rendered volume regardless of the user setting
• The clipping range thickness (far - near) appears to correspond to the difference between the user near/far clip parameters with proper adjustment for physical space. That is, if the clipping range is initialized with camera.setClippingRange(5 * physicalScale, 10 * physicalScale) then the volume starts to clip out of the scene when the camera is 5m in physical space away from the volume.
• The scene otherwise responds appropriately to camera rotation and translation. If I move the camera away from a cube volume at an oblique angle the volume clips with a far clip plane perpendicular to the camera direction as expected.
• An issue in OpenGL/VolumeMapper would explain why this issue only appears for volume rendering and not surface rendering. However, as far as I can tell the openGLCamera object is getting the correct world-to-view matrix from the underlying vtkCamera for rendering. EDIT: Seeing inconsistent behavior in AR surface rendering example, clipping range seems to be entirely ignored.