Explaining the Need for Varifocal Lenses in VR

 Vision comes with a few components that give you a sense of depth. Parallax is the effect where (over time) moving a perspective shows near objects moving faster than far objects giving the effect of depth. Stereoscopic vision is similar but you have two perspectives in different places which place near objects further apart than far objects and your mind creates a sense of depth from that. Finally there is focal distance. Focal distance can give you an effect of scale and distance from a single perspective. Much like 'tilt-shift' makes things look small, messing with the focus can emphasize the relative and even the absolute distance and size of objects.

The Index & Rift have a set focal distance of about 2 meters while the vive has a focal distance of 0.75m. That means that if you look at an object at that exact distance then it will be perfectly in focus in addition to having correct stereo and parallax effects. When you look at an object at a different distance then the dissonance in the focal distance from the other effects causes discomfort and 'blurriness'.

Focus on your finger a few cm from your eye with a background a few meters away. Move your finger further and further until you notice the background come into focus. You'll notice that there's a bit of a logarithmic scale in terms of how much the focal effect presents itself based on distance. The difference in focus between 2 meters and infinity is a lot less significant than the difference between 5cm and 25cm.

Effectively, current headsets are well tuned to keep objects in the same virtual room as you in focus until you get to close. Objects in the far distance aren't easy to make out because of low resolution but if they weren't, you would see that they are also slightly blurry because they're out of focus. Near objects like your hands that you bring to your face will go out of focus a lot more easily and they're also large relative to the display so resolution isn't an issue. They will be stereoscopically accurate but completely out of focus. You should be able to notice that if you close one eye and try and focus on an still object (like a book with text) near the camera, you should be able to focus on it clearly and even read it easily but the scale will appear to be incorrect and very large.

Varifocal lenses can either move the lenses like a camera to change the focal distance based on your eye movements and where in the virtual world you are looking or they can toggle lenses on and off to match the focal distance you should be seeing as closely as possible. The new issue that arises is that now all objects in the scene are going to be in focus all the time regardless of where you are focused. While better, this is still quite inaccurate. The solution can be digital by artificially blurring all objects as accurately as possible but there are also ways to have multiple displays with different focal distances or even change the focal distance differently in different parts of the screen.

Eye tracking is a prerequisite for such tech so foveated rendering will appear first in consumer headsets then we'll probably have to wait another generation to get this tech. I've used the Half-dome oculus prototype varifocal headset and while it's nice to be able to focus correctly, the digital blurring and eye tracking leave a lot to be desired and honestly, I think improving refresh rate and FOV along with adding effective foveated rendering will provide far larger improvements than varifocal lenses.

Further Pathtracing and Lessons Learned

Since the last article, I've made some substantial changes. I've added screenspace settings to the build, added multiple rays per pixel, randomly generated the scene at runtime, added movement controls, added fast movement compensation with adaptive temporal accumulation for VR, and finally I added a light Gaussian blur effect to reduce noise.

Dynamically changing the temporal accumulation in addition to improving the reprojection warping allows the user to get a clean image while still remaining temporally stable and responsive to faster movement.

It's clear that simply increasing the rays per pixel has little effect on image quality while absolutely demolishing any semblance of performance. The resulting image remains noisy and generally unsatisfactory.
Temporal accumulation does a substantially better job while not impacting performance and only introducing a somewhat obnoxious temporal artifact that is mitigated by higher framerates and movement compensation. Temporal accumulation alone however still leaves a fine grain noise with pixel sized bright specks and black spots.
Finally, the softer Gaussian blur helps remove those pixel specks and spots and results in a fairly clean image better than nearly anything short of intelligent hardware accelerated denoising.

As a side experiment I tried building to UWP and using holographic remoting to run the project on the Hololens. Performance was great even at 4x resolution due to the low 720p resolution of the Hololens displays but wifi speeds & reliability as well as compression artifacts were problematic.

Simple high raycount per pixel

Lower rays/pixel and no Gaussian blur but high temporal accumulation

With light Gaussian blur, lower rays/pixel, and heavy temporal accumulation

Things that worked:

• Separate rendering for each eye for XR
• Moving Unity GameObjects and meshes each frame
• Gaussian blur to reduce noise
• Dynamically changing RenderTexture resolution
• Distorting temporal accumulation based on 3DOF head rotation
• Changing temporal accumulation rate based on head movement rate
• Hololens support with Holographic remoting

Things that didn't:

• Foveated rendering with Graphics.CopyTexture()
• OpenGL ES 3.1 compute shaders for Android
• MultiGPU parallel compute shaders

Things that might be possible but aren't easy:

• RT core acceleration
• MultiGPU support with RT cores and/or shaders
• AI tensor core or CPU intelligent realtime denoising
• Dynamic rate of rays per pixel (based on pixel coordinates)
• Foveated rendering with that rays/pixel value
• Complex meshes without stranglehold bandwidth and calculation bottlenecks

How to Pathtrace in VR

Before I start, I want to thank David Kuri of the VW Virtual Engineering Lab in Germany for his example project and tutorial that I've based my modifications off of.

The following is his tutorial:
http://blog.three-eyed-games.com/2018/05/03/gpu-ray-tracing-in-unity-part-1/

This link is to the BitBucket Git repo:
https://bitbucket.org/Daerst/gpu-ray-tracing-in-unity/src/Tutorial_Pt3/

My project can be found at  https://github.com/Will-VW/Pathtracing

To start, this is a project I worked on as an experiment while part of the Volkswagen Virtual Engineering Lab in the Bay Area.

While searching for interesting raytracing implementations to challenge our 2080 TI, we were coming up fairly short. We found a couple lackluster and simple DirectX projects, poor Unity and Unreal support, and marginally impressive demos on rails. That is until we found our colleague's project using simple compute shaders in regular Unity.

Obviously we forgo RTX and DXR acceleration and for the moment we do not have denoising or proper mesh support but we gain near unlimited flexibility and control.

As far as we are aware (as of June 2019), there are no raytracing or pathtracing applications, demos, games, roadmaps, plans, or anything else with any sort of VR support or connection. We decided to change that.

I took David's existing project that worked in "simple" single camera pancake views and modified it until I could get it to work properly with SteamVR.

Temporal accumulation was the biggest issue and due to the fact that both right and left eyes were from different perspectives and the constant VR camera movement it was effectively impossible to have anything more than a single frame of accumulation in the headset. I first used instances to split the left and right eye textures so accumulation would be separated. I then used a shader to account for head rotation and adjust the accumulated frames accordingly to attempt to line up with the newest perspective.

Both efforts were effective. The former far more so than the latter which still suffered from artifacts due to the distortion at high FOVs like VR headsets have.

Next I targeted a dynamic resolution so that the number of rays projected each frame could be controlled to suit our needs. I changed the RenderTexture resolution appropriately and allowed the program to render based off of the VR headset resolution rather than the Unity mirrored view resolution.

Later, I attempted foveated rendering to improve performance which remained lackluster. I learned how expensive the Graphics.Blit() and Graphics.CopyTexture() methods are in bandwidth and found that while foveated rendering was possible, it hurt performance more than it helped.

I then added in support for moving around mesh objects that were imported at runtime based on the GameObject Transform. This allowed me to put in a moving and rotating block that moved around the scene. I also learned of the inefficiencies of the existing mesh pathtracing technique and began to explore more implicit surface calculations.

I looked at SLI support to parallelize the pathtracing but found that compute shaders do not support multi-GPU and CUDA/OpenCL options were unviable for our purposes.

Thus we managed to get a fully pathtraced program running at full 90FPS in a Vive Pro at native resolution and 1 ray per pixel with perfectly accurate reflections, lighting, ambient occlusion, and shadows. Albeit with a lot of noise, no RTX acceleration, limited mesh usage due to performance, no standard materials or shaders, and no effective foveated rendering.