FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks

David Bauer, Qi Wu, Kwan-Liu Ma

View presentation: 2022-10-19T19:00:00Z GMT-0600 Change your timezone on the schedule page
Exemplar figure, described by caption below
Our neural rendering pipeline improves rendering performance while preserving image quality. A foveated ray marcher sparsely samples a volume (top left). The full image is then reconstructed using a multi-stage hybrid neural network (lower row). The resulting image quality is very similar to the ground truth image (top right) while providing between two to three times speed-up.

Prerecorded Talk

The live footage of the talk, including the Q&A, can be viewed on the session page, (Volume) Rendering.

Fast forward

Volume data is found in many important scientific and engineering applications. Rendering this data for visualization at high quality and interactive rates for demanding applications such as virtual reality is still not easily achievable even using professional-grade hardware. We introduce FoVolNet --- a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network. Foveated rendering is a technique that prioritizes rendering computations around the user's focal point. This approach leverages properties of the human visual system, thereby saving computational resources when rendering data in the periphery of the user's field of vision. Our reconstruction network combines direct and kernel prediction methods to produce fast, stable, and perceptually convincing output. With a slim design and the use of quantization, our method outperforms state-of-the-art neural reconstruction techniques in both end-to-end frame times and visual quality. We conduct extensive evaluations of the system's rendering performance, inference speed, and perceptual properties, and we provide comparisons to competing neural image reconstruction techniques. Our test results show that FoVolNet consistently achieves significant time saving over conventional rendering while preserving perceptual quality.