HyperReel is a novel 6-DoF video representation that has been proposed by a collaboration research team from Meta, Carnegie Mellon University, University of Maryland, and Reality Labs Research.
Using this brand-new, highly efficient technology, static scenes can have a photorealistic view synthesis that goes beyond the traditional 2D or 3D video representation.
The new approach outperforms most other ones “in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code”.
The findings were published in the paper “HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling” on arXiv.
HyperReel is a video representation method that aims to capture the six degrees of freedom (6-DoF) of movement in a three-dimensional space.
The 6-DoF are made of three spatial dimensions (x, y, z) and three rotational dimensions (pitch, roll, yaw). In this new environment the users can change their head position in space in addition to the viewing direction.
HyperReel
HyperReel is a novel representation for 6-DoF video that offers:
- A new sample prediction network for volumetric view synthesis with a faster volume rendering
- The ability to represent with high accuracy complex view-dependent effects
- A memory-efficient dynamic volume representation
- A desirable balance between speed, quality, and memory, while rendering in real time at megapixel resolution
HyperReel is made of two main components:
- A ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions
- A keyframe-based dynamic volume representation
Conclusion and future research
HyperReel is intended to enhance the immersion of virtual reality (VR) and augmented reality (AR) applications. As this new model achieves between 6.5 and 29 frames per second, it is not yet suitable for real-time virtual reality applications, where ideally at least 72 frames per second must be rendered in stereo.
According to the Meta research team, this could be achieved in the future with additional technical effort.
Learn more:
- Research paper: “HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling” (on arXiv)
- Project homepage
- Code (on GitHub)
