Neural étendue expander for ultra-wide-angle high-fidelity holographic display

Holographic displays, promising for virtual and augmented reality, generate light fields by modulating a coherent light beam's wavefront using an SLM. However, SLM limitations, primarily the limited spatial resolution of current LCOS technology, restrict diffraction angles, resulting in low étendue (the product of display area and maximum solid angle of diffracted light). This low étendue necessitates trade-offs between FOV and display size, hindering the creation of immersive VR/AR devices requiring large FOVs (at least 120°) and substantial eyebox sizes (e.g., 10 x 10 mm²). Achieving the necessary étendue would necessitate billions of SLM pixels, far exceeding current technological capabilities. Existing attempts to circumvent this limitation, such as dynamic feedback (eye tracking), spatial integration with multiple SLMs, and temporal integration with laser arrays or DMDs, introduce complexity, increased form factor, and power consumption issues. Rewritable photopolymers and MEMS offer alternatives but have drawbacks like low refresh rates and limited pixel counts/bit depth, respectively. Methods optimizing within the limited étendue (e.g., trading spatial resolution for depth resolution) don't fundamentally address the étendue problem. While employing optical elements with randomized scattering properties in front of an SLM has been explored, these methods result in low-fidelity holograms due to the agnostic nature of the scattering and limited SLM degrees of freedom. This paper presents a novel approach, neural étendue expanders, to overcome these limitations.

Several approaches have been proposed to address the limited étendue of holographic displays. Dynamic feedback mechanisms, such as eye tracking, aim to optimize the displayed content for the viewer's current gaze. However, these methods introduce latency issues that can lead to motion sickness. Spatial integration techniques use multiple SLMs to expand the FOV, but this increases complexity and size. Temporal multiplexing with fast-switching DMDs or laser arrays offers another approach, but precise timing control is critical, adding another level of challenge. Static optical elements, such as those with randomized scattering properties, have shown promise in increasing the diffraction angle, but the resulting holograms often suffer from low fidelity and require extensive calibration. The use of lenses or lenslet arrays to expand the field of view has also been explored, but this generally reduces the effective eyebox size. Tilting cascades offer a way to increase the étendue but have a large physical footprint. Previous work focusing on optimization within the existing étendue limitations or using random scattering masks have not fundamentally addressed the problem of achieving both high fidelity and wide field of view in holographic displays. This study proposes a fundamentally different approach.

This research introduces neural étendue expanders, learned static optical elements placed in front of an SLM to enlarge the diffraction angle and thus the étendue. Unlike existing random scattering masks, these expanders are jointly learned with the SLM modulation patterns using a differentiable holographic image formation model. This model, based on Fourier optics, relates the displayed holographic image (I) to the wavefront modulation of the neural étendue expander (ε) and the SLM modulation (S): I = F(ε ⊙ U(S))², where F represents the 2D Fourier transform, U(.) is a zeroth-order upsampling operator, and ⊙ denotes the Hadamard product. The differentiability of this model allows for the optimization of both ε and S using first-order stochastic optimization techniques. The optimization objective minimizes the difference between the generated hologram and a set of target natural images, incorporating a low-pass Butterworth filter to account for the human visual system's limitations. A dataset of 105 high-resolution natural images is used for training, and 20 images are used for testing. The process jointly optimizes a single static neural étendue expander and multiple SLM patterns, akin to training a shallow neural network. The design process considers the frequency characteristics of natural images and aims to push reconstruction noise beyond the perceivable frequency bands of the human visual system. Physical fabrication of the neural étendue expander is achieved using resin stamping with a 2 µm pitch, placing it at the conjugate plane of the SLM. A DC block filters out undiffracted light. Experimental validation uses a holographic display prototype incorporating a 1K-pixel SLM, a 4f system, and a camera for imaging.

The proposed neural étendue expanders achieve a 64x étendue expansion factor experimentally, resulting in an 8x increase in FOV horizontally and vertically. High-fidelity reconstruction is demonstrated, with a PSNR exceeding 29 dB for retinal-resolution images. The method is validated using a 1K-pixel SLM, resulting in an 8x FOV expansion in each direction. The results show that the conventional holographic display produced high-fidelity content with a small FOV. Increasing the étendue using a binary random expander increased the FOV but resulted in low image fidelity and chromatic artifacts. The neural étendue expanders achieved both ultra-wide FOV and high fidelity. The generated holograms displayed high contrast and were free of chromatic aberration. Comparisons against uniform random expanders show a significant improvement in PSNR (over 14 dB for trichromatic holograms and over 10 dB for monochromatic holograms). The neural étendue expander's design preserves the major frequency bands of natural images, pushing reconstruction artifacts outside the human perceivable frequency range. The expanders' performance remains robust across different SLM resolutions (1K and 8K simulations show consistent improvements) and pupil positions, unlike methods using quadratic phase profiles. Finally, the approach extends to 3D color holograms, outperforming existing techniques in fidelity.

The results demonstrate that neural étendue expanders provide a significant advancement in holographic display technology. The ability to achieve a 64x étendue expansion with high fidelity represents an order-of-magnitude improvement over existing methods. This advancement addresses a critical limitation in holographic display technology, paving the way for more immersive and realistic VR/AR experiences. The joint optimization of the neural étendue expander and SLM patterns, along with the incorporation of human visual system characteristics, is key to the success of this approach. The robustness of the expanders to pupil movement and their ability to produce high-fidelity 3D color holograms further expand their potential applications. The use of a data-driven approach for optical element design introduces a new paradigm that may inspire innovations in other optical systems.

This work introduces neural étendue expanders, a novel approach to expanding the étendue of holographic displays while maintaining high image fidelity. These learned optical elements achieve a 64x étendue expansion factor with over 29 dB PSNR, a significant improvement over existing techniques. The method is robust to variations in pupil position and extends to 3D color holography. This approach holds great promise for future holographic display development, particularly for VR/AR applications. Future work could explore the use of other emerging optical technologies, such as metasurfaces, to further enhance performance and potentially enable even larger étendue expansions.

While this research demonstrates significant improvements in holographic display technology, some limitations exist. The current fabrication process for the neural étendue expanders might not be easily scalable to mass production. The training process requires a dataset of natural images, and the quality of the generated holograms may depend on the characteristics of this dataset. The optimization process is computationally intensive, potentially limiting the real-time application of the technique. Future research should address these aspects to further enhance the practicality and applicability of the proposed approach.