Large depth-of-field ultra-compact microscope by progressive optimization and deep learning

Microscopy enables critical applications in biology, neuroscience, and clinical diagnostics, yet conventional systems are bulky, complex, and require trained operation. Geometric aberrations constrain resolution across millimeter-scale fields of view (FOV), creating a trade-off between space-bandwidth product and optical complexity. Higher numerical aperture (NA) needed for resolution reduces depth of field (DOF), degrading imaging of 3D-distributed samples. While sophisticated optical designs and multiview acquisition can improve performance, bulkiness remains a challenge. Miniaturized microscopes have advanced neural recording, high-throughput screening, and cytometry, and computational methods can extend DOF and correct color, but current miniaturized designs face limitations in FOV, distortion, size/weight, cost, acquisition speed, and often monochromatic operation due to limited space for compound lenses. Deep optics that co-optimize optics and algorithms have shown promise for large FOV and DOF, HDR, and hyperspectral imaging, but remain constrained to simpler systems; microscopic applications with small working distances and large FOVs present large solution spaces and aberrations. In addition, megapixel restoration networks are resource-intensive and hard to deploy in integrated systems. The study aims to overcome these challenges by proposing a progressive optimization pipeline integrating aspherical optics, diffractive optical elements, and physics-based deep learning to realize a compact, low-cost microscope with extended DOF and high resolution suitable for mobile deployment.

Prior work on miniaturized microscopy demonstrated utility in neural activity recording in freely behaving animals, high-throughput screening, and flow cytometry. Computational imaging approaches can extend DOF and correct chromatic aberrations. However, simple-lens approaches suffer from sub-millimeter FOVs with distortions, while larger FOVs require complex multi-lens assemblies that increase length and weight. Multiphoton miniaturized microscopes offer depth penetration and sectioning but need specialized optics and have lower throughput speeds. Most miniaturized systems are monochromatic due to space constraints for compound lenses. Deep optics integrating optical design with neural reconstruction have achieved large FOV/DOF and other capabilities, but end-to-end optimization is challenged by the nonlinearity, high dimensionality, and computational burden, especially for high-NA, large-FOV microscopes. Existing shift-variant deconvolution methods can handle irregular PSFs over large FOVs, and unsupervised image-to-image translation avoids paired data, but may yield artifacts and lower PSNR/SSIM than supervised or simulation-supervised approaches. This study builds on these advances, addressing the limitations by a staged optimization combining ray-tracing and DL.

The authors propose a three-stage progressive optimization pipeline: (1) Ray-tracing-based lens design: Using ZEMAX and Rayleigh–Sommerfeld diffraction modeling, they optimize a four-element aspherical plastic lens group (materials EP-9000 and ZEONEX_K22R/K26R_2017) targeting NA 0.16, focal length 1 mm, conjugate distance 6 mm, and FOV diameter >3.6 mm. A multi-dimensional coupling optimization equalizes MTF across 470–650 nm, reduces chromatic aberration without cemented doublets, and achieves ~3 µm resolution across the FOV. Aspherical surface shapes are optimized via adaptive gradient descent. (2) DOE-based extended DOF: A diffractive optical element (cubic phase plate) is placed near the pupil plane to encode the wavefront, making PSFs more depth-invariant. The phase strength α is optimized using an MTF-similarity merit across defocus, quantified by Fisher Information to minimize sensitivity to defocus, with constraints to avoid overmodulation and ensure MTF at Nyquist >0.1. Candidates for α in [0.005, 0.075] (15 values) are optimized along with lens parameters to produce consistent MTFs over 300 µm DOF. (3) Neural reconstruction co-optimization and selection: For each optical candidate, a simulation-supervised deep network is trained to restore sharp images from coded captures; the best-performing optical+network configuration is selected (optimal α≈0.03). Simulation-supervision training data: A standard 5× microscope with a motorized stage captures focal stacks from −150 µm to +150 µm (10 µm steps). All-in-focus targets are generated by depth fusion of in-focus regions. Inputs are generated by convolving per-depth slices with depth- and field-dependent PSFs and summing. Shift-variant forward model: Due to PSF variability across the 3.6 mm FOV and NA 0.16, PSFs are modeled via non-negative matrix factorization into spatial bases h_i(x,y,z) and coefficient maps w_i(u,v,z), enabling efficient FFT-based convolutions for simulations and deconvolution. Network architecture and training: A pix2pix-style GAN with a U-Net generator (4 down/4 up blocks, 9 residual blocks) and PatchGAN discriminator is trained with GAN, L2, and perceptual (VGG19) losses using AdamW (lr=2e−4 with warmup and linear decay), 300 epochs on ~110 images (cropped to 512×512 patches). Deconvolution baseline: A modified shift-variant Richardson–Lucy with total variation regularization is implemented for comparison. Fabrication and calibration: Lenses are injection-molded; DOE via nanoimprint; housing CNC-machined; assembled by a manufacturer. Calibration uses a 1 µm pinhole array over the 3.6 mm FOV mounted on a phone-integrated sensor. Theoretical PSFs are preferred for training to mitigate fabrication noise and variability; robustness to decenter (20 µm) and tip/tilt (0.1°) tolerances is validated in simulation. Mobile deployment: The trained network is pruned (encoder/decoder channels reduced) to cut parameters by 78% with similar PSNR/SSIM/LPIPS, achieving ~5× inference speed-up; final activation changed to sigmoid for mobile acceleration; typical inference time ~1.73 s per 2160×2560×3 image on device. Application demo: A MobileNetV2 classifier trained on ~9000 images (CE loss, Adam) categorizes skin hydration (dry/normal/overhydration) from reconstructed skin micrographs for portable diagnostics.

- Ultra-compact microscope module volume 0.15 cm3 and weight 0.5 g; overall size reduction by ~5 orders of magnitude versus a tabletop microscope; volume reduction factor reported up to 6.7×10^5 compared to an Olympus IX73. - Optical performance comparable to a commercial 5×, NA 0.1 objective, achieving ~3 µm resolution across a >3.6 mm diameter FOV. - Extended depth of field (EDOF) of 300 µm at NA 0.16 (approximately 10× larger DOF than typical microscopes achieving 2–3 µm lateral resolution). - Progressive optimization reduces computational memory from >600 GB (direct end-to-end) to ~20 GB (over 30× reduction), enabling desktop-level design. - DOE coding yields depth-invariant MTF and non-degraded Strehl ratios across 300 µm; uncoded system degrades beyond ~30 µm defocus. - Calibration: Simulated and experimental PSFs closely match across depths and lateral positions; PSF sizes vs. depth align; robustness to manufacturing tolerances validated (decenter 20 µm, tip/tilt 0.1° with marginal degradation). - Chromatic performance: PSFs under blue/green/red illumination exhibit structural similarity index >0.7 across the FOV; similar EDOF across channels. - Resolution retention across depth: 1 µm emitter reconstructed with FWHM of 3.1 µm (z=0), 3.5 µm (z=100 µm), 4.8 µm (z=150 µm). - Contrast advantage over conventional microscope across the 300 µm depth range; maintains sharp USAF-1951 target features when defocused (z=150 µm), indicating generalization beyond training data. - Neural reconstruction outperforms shift-variant deconvolution in PSNR, SSIM, and LPIPS; maintains high SSIM across depths where deconvolution degrades beyond ~50 µm. Also outperforms unsupervised translation methods with fewer artifacts. - Pruned network: 78% parameter reduction with similar reconstruction metrics; ~5× faster rendering; on-phone pruned inference ~1729 ms per image. - Cost: mass-producible plastic lenses and nanoimprinted DOE, total cost below US$10 per unit for mass production; no cemented lenses. - Mobile integration: module integrated into a commercial smartphone with ring LED illumination; real-time EDOF reconstructions across diverse samples without color fringing. - Portable diagnostics: Skin moisture classification on-device achieved high accuracy; reported over 80% accuracy vs. electrical sensors on n=28 samples; application demonstrates before/after skincare hydration improvement (n=100 tests).

The progressive optimization approach effectively addresses the inherent trade-offs among resolution, DOF, chromatic aberration correction, and miniaturization by constraining the complex optical design space with ray-tracing merits and then refining performance via DOE coding and deep learning-based reconstruction. The resulting system delivers tabletop-level resolution and FOV with an order-of-magnitude DOF extension in a form factor amenable to mobile integration, directly tackling accessibility and portability barriers. The simulation-supervision strategy bridges the lack of paired ground truth by generating realistic coded inputs and all-in-focus labels from focal stacks, enabling a network that generalizes to varied specimens and maintains performance across depth, outperforming classical deconvolution and unsupervised methods. Chromatic correction without cemented elements demonstrates that aspherical plastic optics can meet multi-wavelength demands for practical imaging. The demonstrated smartphone integration and skin hydration monitoring showcase the practical relevance for point-of-care diagnostics and high-throughput screening. Beyond microscopy, the scalable pipeline is applicable to other compact imaging systems (e.g., telescopes, surveillance) and neuroscience (ultra-light head-mounted imaging), suggesting broader impact in mobile analysis and diagnostics. Remaining design-tool incompatibilities hint at future advances with differentiable ray tracing to more tightly integrate optical and algorithmic co-design.

This work introduces a comprehensive progressive optimization framework combining aspherical lens design, DOE-based wavefront coding, and physics-aware deep learning to realize an ultra-compact, low-cost microscope with 3 µm resolution over a >3.6 mm FOV and a 300 µm DOF—about 10× greater than conventional systems—while reducing size and weight by five orders of magnitude. The simulation-supervised network enables high-fidelity, depth-robust reconstructions and deployment on mobile devices via pruning for real-time use. The mass-producible design, successful smartphone integration, and a portable skin hydration application demonstrate immediate translational potential. Future research directions include leveraging differentiable ray tracing for tighter optics–algorithm co-optimization, exploring alternative phase coding profiles (e.g., higher-order asymmetric or circularly symmetric masks), integrating metasurfaces for further size reduction and wider FOV, expanding to fluorescence modalities via ring-LED excitation, and extending applications to in vivo neural imaging, flow cytometry, environmental monitoring, and mobile diagnostics.

- Training data generation relies on focal stacks from a benchtop microscope and physics-based simulation of coded captures; while distributions matched well, real-world deviations (fabrication noise, calibration artifacts) necessitated using theoretical PSFs, which may introduce domain gaps in other settings. - Fabrication variability exists (e.g., decenter, tip/tilt); simulations indicate robustness to specified tolerances, but broader manufacturing variations could affect performance. - The reconstruction pipeline depends on a trained deep network; performance and generalizability depend on training data diversity and may vary for unseen sample types or illumination conditions. - Current toolchain limits full end-to-end co-optimization due to incompatibilities between traditional ray tracing and deep learning frameworks; differentiable ray tracing is suggested but not implemented here. - The mobile diagnostic demonstration for skin hydration was evaluated on limited data (n=28 for accuracy comparison; a single 35-year-old volunteer for paired moisture measurements), limiting generalizability of clinical claims. - Inference on mobile devices, though pruned, still requires ~1.7 s per megapixel-scale image, which may constrain certain real-time applications.