Deep learning's widespread use is hampered by the escalating energy costs of larger neural networks. Optical neural networks (ONNs) offer a potential solution. The energy consumption of large-scale commercial deep neural network (DNN) deployments is dominated (80-90%) by inference, creating a strong need for more energy-efficient hardware specialized for this task. Optical processors are proposed as deep-learning accelerators capable of superior energy efficiency and lower latency than electronic processors. Their primary role is implementing matrix-vector multiplications, the most computationally intensive operations in DNNs. Theory and simulations suggest ONNs using optical matrix-vector multipliers can achieve extreme energy efficiency exceeding the fundamental limit for irreversible digital computers. For sufficiently large vectors, matrix-vector multiplication could theoretically be performed with less than 1 photon per scalar multiplication, significantly outperforming electronic processors. The energy efficiency of optical matrix-vector multiplication improves with the size of the matrix and vectors. While various multiplexing approaches and architectures (wavelength, spatial multiplexing in photonic integrated circuits and 3D free-space) are advancing ONN efficiency, the potential of operating close to the shot-noise limit remains largely unexplored. This paper reports on the experimental validation of an ONN operating in the sub-photon-per-multiplication regime.

The paper reviews existing research on optical neural networks and their potential for energy efficiency. It highlights the theoretical predictions of sub-photon-per-multiplication performance and the various architectural approaches being explored to improve efficiency. The authors note the rapid progress in ONNs across multiplexing approaches and architectures, but emphasize that the ability to operate close to the shot-noise limit of detection has not yet been fully exploited. This sets the stage for their own experimental work, aiming to demonstrate operation in the sub-photon regime.

The researchers constructed an experimental setup using a modified Stanford-Vector-Multiplier architecture with optical fan-out to compute optical vector products. The setup encoded vector elements in the intensity of spatial modes from an organic light-emitting diode (OLED) display, with weights encoded as the transmissivity of a digital light modulator (SLM). Dot products were computed in two steps: element-wise multiplication (OLED pixel intensity aligned to corresponding SLM pixel) and optical fan-in (intensity-modulated pixels summed onto a detector). The system could perform 711 x 711 = 505,521 scalar multiplications and additions in parallel. The large-scale summation improves the signal-to-noise ratio (SNR), enabling accurate readout even with sub-photon inputs. Dot product accuracy was characterized by computing dot products of randomly chosen vector pairs, varying the number of photons using neutral-density filters and measuring the root-mean-square (RMS) error. To demonstrate ONN functionality, they trained a 4-layer fully connected multi-layer perceptron (MLP) using quantization-aware training (QAT) for low-precision hardware. The trained network was executed on the optical setup, and classification accuracy was measured as a function of the photon budget. The experimental results are compared to simulations with added shot noise to isolate the effect of photon noise.

The experiment achieved high accuracy in dot products using as few as 0.001 photons per scalar multiplication, with the dominant error source being shot noise at low photon counts. For vectors of size ~0.5 million, an RMS error of ~6% was observed with 0.001 photons per multiplication, corresponding to approximately 4 bits of precision. Increasing the photon budget reduced the error, reaching a minimum of ~0.42% at 2 photons per multiplication or higher. The researchers then used their setup to perform MNIST handwritten digit classification with a trained 4-layer MLP. With 3 photons per multiplication, they achieved ~99% accuracy, nearly identical to the digital computer result. Remarkably, even with 0.66 photons per multiplication (2.5 × 10⁻⁹ J of optical energy), the ONN achieved ~90% accuracy. The total optical energy for a single inference with 99% accuracy was approximately 10⁻⁷ J, considerably less than that required for a single floating-point scalar multiplication in electronic processors. The close agreement between experimental and simulated results indicates that shot noise was the primary accuracy limiter at low photon counts. The optical energy per inference was approximately 230 fJ (accounting for SLM transmission losses), orders of magnitude lower than state-of-the-art electronic accelerators for the same neural network.

The experimental results strongly support the potential of ONNs for significantly improved energy efficiency compared to electronic neural networks. The sub-photon-per-multiplication operation demonstrates the dominance of the standard quantum limit (shot noise) in determining accuracy. The achievement of high classification accuracy with extremely low photon budgets suggests potential orders-of-magnitude energy savings not only on a per-operation basis but also on a per-inference basis. The authors acknowledge that optical energy consumption might represent only a small fraction (1%) of the total system energy, but even then, an ONN system could still reach energy consumption per inference orders of magnitude lower than current electronic accelerators.

This study experimentally confirms the theoretical potential of ONNs to provide drastically improved energy efficiency over electronic neural networks. Sub-photon-per-multiplication operation was demonstrated, achieving high classification accuracy with extremely low optical energy consumption. Future research should focus on developing integrated matrix-vector multipliers with higher optical efficiency and faster modulators to further enhance the overall energy efficiency of ONN systems and explore applications beyond neural networks.

The experimental setup's data throughput was limited by the update rate of the OLED and SLM (10 Hz). Although the detector was much faster, this limitation restricts the applicability of the findings to high-speed systems. The 2D-block matrix-vector multiplier used is not the most suitable architecture for integrating photonic modules; its immediate applicability might be confined to specific tasks utilizing incoherent light sources.