This research introduces a deep-learning paradigm called functional learning (FL) to train a loose neuron array—a group of non-handcrafted, non-differentiable, and loosely connected physical neurons. FL addresses challenges in training non-differentiable hardware, including precise modeling of high-dimensional systems, on-site calibration of hardware imperfections, and end-to-end training of physical neurons. It enables hardware creation without strict fabrication and precise assembly, impacting hardware design, chip manufacturing, physical neuron training, and system control. The paradigm is verified using a light field neural network (LFNN), creating a programmable incoherent optical neural network (ONN) that processes parallel visible light signals for high-bandwidth, power-efficient inference. Potential applications include brain-inspired optical computation, high-bandwidth neural network inference, and light-speed programmable lenses/displays/detectors.