The human sensory system demonstrates a remarkable ability to discern subtle variations in frequently encountered stimuli, such as horizontal orientations, more readily than those in less frequent stimuli, such as diagonal orientations (the 'oblique effect'). This non-uniform sensitivity across sensory stimuli is a ubiquitous phenomenon observed across various animal species, prompting investigation into the underlying mechanisms responsible for this phenomenon. The efficient coding hypothesis posits that sensory systems prioritize encoding common aspects of the world to optimize limited coding resources. This aligns with the observation that perceptual sensitivity often mirrors the statistical distribution of visual inputs. While the link between efficient neural codes, stimulus statistics, and perceptual behavior is well-established, the mechanisms generating these codes remain elusive. Brain development reveals that sensory representations are not innate but refine with age and visual experience, suggesting neural plasticity downstream of the retina plays a crucial role. The authors hypothesize that general task-oriented learning rules, such as gradient descent, are sufficient to generate efficient sensory representations without requiring explicit efficient coding objectives or constraints like noise. This hypothesis stems from the premise that gradual learning algorithms prioritize more prevalent features because they are learned more easily from limited exposure. Consequently, effective learning algorithms introduce a second, implicit constraint on neural coding, complementing the constraint imposed by limited resources. This suggests that efficient learning algorithms intrinsically favor the representation of common features, irrespective of coding resource limitations.

Extensive psychophysical studies have established the non-uniform sensitivity of neural representations to stimuli, exemplified by the oblique effect. The efficient coding hypothesis, a prominent explanation for this phenomenon, proposes that sensory systems prioritize encoding common environmental features due to limited coding resources. Studies in infants and children demonstrate a developmental aspect to sensory sensitivity. Research has shown that visual sensitivities improve with age and experience, even into adolescence, largely depending on visual experience and neural changes outside the retina. This developmental improvement suggests that perceptual sensitivity relies on the neural representation of sensory information and its modification through experience. The research of efficient coding and its relationship to the stimulus statistics and perceptual behavior is extensive; however, a key gap in this research is the lack of a mechanistic explanation of how such codes arise. Deep learning research has highlighted the importance of the learning algorithm in shaping what neural networks learn, showing that they can memorize pure noise. This observation shows that what is learned about inputs is influenced by the algorithm. Several theories exist that explain how large neural networks generalize well to unseen data; however, there is no theory currently regarding how gradient descent learning produces efficient coding.

To test their hypothesis, the researchers investigated whether efficient coding emerges in artificial neural networks trained on visual tasks. They focused on deep convolutional neural networks (CNNs) and Vision Transformers trained on ImageNet, a large-scale image database. The sensitivity of each network layer to variations in oriented Gabor stimuli and hue was measured using the squared magnitude of the change in network activations. To understand the underlying mechanism, they employed a simplified mathematical model: deep linear networks, which lack nonlinearities and allow for a more tractable analysis of learning dynamics. They analyzed how gradient descent biases feature learning towards common input features in this linear system. The researchers also investigated how the learning dynamics of linear networks causes a correspondence between network sensitivity and input statistics. They analyzed the sensitivity of the output to changes in the magnitude of each principal component that makes up an image. For the supervised learning experiments, a three-layer nonlinear neural network was trained to classify the orientation of sinusoidal gratings. Noise was added to the output labels to control the information content of each stimulus. They manipulated the frequency and informativeness of input features independently to assess the effects of frequency and task usefulness on sensitivity. In additional experiments, they investigated the effects of modifying image statistics (rotation, hue shift) during training and analyzed the sensitivity of untrained and weight-shuffled networks to establish the role of training data and network architecture.

The study's key findings demonstrate that artificial neural networks trained on ImageNet exhibit sensitivity patterns similar to those observed in humans and animals, showing increased sensitivity to common visual features (cardinal orientations, common hues) mirroring input statistics. This effect was robust across different network architectures (CNNs, Vision Transformers). The researchers demonstrated that this effect was not due to network architecture alone, by showing that untrained networks or networks with shuffled weights did not show the effect, and that altering the image statistics during training correspondingly changed the sensitivity of the networks. Analyzing deep linear networks allowed the researchers to establish a mathematical framework to explain how gradient descent biases what is learned toward common features. This bias occurs even in over-parameterized, noiseless networks and across supervised and unsupervised learning objectives. The singular values of the weight matrix and the variance of the corresponding principal components of the input data governed the relationship between sensitivity and the frequency of features. A simple linear model of image reconstruction accurately reproduced human-like sensitivity to spatial frequency and the developmental trajectory of visual acuity. Even when the informativeness of input features was balanced through added noise to the labels, gradient descent still preferred more common orientations. This finding indicated the preference for common features was a general property of gradient descent learning, independent of the influence of frequency on the information about labels.

This study provides compelling evidence that efficient coding, a hallmark of biological sensory systems, can emerge naturally from gradient-based learning in artificial neural networks. The findings suggest that the non-uniform sensitivity patterns observed in animals may arise from similar learning dynamics. The research provides an alternative or complementary mechanism to previously proposed explanations for efficient coding, which often focus on local and unsupervised learning objectives. Gradient descent's inherent bias toward frequent, high-variance features explains the correspondence between perceptual sensitivity and environmental statistics. The linear network model, while simplified, offers valuable insights into the behavior of more complex nonlinear networks and captures key qualitative features of human perceptual learning. The linear model effectively demonstrates that the link between sensitivity and input statistics is independent of variations in the task's usefulness. While the analysis of linear networks simplifies a complex phenomenon, it serves as a valuable tool for understanding learning dynamics in more complex systems. Future work may characterize the dynamics of learning in more complete detail, leading to an improved understanding of how this applies to nonlinear networks and the brain.

This research demonstrates that efficient neural codes, characterized by heightened sensitivity to common features, naturally emerge through gradient descent learning. This finding, supported by both empirical results from artificial neural networks and analytical results from linear network models, offers a novel mechanistic explanation for efficient coding in biological systems. The simple linear model successfully replicates key aspects of human perceptual sensitivity and its development. Future studies could investigate the interplay between gradient descent learning, other learning mechanisms, and the role of stochasticity in shaping neural codes. This work also highlights the importance of considering learning dynamics in understanding the optimality of sensory systems.

The mathematical analysis of linear networks provides a simplified model that may not perfectly capture the complexity of learning in nonlinear networks and the brain. Nonlinearity introduces complexities that limit the direct applicability of the linear model's analytical results. While the linear model captures key qualitative features of human perceptual learning, the quantitative aspects may be more nuanced in real biological systems. The studies conducted used primarily artificial stimuli (Gabor patches, sinusoidal gratings) which may not fully capture the richness and complexity of natural visual scenes.