The research question centers around understanding the relationship between dataset size and test error in deep learning, and whether this relationship follows a power-law scaling. The context is the growing use of deep learning across numerous applications, including physics, where the asymptotic behavior of deep learning algorithms is of significant interest. The purpose is to investigate this power-law scaling, determine its parameters, and explore its implications for dataset size estimation, training complexity, and real-time applications. The importance lies in the potential to provide a quantitative framework for comparing machine learning tasks and algorithms, and for optimizing performance in situations with limited data or computational resources, such as rapid decision-making scenarios.

The study draws upon the established concept of power-law scaling from statistical mechanics, citing its prevalence in diverse fields like earthquakes, network topology, turbulence, and brain activity. It also acknowledges the increasing use of deep learning in physics-related data analysis, highlighting the potential connection between the asymptotic behavior of deep learning and critical physical systems. The literature review implicitly establishes a basis for expecting power-law scaling in deep learning, given its presence in other complex systems exhibiting critical phenomena.

The researchers used the MNIST database of handwritten digits for supervised learning. They employed a feedforward neural network with varying architectures (number of hidden layers, presence of input crosses), and two training strategies: a momentum strategy and an accelerated, brain-inspired strategy. The backpropagation algorithm was used to minimize a cross-entropy cost function. The test error was minimized across various parameters for each dataset size and training strategy. Both single-epoch and multi-epoch training were performed, and a soft committee decision method was used to further improve test accuracy. The test error was then analyzed for power-law scaling using a log-log plot, determining the exponent (p) and constant (c0) of the power-law relationship (ε ≈ c0 (dataset size/label)^-p). The study investigated the robustness of the power-law phenomenon across different network architectures, learning strategies, and training epochs. Detailed descriptions of the network architecture, preprocessing steps, and parameter optimization are provided in supplementary appendices.

The key finding is the observation of consistent power-law scaling between test error and dataset size across various conditions. For multi-epoch training with momentum and accelerated strategies, the power-law exponent (p) was around 0.5, with extrapolated test errors near state-of-the-art algorithms. With single-epoch training, the exponent was slightly lower (around 0.48-0.49), but the extrapolated test errors were still reasonably close to multi-epoch results. The exponent (p) increased with the number of hidden layers, indicating that deeper networks might require more data to achieve comparable accuracy. The presence of input crosses significantly improved the test error. For the momentum strategy and multiple epochs, the saturated test error was relatively similar across one, two, and three hidden layers (~0.017), suggesting a potential question about the advantage of deep learning with many hidden layers in this particular context. However, with one epoch, the test error and exponent were strongly dependent on the number of hidden layers, highlighting the importance of dataset size relative to the complexity of the model. The soft committee method consistently improved test error across all conditions.

The observed power-law scaling provides a quantitative framework for comparing the complexity of different machine learning tasks and algorithms. A smaller power-law exponent implies a more challenging task, requiring a larger dataset to reach a given level of accuracy. The findings highlight the potential for rapid decision-making using single-epoch training, achieving test errors close to those obtained with many epochs. This is crucial for applications where real-time performance is critical, such as robotics and network control. The similarity in test error across varying numbers of hidden layers in multi-epoch training raises interesting questions about the optimal network architecture for specific tasks and datasets. The results suggest that the benefits of deep learning (many hidden layers) may be less apparent with limited data and computational resources. The study's findings contribute to a deeper understanding of the fundamental relationships governing deep learning and suggest potential avenues for optimization and improved efficiency.

This study demonstrates the existence and robustness of power-law scaling between test error and dataset size in deep learning. This scaling provides a valuable quantitative framework for comparing the complexity of different learning tasks and algorithms. The ability to achieve near state-of-the-art performance with single-epoch training opens exciting possibilities for real-time applications. Future work could explore the generality of this power-law scaling across other datasets and tasks, investigate the optimal architecture for different learning scenarios and examine further the implications for resource-constrained learning environments.

The study primarily focuses on the MNIST dataset, which may not fully generalize to other datasets with different characteristics. The parameter optimization process is computationally intensive, particularly for the accelerated strategy, which could limit scalability. The investigation of power-law scaling focuses on specific network architectures and learning strategies; exploring other architectures and training methods would broaden the scope of the conclusions.