For centuries, scientists have relied on closed-form mathematical models—those expressible with a small number of basic functions—to describe natural phenomena. These models were sometimes derived deductively from first principles, but more often inductively from data. The current abundance of data across diverse fields suggests a potential for inductive discovery of new interpretable models. Recently, machine learning techniques have been developed to automatically uncover such models from data, with successful applications in various scientific domains. However, a key underlying assumption of these methods—that given sufficient data the correct model can always be identified—needs further scrutiny. This research directly addresses this assumption, focusing on scenarios with a relatively small feature space, typical of symbolic regression and model discovery problems. The study investigates the conditions under which a true generating model (m*), generating a dataset D = {(yi, xi)} with added Gaussian noise, can be reliably identified from the data alone. Crucially, the focus is on learning the *structure* of the model (m*), not simply its parameters (θ*). This is a significant departure from much existing theoretical work that primarily concentrates on parameter estimation. The researchers formulate the model identification problem probabilistically, using Bayesian model selection to determine the most likely model given the data.

The paper cites numerous existing works on machine learning approaches to model discovery, noting their successful application in various fields like quantum systems, non-linear dynamics, fluid mechanics, and astrophysics. It also acknowledges the substantial body of theoretical work on parameter estimation, highlighting the relative lack of theoretical understanding concerning model structure learning. Key references include work on Bayesian machine scientists, information-theoretic model selection, and the Minimum Description Length (MDL) principle. The authors position their work within the broader context of the interplay between statistical learning theory and statistical physics, referencing past research on learning parameters and specific problems such as learning probabilistic graphical models and network models. The paper particularly emphasizes the gap in the literature regarding the statistical physics perspective on learning the structure of closed-form mathematical models.

The authors frame the model identification problem probabilistically, aiming to find the posterior distribution p(m|D) over models given the data. This posterior represents the probability that each model m = m(x, θ) is the true generating model. The focus remains on the posterior over model structures, marginalizing over parameter values (θ). The posterior is expressed in terms of the model likelihood p(D|m, θ), prior distributions over model parameters p(θ|m), and model structures p(m). They use the Bayesian Information Criterion (BIC) to approximate the posterior, relying on assumptions about the likelihood and prior distributions. This approximation simplifies calculation, particularly in regression problems. The BIC approximation, interpreted information-theoretically, reflects the description length—the number of bits needed to encode both the data and the model. The minimum description length (MDL) model is considered the most plausible one. The posterior distribution over models p(m|D) is then sampled using a Markov chain Monte Carlo method with the Metropolis algorithm, employing a Bayesian machine scientist. The MDL model—the model with the minimum description length—is selected from the sampled models. To evaluate generalization performance, synthetic datasets are generated using a known model m', with varying numbers of data points N and noise levels σe. The MDL model's predictive accuracy on a separate test dataset D' is compared to that of an artificial neural network (ANN), providing a benchmark for comparison. The learnability of a model is assessed by examining whether the true model is the most plausible one in an ensemble of sampled models, based on comparing description lengths.

The study's key findings demonstrate that probabilistic model selection, employing the MDL principle, yields quasi-optimal generalization performance in both low- and high-noise regimes. In the low-noise phase, the MDL model accurately identifies the true generating model, achieving near-perfect interpolation. Conversely, in the high-noise phase, the prediction error is predominantly determined by the observation noise, and the MDL model's predictions are comparable to those of other methods. However, a transition region exists between these two phases where generalization becomes challenging for all methods, including probabilistic model selection. The research establishes the existence of distinct learnable and unlearnable phases. The transition between these phases is characterized by a peak in the prediction error relative to the irreducible error. The authors derive an upper bound for the noise level at which this learnability transition occurs, based on the competition between the description lengths of the true model and the most plausible trivial models (e.g., constant models). This bound provides a reasonable approximation of the actual transition point, particularly for larger datasets. The learnability curves, when scaled by the observation noise, collapse onto a single curve, suggesting potential universality in the transition behavior. As the dataset size N increases, the transition becomes more abrupt, indicating a possible discontinuous phase transition.

The findings highlight the inherent limitations in learning closed-form mathematical models from data, particularly in the presence of noise. The phase transition identified demonstrates that successful model learning is not simply a matter of increasing data quantity; noise level plays a crucial role. The success of probabilistic model selection in both low- and high-noise regimes, contrasted with the limitations of standard machine learning techniques (like ANNs) in the low-noise region, underscores the importance of principled probabilistic approaches for model discovery. The existence of a challenging transition region, where generalization is difficult, emphasizes the non-trivial nature of the model learning problem. The paper's results offer insights into the interplay between data size, noise, and the complexity of the model itself. The upper bound on the noise for the learnability transition provides a practical tool for assessing the feasibility of learning models from data. The work establishes a connection between model learning and phase transitions, a theme seen in other statistical physics problems. This theoretical contribution opens avenues for more research in this area.

This research provides a rigorous theoretical framework for understanding the limits of learning closed-form mathematical models from noisy data. The demonstration of a learnable-unlearnable phase transition, and the development of an upper bound for the transition noise, are significant contributions. The superior performance of probabilistic model selection, particularly in the low-noise regime, underscores its importance in model discovery. Future research directions include investigating more complex models, exploring different noise structures, and further characterizing the behavior in the transition region. The connection to phase transitions suggests that tools and insights from statistical physics could offer further advancements in machine learning for scientific model discovery.

The study focuses on relatively simple, closed-form mathematical models and Gaussian noise. The applicability of the findings to more complex models or non-Gaussian noise distributions remains to be investigated. The BIC approximation, while effective in many cases, introduces some level of approximation that could affect the results, particularly for small datasets. The assumption that the most relevant minima in the description length landscape are those of the true and trivial models may not always hold, potentially impacting the accuracy of the learnability transition bound.