Evaluating the quality of machine learning (ML) models is crucial, especially in industrial AI where model users often lack access to training or testing data. Traditional methods relying on training/testing error curves provide limited insight and fail to account for factors like robustness, fairness, and interpretability. This limitation is particularly pronounced when model users are not the developers, lacking knowledge of data specifics, loss functions, hyperparameters, and regularization techniques. This study aims to develop a practical theory for evaluating pre-trained models by analyzing their inherent properties, even without access to training data. The lack of standard test metrics in many large-scale applications, such as text generation or user profile clustering, further highlights the need for alternative evaluation methods. High-quality labeled data is expensive, making the ability to assess model quality independent of data availability an important practical and economic consideration. The paper investigates whether it's possible to predict trends in the quality of pre-trained deep neural networks (DNNs) solely by analyzing their weight matrices.

The paper leverages the recently developed Theory of Heavy-Tailed Self Regularization (HT-SR) and traditional norm-based metrics from statistical learning theory. HT-SR Theory analyzes the eigenvalue spectrum of weight matrices to characterize correlations and implicit self-regularization. Norm-based metrics, such as Frobenius and Spectral norms, have been used in traditional statistical learning theory to bound capacity and construct regularizers. However, the application and goals here differ from traditional uses; the focus is not on generalization bounds or new regularizers, but rather on comparing models across different architectures and identifying well-trained versus poorly trained models.

The study employs a large-scale empirical analysis of hundreds of publicly available pre-trained DNN models from computer vision (CV) and natural language processing (NLP). This meta-analysis approach is common in other fields but relatively rare in ML. The analysis utilizes the WeightWatcher tool to analyze individual DNN weight matrices, focusing on properties like norms and power law (PL) exponents of eigenvalues. Four key metrics are considered: Frobenius Norm, Spectral Norm, Weighted Alpha (a weighted average of PL exponents), and α-Shatten Norm. The PL exponent (α) from HT-SR Theory characterizes correlations within weight matrices, with smaller α values indicating better correlation and potentially better model quality. The methodology involves Singular Value Decomposition (SVD) of weight matrices to obtain eigenvalues and subsequent fitting of these eigenvalues to a truncated power law distribution to determine α. For convolutional layers, the authors describe different approaches for applying SVD, opting for the most computationally efficient method to analyze the large number of models. Normalization of empirical matrices is carefully addressed, acknowledging the scale-invariance of PL exponents and the scale dependence of norm-based metrics. Special considerations are given to NLP models which often contain large initial embeddings, potentially lacking the implicit normalization of other layers.

The key findings are summarized as follows: 1. **Norm-based metrics:** These perform reasonably well in predicting quality trends among well-trained models but can give spurious results when applied to poorly trained models, exhibiting what the authors term "Scale Collapse". 2. **Power law (PL)-based metrics:** These significantly outperform norm-based metrics. The Weighted Alpha metric (α) effectively discriminates well-trained versus very well-trained models within the same architecture. The average PL exponent (α) effectively discriminates well-trained from poorly trained models. 3. **CV model analysis:** Analysis of VGG, ResNet, ResNet-1K, and DenseNet series shows strong correlations between PL-based metrics and reported test accuracies. The α-Norm metric is particularly effective. ResNet models, known for efficiency, exhibit consistently smaller and more stable α values across layers, implying strong optimization. DenseNet models, with more connections, display larger and less stable α values, suggesting suboptimal layer optimization. These fine-scale analyses reveal insights into architectural differences and optimization effectiveness. 4. **NLP model analysis:** Analyzing OpenAI GPT and GPT2 models reveals that norm-based metrics behave anomalously in poorly trained models, often exhibiting Scale Collapse. However, PL-based metrics consistently perform well, even on less well-trained models. Comparisons between GPT and GPT2-small (same architecture, different training data) demonstrate the superior ability of PL-based metrics to identify poorly trained models. The analysis of the GPT2 series (small, medium, large, xl) shows expected trends of decreasing metrics with increasing model size, further validating PL-based metrics. The authors also find unexpected trends in the larger GPT2 models, suggesting that these might still have capacity for even larger datasets. 5. **Large-scale analysis:** A broad analysis of hundreds of CV and NLP models confirms the superior performance of PL-based metrics over norm-based metrics across datasets and architectures. While some outliers and imperfect fits are noted, the overall conclusions remain robust.

This study directly addresses the practical challenge of evaluating pre-trained models in the absence of training data. The superior performance of power law-based metrics, particularly the average PL exponent (α), demonstrates the importance of capturing information correlations across multiple size scales within the network. The concept of Correlation Flow, illustrated in layer-wise analysis, and Scale Collapse, observed in poorly trained models, provide new insights into DNN behavior. The findings challenge the traditional reliance on norm-based metrics for model quality assessment, offering a more effective alternative that works well across various model architectures and training conditions. The results contribute to a more practical and comprehensive theory of DNNs, moving beyond idealized toy models to a semi-empirical approach that incorporates rigorous statistical mechanics theory and addresses real-world problems.

The paper presents a novel approach for predicting the quality of pre-trained neural networks without access to training or testing data. The superior performance of power law-based metrics over norm-based metrics has been demonstrated through a large-scale meta-analysis. The concepts of Correlation Flow and Scale Collapse provide valuable insights into DNN behavior. Future research could explore the extension of this framework to other types of neural network architectures and tasks. Further investigation into the relationship between power law exponents and information theoretical measures could deepen our understanding of the information processing capabilities of DNNs. The methodology developed in this paper offers a practical tool for evaluating pre-trained models, especially beneficial in industrial settings where access to training data might be limited.

While the study analyzed a large number of publicly available pre-trained models, the results are limited by the available data. The reported test accuracies were taken as given and not independently verified due to the lack of access to training/testing data. Some of the power-law fits may not be perfectly accurate, particularly for models with unusually large or small PL exponents. This is inherent to the fitting procedure and to the finite size of the models. The study primarily focuses on analyzing weight matrices, potentially overlooking other factors that could influence model quality. Finally, the specific choices made regarding SVD methodology for Conv2D layers might have an impact on the results.