Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

The paper addresses the problem of evaluating the quality of pretrained neural network models when training and test data, and often details of training such as loss functions and hyperparameters, are unavailable. Traditional evaluation via training/testing errors offers limited insight, ignores hyperparameter search costs, and often does not correlate with properties like robustness or fairness. In industrial settings, model users may be distinct from developers and may lack access to data; moreover, some tasks lack clear test metrics (e.g., generative models or unsupervised clustering). The central research question is whether one can predict trends in the quality of pretrained deep neural networks (DNNs) without any access to training or testing data. The authors propose using weight-matrix-based diagnostics—norm-based capacity metrics and power-law (PL) metrics from Heavy-Tailed Self-Regularization (HT-SR) Theory—to evaluate and compare pretrained models across architectures and to distinguish well-trained from poorly trained models.

The work builds on two strands: (1) norm-based capacity control and generalization bounds in statistical learning theory, where matrix norms (e.g., spectral, Frobenius) inform regularization and capacity; and (2) statistical mechanics/RMT-based analyses of DNNs, specifically Heavy-Tailed Self-Regularization (HT-SR) Theory, which models the empirical spectral density of layer correlation matrices with truncated power laws and links smaller PL exponents to stronger multi-scale correlations and better generalization. Prior HT-SR studies examined a small number of models (e.g., AlexNet, Inception V3) and observed PL behavior with exponents typically in (1.5, 5). Related work includes information bottleneck analyses (requiring data access), and contemporaneous studies (2020) also predicting model accuracy from weights but on smaller, narrower sets of models. The authors position their contribution as a large-scale, practical, semi-empirical theory-driven meta-analysis across state-of-the-art CV and NLP models.

Design: A statistical meta-analysis of hundreds of publicly available pretrained DNNs (primarily CV and NLP), without access to training or test data and without reproducing training. Reported test accuracies/errors from original sources are taken as given. Data/Models: CV series (VGG, ResNet, ResNet-1K, DenseNet), an additional ResNet20/CIFAR10 distillation case, and NLP series (OpenAI GPT and GPT-2 small/medium/large/xl). Broader analysis spans 100+ models across multiple datasets. Tooling: WeightWatcher (v0.2.7) to extract and analyze layer weight matrices, compute norms, and fit PL exponents; Jupyter/Colab notebooks provided publicly. Layer extraction: For linear/FC/attention/Conv1D layers, use the NxM weight matrix W. For Conv2D layers (4D tensors), use pre-activation maps W_ij (method 1) and perform SVD per map to obtain singular values/eigenvalue spectra; rescale Conv2D maps by k/√2 for comparability. Bias terms are ignored. Normalization: Empirical matrices used as-is (reflecting implicit initialization scales); embedding layers in NLP models are treated carefully due to differing scale (often excluded from averages because of anomalously large eigenvalues). Metrics: - Log Frobenius norm: average over layers of log(||W||_F^2) up to constant factors. - Log Spectral norm: average over layers of log(||W||_2^2) (or log λ_max). - Weighted Alpha (â): per-layer PL exponent α (from fitting ESD of X=W^T W to truncated PL ρ(λ) ~ λ^α on [λ_min, λ_max]) weighted by log λ_max, then averaged over layers: â = (1/L) Σ_i α_i log λ_max,i. - Log α-Norm (log ||X||_α): generalized Schatten-α norm proxy aggregated as log ||X||_α = Σ_i log ||X_i||_α with layer-varying α; approximates a weighted log spectral norm for small α. PL fitting: Maximum Likelihood estimation for truncated PLs (reliable for α in ~[2,4], less precise outside); λ_min chosen to minimize K–S distance; α is scale-invariant, λ_max captures scale. Analyses: - Correlate average metrics with reported test accuracy/error within architecture series via linear regression; report RMSE, R^2, Kendall-τ. - Layer-wise profiles of α and λ_max across depth to study Correlation Flow (stability/propagation of correlations) and detect Scale Collapse (anomalously small/large norms concentrated in subsets of layers). - Diagnostics by decomposing â into α and log λ_max distributions to distinguish correlation strength from scale effects, especially for poorly trained or distilled models.

- Overall: Norm-based metrics (log spectral, log Frobenius) correlate well with reported accuracies among well-trained models but can be misleading for poorly trained or otherwise problematic models (e.g., undertrained, distilled), where Scale Collapse appears. PL-based metrics (â, average α, log α-norm) perform better for discriminating both within-series quality trends and between well-trained and poorly trained models. - VGG series (ImageNet, PyTorch v1.4): All four metrics correlate with Top-1 accuracy; Log Spectral norm shows strongest quantitative fit (e.g., R^2 ≈ 0.98, RMSE ≈ 0.23; figure reports Kendall-τ ≈ -0.93). Log α-Norm also strong (R^2 ≈ 0.96, RMSE ≈ 0.34). Log Frobenius weaker but still good (R^2 ≈ 0.88, RMSE ≈ 0.56). - DenseNet series: Similar behavior to VGG; norm and PL metrics correlate with accuracy (plots referenced in repo). - ResNet vs ResNet-1K: Log α-Norm correlates strongly for full ResNet (trained on full ImageNet): RMSE ≈ 0.66, R^2 ≈ 0.95, Kendall-τ ≈ -1.0. Correlation weaker for ResNet-1K (trained on smaller ImageNet-1K): RMSE ≈ 2.0, R^2 ≈ 0.86, Kendall-τ ≈ -0.88. Weighted Alpha slightly outperforms Log α-Norm in some cases. - Layer-wise Correlation Flow (CV): ResNet-152 exhibits smaller, more stable α across layers (often near ≈2), indicating strong, well-propagated correlations; VGG shows gradual worsening (larger α) with depth; DenseNet’s α is more erratic and larger, suggesting over-connectedness diluting high-quality interactions. - Distillation case (ResNet20 on CIFAR10, Group Regularization): Despite slightly worse Top-1/Top-5 accuracy after distillation, layer-wise log spectral norms unexpectedly increase in baseline vs. fine-tuned for two Conv2D layers (indicative of spurious scale effects); α values are more consistent and indicate the fine-tuned model compresses problematic layers (baseline shows α ≥ 6 outliers). This highlights that norm metrics can give spurious indications, while PL metrics reveal true correlation structure. - NLP (GPT vs GPT-2 small): Poorly trained GPT shows many anomalously small spectral norms (Scale Collapse) and larger α values with outliers (α > 6), indicating under-correlation/over-parameterization. GPT-2 small has larger spectral norms overall and smaller α distribution (all α ≤ 6), revealing better training. Aggregated averages (excluding embeddings) from Table 2 illustrate that all four average metrics increase from GPT to GPT-2 small (e.g., average log spectral norm 1.72→2.54; â 7.01→9.62), which is counter to “smaller is better” and explained by scale differences and collapse; decomposing â into α and log λ_max clarifies the diagnosis. - NLP (GPT-2 medium/large/xl): For well- and very-well-trained models, metrics trend as expected with increasing size: average log spectral norm decreases (≈2.58→1.99→1.92) and Log α-Norm decreases (≈9.96→7.98→7.49); average α is low and similar (≈3.82, 3.97, 3.81), with means/peaks moving towards ≈2. However, right tails of α distributions grow with size, suggesting remaining under-optimization and capacity for larger datasets. - Large-scale cross-architecture analysis (hundreds of models): Across diverse CV datasets/architectures, PL-based metrics yield better or comparable R^2 with substantially better RMSE and variability relative to norm-based metrics, despite outliers and imperfect fits, supporting broader generality. - Concepts: Correlation Flow (stable, small α across layers) associates with better models; Scale Collapse (abnormally small or heterogeneous norms across layer subsets) is a red flag for poorly trained/distilled models and can confound norm-based assessments.

The findings demonstrate that it is indeed possible to predict trends in pretrained model quality without access to any training or test data, by analyzing weight matrices. Norm-based metrics can rank well-trained models within a series, but they may fail to distinguish poorly trained models due to scale effects (Scale Collapse). PL-based, scale-invariant metrics (α) and their scale-aware aggregations (â, log α-norm) more robustly capture the strength and propagation of correlations across layers. Layer-wise α profiles reveal architectural differences: ResNet’s residual design fosters stable correlation flow (small, stable α), whereas VGG degrades with depth and DenseNet exhibits erratic α, aligning with known performance/efficiency differences. In NLP, attention-heavy models have larger matrices and different spectra; diagnosing GPT vs GPT-2 shows how α distributions and scale decomposition identify undertraining and improvements from better data/optimization. Collectively, these results inform quality prediction, model diagnostics (identifying under-optimized layers), and evaluation of enhancements (distillation, fine-tuning) without data access, advancing a practical, semi-empirical theory of DNNs grounded in statistical mechanics of heavy-tailed systems.

The paper introduces a reproducible, data-free methodology to assess and compare pretrained DNN quality using weight-matrix meta-analysis informed by HT-SR Theory. Key contributions include: (1) evidence that norm metrics predict accuracy trends for well-trained models but can mislead for poorly trained ones; (2) demonstration that PL-based metrics, especially average α (for distinguishing well- vs poorly trained) and Weighted Alpha/log α-norm (for ranking within architecture series), perform better across CV and NLP; (3) layer-wise diagnostics revealing Correlation Flow and Scale Collapse, enabling identification of problematic layers and evaluation of training enhancements; and (4) broad validation across hundreds of models. Future directions include refining PL estimation for extreme α, improving handling/normalization of special layers (e.g., embeddings), extending diagnostics to more architectures and tasks, integrating metrics into automated model validation/compression pipelines, and exploring training strategies that explicitly promote stable, low α across depth.

- Reliance on reported test accuracies/errors from external sources (no reproduction) may introduce inconsistencies. - PL exponent estimation via truncated power-law MLE is most reliable for α in ~[2,4]; estimates for α > 6 (common in under-trained layers) are less precise, though relative comparisons remain informative. - Conv2D spectral analysis uses a fast approximation (SVD per pre-activation map), not the more exact but expensive FFT-based approach. - Norm-based metrics are sensitive to scaling/normalization; NLP embedding layers can dominate unless treated specially. - Linear regression fits show outliers and imperfect trends across diverse architectures/datasets; metrics predict trends rather than exact accuracies. - The approach diagnoses model quality and training pathologies but does not directly provide causal mechanisms or guarantees of downstream robustness/fairness.