Validating neural networks for spectroscopic classification on a universal synthetic dataset

The study addresses the need for robust, generalizable machine learning models to automate the classification of one-dimensional spectroscopic data (e.g., XRD, NMR, Raman). While these techniques differ in mechanism, their spectra share common features such as peaks with characteristic positions, widths, and intensities, which serve as fingerprints for identification. Existing database-driven identification is complicated by experimental artifacts (noise, background, shifts) and natural variations, motivating machine learning approaches that can outperform similarity-based metrics. Prior neural network applications to spectroscopic classification use diverse, bespoke architectures and report high accuracies on domain-specific datasets, but lack cross-domain validation and systematic analysis of architectural components. The paper’s purpose is to introduce a universal, structure-free synthetic dataset that mimics shared spectral properties across techniques to benchmark multiple architectures fairly, identify general limitations (e.g., overlapping peaks), and distill design principles that improve performance and generalization.

Early CNN applications to spectra classified crystal structures (space groups, Bravais lattices) and phases in XRD, including multi-phase mixtures, and similarly enabled automated identification in NMR and Raman. Architectures often borrowed from image analysis (VGG, Batch-Normalization, residual and inception blocks). Although effective in images, their benefits for 1D spectra remain unproven. Studies frequently trained on simulated data and reported high performance within their domains without cross-architecture comparison. The literature indicates CNNs can remove artifacts and classify spectra, but it is unclear whether observed performance differences arise from architecture choices or intrinsic task difficulty (e.g., Raman vs XRD). This gap motivates a controlled, cross-domain benchmark to assess which components matter for 1D spectroscopic classification.

Synthetic dataset: The authors generated a universal, physics-agnostic synthetic dataset reflecting shared characteristics of XRD, Raman, and NMR spectra. Classes are defined by discrete peaks (positions, intensities), with Gaussian peak shapes and stochastic variations to emulate experimental artifacts. Benchmark configuration: 500 unique classes; per class 60 samples (2–10 peaks in abstract; 2–7 peaks in Methods), with variations in peak position, intensity, and width; 5000 datapoints per spectrum; peaks at least 100 datapoints from spectrum borders. Data splits: 50 training and 10 validation samples per class; an additional blind test set (10 per class) with smaller, uniformly distributed variations to avoid leakage and ensure coverage within training variation ranges. Total: 30,000 training/validation patterns; 4,500 test samples. Training protocol: Eight published CNN-based architectures were implemented unmodified: CNN2, CNN3, CNN6, VGG, CNN BN, Resnet, INC3, INC6. All share a 500-neuron output layer. Training used Adam (lr=3e-4, β1=0.9, β2=0.999), batch size 128, ReduceLROnPlateau (factor 0.5 after 10 stagnant epochs), Early Stopping (25 stagnant epochs). Each model was trained five times with different random seeds to account for initialization sensitivity; deterministic settings and Docker containers were used to improve reproducibility; data shuffling was fixed per seed so models saw identical batches per step. Challenging test dataset: A harder benchmark was constructed with 27 classes arranged in three groups of nine tailored to induce the three observed misclassification modes: (1) overlooked minor peaks, (2) overlapping peak positions, (3) overlapping peak intensities. Models were retrained in two variants (linear vs ReLU activations between fully connected layers) to isolate activation function effects. Architectural ablations: For difficult cases, modified versions removed Batch-Normalization or convolutional dropout in CNN BN and VGG; Resnet was augmented with fully connected layers using ReLU. Additional tests added BN (CNN6 BN) or convolutional dropout (CNN6 ConvDO) to CNN6 to assess their effects. Code and data: All simulation scripts, configs (e.g., dataset_configs/benchmark.json), model implementations, and trained weights are publicly available; training tracked on Weights & Biases; data on figshare.

- On the general synthetic benchmark (500 classes; 4,500 test samples), all eight models achieved at least 98.9% test accuracy (≥4450/4500 correct). - CNN6 achieved the best test performance, averaging 8 misclassifications (of 4500). Most models misclassified 10–40 samples depending on initialization. CNN BN performed worst and showed high variance across runs. - Number of convolutional layers did not correlate strongly with accuracy. Complex image-inspired components (VGG depth, residual blocks, Batch-Normalization) did not improve results over simpler CNNs. Inception blocks (INC3/INC6) yielded modest gains over similarly simple CNNs but did not surpass CNN6. - Training stability: Models with Batch-Normalization (CNN BN, Resnet) showed unstable validation loss trajectories, though Resnet sometimes attained low final loss. Resnet, lacking FC layers and FC dropout, overfit training data more than others and generalized worse than CNNs with FC dropout. - Regularization: CNN6 used high dropout (70%) in FC layers, resulting in more training misclassifications but superior test generalization; other models used 20–50% dropout. - Misclassification analysis revealed three primary failure modes: (1) overlooking minor peaks (e.g., classes 246 vs 382), (2) overlapping peak positions near pooling-induced resolution limits (e.g., classes 8 vs 318), and (3) overlapping peak intensities necessitating intensity-based thresholds (e.g., classes 164 vs 281). - Non-linear activations in FC layers are critical. CNN6 (with ReLU between FC layers) learned appropriate intensity thresholds and clean decision boundaries, while FC-linear models behaved like linear classifiers and failed on intensity-only distinctions. - On the challenging test dataset (27 classes), using ReLU in FC layers improved accuracy across models relative to linear FC activations; CNN6 again performed best, near 90% test accuracy. INC3/INC6 underperformed CNN6 despite non-linear FC. - Convergence issues: Resnet failed to converge on the challenging dataset (accuracy near random) without FC layers; CNN BN improved little due to BN-induced rescaling; VGG performed poorly likely due to convolutional dropout removing peak evidence. - Architectural ablations on the challenging dataset (median accuracies): removing BN or convolutional dropout improved performance. Reported median test accuracies: CNN BN original 52.47% → modified 58.64%; Resnet original 3.70% → modified (with FC+ReLU) 80.74%; CNN6 BN original 67.90% → modified 85.19%; VGG original 68.52% → modified 83.95%; CNN6 ConvDO original 79.63% → modified 85.19%.

The work demonstrates that a structure-free, universal synthetic dataset enables fair, cross-domain evaluation of spectroscopic classifiers and reveals design principles distinct from image classification. Despite uniformly high accuracy on the general benchmark, the challenging dataset exposes meaningful differences: non-linear activations in fully connected layers are essential to learn multiple thresholds, especially for intensity-only distinctions; high FC-layer dropout enhances generalization and mitigates overfitting. Conversely, image-derived components (deep VGG stacks, residual blocks) and Batch-Normalization provided no benefit and often impaired stability or performance, likely because BN perturbs relative intensities critical to spectral discrimination and deep stacks are unnecessary for simple local peak patterns. Misclassifications trace to three interpretable causes—missed minor peaks, near-overlapping positions limited by pooling resolution, and overlapping intensities—clarifying when CNNs struggle. These insights address the central question of which architectural features generalize across spectroscopic domains: prioritize simple convolutional feature extractors, use non-linear FC layers, and apply strong regularization in FC layers rather than BN or convolutional dropout in conv stages. The findings suggest that spectra, as 1D signals with sparse local features, benefit from architectures tailored to preserve and threshold peak evidence rather than deep hierarchical feature extraction.

The paper introduces a fast, customizable synthetic dataset that captures shared properties of XRD, Raman, and NMR spectra and uses it to benchmark eight published CNN-based architectures. All models achieved ≥98.9% accuracy on the general dataset, but performance diverged on a challenging dataset emphasizing peak overlap. The best results consistently came from a relatively simple CNN (CNN6) that employs ReLU between fully connected layers and strong FC-layer dropout, enabling non-linear decision boundaries and improved generalization. Batch-Normalization, residual blocks, inception modules, and convolutional dropout did not confer advantages and sometimes degraded stability or accuracy. The authors release all code, configurations, and trained models to spur further development. Future work includes designing architectures that better attend to minor peaks, mitigating resolution loss from pooling (e.g., alternative downsampling or kernel choices), exploring regularization that preserves sensitivity to small peaks, and extending datasets to include coupled variations, non-Gaussian peak shapes, and backgrounds to narrow the gap to experimental conditions.

- Synthetic spectra use Gaussian peaks without explicit background or noise; artifacts are modeled as independent variations in peak position, intensity, and width, omitting realistic couplings (e.g., strain-position correlations) and non-Gaussian profiles. - Pooling-induced resolution loss can hinder discrimination of closely spaced peaks. - Networks tend to overprioritize largest peaks, leading to errors when only minor peaks distinguish classes; increasing weights for minor peaks risks overfitting under standard regularization. - Results are demonstrated on single-phase spectra; generalization to complex multi-phase or strongly noisy experimental data may require additional validation. - Batch-Normalization instability observations are specific to the tested implementations and datasets; conclusions may vary with alternative normalization or training schemes.