Anthropogenic aerosols exert a cooling effect on Earth's climate, offsetting a portion of greenhouse gas warming. Despite advancements in understanding aerosol-climate interactions, uncertainties in aerosol forcing persist, partly due to insufficient observational constraints on crucial aerosol properties used in Earth System Models (ESMs). While past satellite and suborbital measurements have provided valuable insights, they lack the accuracy, spatiotemporal resolution, and coverage needed to significantly reduce uncertainties in ESM simulations. A particular challenge lies in accurately measuring the vertical distribution of key aerosol properties, especially in remote atmospheres and near clouds susceptible to aerosol-induced modifications. This paper introduces a novel ML-based paradigm to address this observational gap, focusing on retrieving higher-level aerosol properties—aerosol light absorption (ABS) and cloud condensation nuclei (CCN) concentrations—from lidar observations. This approach is particularly valuable because active lidar measurements are less affected by artifacts that hinder passive remote sensing near clouds. The paradigm leverages existing high-accuracy lidar and in situ measurements to train ML algorithms, enabling the application of these models to future satellite lidar data, such as those from the recently launched EarthCARE satellite and the upcoming NASA Atmosphere Observing System (AOS). The approach offers a substantial improvement over traditional physics-based retrieval methods which are computationally expensive and limited by uncertainties in underlying electromagnetic scattering processes and a priori assumptions.

Previous research highlights the limitations of traditional physics-based aerosol retrievals. These methods rely on forward models describing complex interactions between electromagnetic radiation and aerosols, the Earth's surface, and the atmosphere. Challenges include uncertainties in the details of electromagnetic scattering processes (especially with non-spherical particles and polarized surface reflectance), propagation of uncertainties from a priori assumptions, and high computational costs. Past studies have shown that linear regression models between lidar-derived aerosol properties and CCN are aerosol-type specific and only work within limited relative humidity (RH) ranges due to non-linear responses of aerosol optical properties to RH increases. This limitation underscores the need for more sophisticated methods capable of capturing these complex relationships. The current study builds upon this research by utilizing ML to overcome these limitations and provide more accurate and comprehensive retrievals of ABS and CCN.

This study employs a machine learning approach using fully connected neural networks (FCNNs) to predict aerosol light absorption (ABS) and cloud condensation nuclei (CCN) concentrations. The core dataset consists of high-accuracy lidar measurements from the NASA Langley HSRL-2 system, which provides independent measurements of aerosol backscattering, depolarization, and extinction at multiple wavelengths. These lidar measurements were collocated with in situ observations of ABS and CCN from four airborne field campaigns conducted across diverse geographical locations (continental US, Southeast Atlantic Ocean, Philippines, and US Atlantic coast). The campaigns captured a wide range of aerosol types and environmental conditions, including dust, smoke, marine, and pollution aerosols. The in situ ABS measurements are from Particle Soot Absorption Photometers (PSAP), while CCN concentrations were obtained from continuous-flow CCN counters, with the majority of CCN data measured at supersaturations of 0.35-0.4%. The researchers used 70% of the data for training, 15% for validation, and 15% for testing. Atmospheric reanalysis data (temperature and relative humidity from ERA5) were incorporated as additional predictors to further constrain the ML predictions. The study included two analyses. The first evaluated the performance of ML models trained with and applied to the full suite of HSRL-2 observations. The second evaluated the potential performance of models trained with a subset of observables anticipated for the EarthCARE/ATLID lidar system (UV observations at 355 nm), simulating real-world satellite data by adding noise to the HSRL-2 UV data to account for the lower signal-to-noise ratio of spaceborne lidars. The Levenberg-Marquardt algorithm was used to train the feedforward neural network regression model, with hyperparameters tuned iteratively using Bayesian optimization. Ten-fold cross-validation was employed to assess model performance. In response to reviewer comments, data weighting was explored to enhance model accuracy in data-sparse regions (clean conditions), particularly affecting CCN estimations.

The ML algorithms demonstrated high accuracy in predicting both ABS and CCN. Using the complete HSRL-2 dataset as input, the models achieved correlations of 0.93 (CCN) and 0.80 (ABS) without reanalysis data, improving to 0.97 (CCN) and 0.90 (ABS) with reanalysis data. Mean relative errors were 22% (CCN) and 25% (ABS) without reanalysis data, reduced to 13% (CCN) and 21% (ABS) with reanalysis data. When applied to simulated EarthCARE/ATLID data (UV-only with added noise), the correlations were lower (0.57 for CCN and 0.55 for ABS without reanalysis data; 0.88 for CCN and 0.74 for ABS with reanalysis data), but still represented substantial improvements over existing physics-based retrieval methods, especially for CCN. Mean relative errors for simulated ATLID data were 51% (CCN) and 40% (ABS) without reanalysis data, reduced to 23% (CCN) and 28% (ABS) with reanalysis data. With reanalysis data, the HSRL-2-based model provided vertically resolved CCN within 30% uncertainty in 85% of retrievals, and the simulated ATLID model provided CCN within 50% uncertainty in 83% of retrievals. These results far exceed the uncertainty of a factor of 2 reported in previous studies using CALIPSO lidar measurements. A curtain plot of CCN derived from HSRL-2 data demonstrated the high spatial resolution and accuracy achievable with this ML approach.

The ML-based retrieval paradigm offers significant advantages over traditional physics-based methods, particularly in providing accurate, vertically resolved aerosol properties, especially near clouds. The high accuracy achieved using the complete HSRL-2 dataset highlights the potential of this approach for improving aerosol observations from both past and future airborne lidar missions. The results from the simulated ATLID data provide a realistic assessment of the capabilities of this method when applied to future spaceborne lidar observations. While this is a simplistic simulation, it shows that this method could still significantly reduce uncertainty. This method is not intended to replace physics-based retrievals; instead it augments them, offering a computationally efficient approach in situations where physics-based methods fall short. The study suggests that independent measurements of aerosol extinction and backscatter, like those obtained from HSRL and Raman lidar systems, are key to the success of this ML approach. The application of this method to future spaceborne lidar observations promises to significantly improve understanding of the global distribution of aerosol properties and their role in climate forcing.

This study introduces a novel machine learning paradigm for retrieving higher-level aerosol properties (ABS and CCN) from lidar observations. The results demonstrate significantly improved accuracy and spatial resolution compared to existing physics-based retrieval methods. This paradigm represents a substantial advance in aerosol remote sensing, with potential to significantly reduce uncertainties in aerosol climate forcing estimates. Future research should focus on applying this approach to actual satellite data, investigating the incorporation of additional predictors, and exploring other ML algorithms to further optimize model performance.

The study’s limitations include the reliance on a specific high-accuracy lidar system (HSRL-2) for training data and the simplistic simulation of EarthCARE/ATLID error characteristics. The generalizability of the model might also be affected by the representativeness of the training dataset, although efforts were made to incorporate data from various geographical regions and aerosol types. While data weighting improved CCN predictions in clean conditions, additional work could focus on more sophisticated data weighting techniques and further exploration of the effects of data sparsity. The study also acknowledges the lack of insight into the underlying physical processes provided by ML models, which is a tradeoff for the enhanced accuracy and computational efficiency.