Drug classification with a spectral barcode obtained with a smartphone Raman spectrometer

The study addresses the need for portable, accurate spectroscopic tools capable of material identification using ubiquitous smartphones. Conventional smartphone-based spectrometers typically rely on gratings in external modules, which limit miniaturization. Alternative on-sensor approaches (e.g., photonic crystals, metasurfaces, quantum dots, nanowires) require complex numerical reconstruction and struggle with weak, high-resolution Raman signatures. Concurrently, the proliferation of online pharmacies has increased the risk of counterfeit or substandard drugs, and current pill identification apps relying on name, shape, color, and etched marks often fail due to similar appearances or missing database entries. Raman spectroscopy provides molecular fingerprints suitable for drug identification, but integrating it into a compact smartphone platform with robust analysis remains challenging. This work proposes a smartphone Raman spectrometer that creates a two-dimensional spectral barcode directly on the phone’s CMOS image sensor via a periodic band-pass filter array, coupled with on-device CNNs to classify drugs by major chemical component and brand name. The objectives are to demonstrate high-accuracy classification across widely used drugs and to show generalization to unseen brands sharing known active ingredients.

Prior smartphone spectrometers predominantly use gratings in external modules, offering high spectral resolution but limiting form factor. Miniaturized spectrometers using photonic crystals, metasurfaces, quantum dots, and silicon nanowires integrated with CCD/CMOS sensors have been reported; however, due to low Q-factors or complex response functions, they require computational spectrum reconstruction and are limited for weak, high-resolution Raman signals. Pill identification apps based on visual attributes (shape, color, imprint) provide insufficient accuracy. Raman-based drug classification has been explored using PLS-DA, PCA, CNN, SVM, and PLS-DA across tasks such as identifying pharmaceutical ingredients, newly emerging psychoactive substances, and illicit drugs—even in complex matrices like urine and fingermarks. These studies establish Raman’s suitability for chemical identification, but do not provide a compact, smartphone-integrated solution capturing 2D spectral data directly on the image sensor and performing on-device AI classification.

Instrument design: A smartphone Raman spectrometer was built on a Samsung Galaxy Note 9 rear-wide camera by integrating a two-dimensional periodic array of 128 channels (CHs) at the image sensor, consisting of 120 distinct narrowband Fabry–Perot band-pass filters (830–910 nm) and 8 metal-blocked CHs (position markers) arranged in a 16×8 mosaic. Each filter uses Si/SiO2-based DBRs with a central Si cavity; FWHM 1.0–1.2 nm, transmission 0.45–0.61, and inter-channel spacing Δλ <1.2 nm (2 nm for the two longest wavelengths). The sensor resolution is 3024×4032 pixels, with 40×40 pixels per CH. An external, compact, attachable Raman module houses a 785 nm laser diode (Thorlabs LD785-SEV300), optical collection, baffling to block ambient/scattered light, thermal management, and an external rechargeable battery power source. No additional electronics interface to the phone is required.

Spectral barcode acquisition and preprocessing: Samples were placed at the objective lens focal position (aperture ~0.5 mm). Raman emissions illuminate a 2×2 array of CH sets near the sensor center; data from these four sets of 128 CHs are extracted. Denoising is performed by averaging 20×20 pixels within each CH; the four sets are averaged and normalized to form a spectral barcode—a 2D map encoding transmitted Raman intensities at fixed wavelengths. Exposure time per acquisition was 10 s to capture weak Raman signals and reduce readout noise. Reference Raman spectra of 58 drugs were acquired with a commercial spectrometer (785 nm excitation; XperRam, Nanobase) for comparison.

Dataset: 54 commonly used drugs covering hypertension (amlodipine, losartan, valsartan), diabetes (glimepiride, metformin), hyperlipidemia (atorvastatin, rosuvastatin, simvastatin), and over-the-counter (vitamin B6, vitamin C, acetaminophen) were used for CNN training/testing; four additional drugs (Glimel 3 mg, Dymit XR, Glucophase 1000 mg, Metofol 500 mg) were held out to test generalization. For each drug, 170–223 Raman images and 198–202 RGB images per side were collected. Data splits were mutually exclusive across training, validation, and testing. For spectral barcodes, approximately 100 images per drug were used for training, 35–67 for validation, and 35–67 for testing. For RGB, roughly 118–122 images per side for training and 40 each for validation and testing were used. For the four excluded drugs, 140–143 Raman images per drug were used to assess expandability. RGB images were taken under room light on black background; preprocessing included denoising, contour extraction, background removal, resizing, and color normalization, with augmentation (position, angle, size).

Neural network architectures: For major component classification (11 classes), a simplified ResNet with one residual block (3×3 conv, batch normalization, shortcut add, ReLU) is used; after flattening, two fully connected layers (batch normalization with ReLU, then batch normalization with softmax) output class probabilities. For brand identification, a cascaded approach is used: one CNN predicts major component, followed by nine separate CNNs of similar architecture to identify brand within each component class (vitamins B6 and C excluded). For appearance-based classification from RGB images, a VGGNet-style CNN with three convolutional layers (batch normalization, ReLU, max pooling), flattening, and three fully connected layers (last with softmax) classifies pill shape (snowman, circle, ellipse, pentagon/octagon) and color (blue, yellow, green, white, pink) and supports brand identification. Final brand prediction combines outputs from spectral-barcode CNN and RGB CNN by multiplying their output probabilities (equal weighting). Overfitting was managed by monitoring training/validation loss, optimizing network depth/parameters, and using batch normalization to mitigate vanishing gradients.

Operational notes: Random placement on the objective ensured variability; slight defocus or etched-mark scattering could alter background intensity. The system’s spectral resolution is lower than benchtop Raman, but high Q-factor filters enable resolving key bands. The entire pipeline (acquisition to on-device inference) runs on the smartphone application processor via a custom Android app (C#), with demonstration at 5 s integration in a supplementary video.

• The smartphone Raman spectrometer produces a 2D spectral barcode that captures sufficient Raman information to classify drugs by major components with high accuracy.
• Major component classification across 54 drugs (11 components: amlodipine, losartan, valsartan, glimepiride, metformin, atorvastatin, rosuvastatin, simvastatin, vitamin B6, vitamin C, acetaminophen) achieved 99.0% overall accuracy. Some classes (valsartan, vitamin B6, vitamin C, Tylenol/acetaminophen) achieved 100% correct classification in tests.
• Generalization to unseen brands: Four drugs excluded from training (Glimel 3 mg—glimepiride; Dymit XR, Glucophase 1000 mg, Metofol 500 mg—metformin) were correctly identified by major component with 99.8% accuracy (one error among 424 metformin trials; 141 glimepiride trials all correct).
• Brand identification using spectral barcodes alone reached 79.5% accuracy; integrating RGB image-based CNN (shape and color) increased brand accuracy to 83.2% (±0.2%p over 10 runs of 1000 random tests).
• Despite lower spectral resolution than grating-based spectrometers, the filter array’s FWHM of 1–1.2 nm and narrow Δλ allowed resolving closely spaced Raman bands; smartphone-based acquisition provided adequate SNR and Q for classification with lower power consumption than CCD-based systems.

The results demonstrate that a smartphone-integrated Raman spectrometer with an on-sensor band-pass filter array can generate robust 2D spectral barcodes sufficient for accurate, on-device AI classification of pharmaceuticals. This approach directly addresses limitations of appearance-based pill identification systems by exploiting molecular fingerprints rather than visual features and mitigates miniaturization constraints of grating-based designs. The high major-component accuracy (99.0%) and successful identification of unseen brands by their active ingredients indicate strong generalization when the component class exists in the database. Brand-level differentiation, while inherently harder due to subtle additive/coating differences, benefits from broader spectral coverage and multimodal fusion with RGB images, which raised accuracy to 83.2%. The compact external Raman module, minimal power needs, and full on-phone workflow enhance portability and usability, suggesting wide applicability in healthcare settings to combat counterfeit drugs and prevent medication errors. Broader spectral range targeting additive bands (e.g., ~810–830 nm) and improved sensor specs (SNR, Q-factor) could further boost brand discrimination.

This work introduces the spectral barcode concept realized by a smartphone Raman spectrometer using a 2D band-pass filter array on the CMOS image sensor and embedded CNNs for analysis. The system accurately classifies 11 drug major components (99.0% accuracy) and identifies brands with up to 83.2% accuracy when fusing spectral and RGB information. It generalizes to unseen drug brands by recognizing their active ingredients. The approach offers a compact, low-power, and practical solution for point-of-need drug identification, with potential to enhance public health and supply-chain integrity. Future improvements include expanding spectral range to capture additive signatures, increasing filter density and dynamic range, enhancing SNR and Q-factor, leveraging additional visual cues (imprints), and exploring techniques like SORS and SERDS to handle coatings and fluorescence. Longer term, shrinking channel size and increasing array density could enable smartphone hyperspectral imaging for broader applications.

• Spectral resolution is lower than benchtop grating spectrometers, causing slight peak shifts and broader FWHM; some closely spaced features are less distinct.
• Brand identification remains imperfect (79.5% with spectral barcodes; 83.2% with RGB fusion) due to subtle differences from additives/coatings and limited spectral coverage (830–910 nm).
• Acquisition conditions (focal variation, etched-mark scattering) can affect background intensity, introducing variability.
• Identification of unknown drugs relies on the presence of their major component class in the training database.
• Fluorescence/background can obscure Raman features; advanced methods (SORS, SERDS) may be needed for coated tablets or strong fluorescence.
• Dataset size per class is modest compared to typical AI datasets, though sufficient here; broader generalization across more brands and formulations needs larger, diverse datasets.