Retrosynthetic reaction pathway prediction through neural machine translation of atomic environments

The study addresses the long-standing challenge of planning reaction pathways in organic synthesis by proposing an efficient, interpretable molecular representation and learning framework for single-step retrosynthesis. Conventional methods are either template-based, relying on reaction rules and atom mapping, or template-free, typically using graph- or sequence-based models with SMILES strings. SMILES-based sequence models, while successful, suffer from fragile syntax and invalid outputs, and often learn token rearrangements rather than chemically meaningful transformations. The authors hypothesize that representing molecules as sets of chemically meaningful atom environments (AEs) allows a neural machine translation model to focus on local changes around reaction centers, improving accuracy, robustness, and interpretability for retrosynthetic prediction. The purpose is to develop RetroTRAE, a Transformer-based model that translates product AEs to reactant AEs, alleviating SMILES grammar issues, removing the need for atom mapping, and enhancing explainability by aligning predictions with reaction-center changes.

The paper situates retrosynthesis prediction within two main paradigms: template-based approaches that apply extracted or expert-curated reaction rules, and template-free methods that learn end-to-end mappings. Template-based methods face challenges with template coverage, specificity vs. generality trade-offs, computational costs for matching, and reliance on accurate atom mapping. Template-free methods divide into graph-based and sequence-based models, with many state-of-the-art predictors built on the Transformer architecture. Prior SMILES-based models (e.g., Molecular Transformer and subsequent works) achieved strong performance but are hampered by syntactic fragility (invalid SMILES) and potential semantic issues (valence/kekulization). Efforts to mitigate include syntax correctors (e.g., SCROP), alternative syntaxes (DeepSMILES, SELFIES), and data augmentation/tokenization tweaks. The authors’ prior work showed fragment sets as effective inputs, motivating this study’s use of atom environments as tokens to capture chemically meaningful local structures and reaction-center changes more directly than SMILES token rearrangements.

Representation: Molecules are represented as sets/lists of atom environments (AEs), which are circular topological fragments centered on an atom, defined by a radius r as the maximum topological distance. AE0 (r=0) encodes central atom information; AE2 (r=1) includes the central atom, its nearest neighbors, and bonds among them. AEs are generated via RDKit’s ECFP procedure and expressed as SMARTS patterns; they are rotation/translation invariant. AE4 (r=2) yields a prohibitively large vocabulary and is not used directly for translation. Fragmentation schemes evaluated: AE0, AE2, and hashed ECFP variants (ECFP0, ECFP2, ECFP4; 1024-bit). A word-based tokenization (SentencePiece) is applied to AE SMARTS and to ECFP bit indices. Model: RetroTRAE is a Transformer-based encoder–decoder NMT model that translates product AE sequences into reactant AE sequences. The architecture follows the original Transformer but with pre-layer normalization before self-attention, multi-head attention, and feed-forward layers; encoder and decoder outputs are normalized. Positional encodings are included and found beneficial despite AE permutation invariance. Datasets: The filtered USPTO-Full dataset is used via Zheng et al.’s canonicalized splits, yielding 480k atom-mapped reactions after cleaning. The study focuses on single-product reactions (97%, 465k), omits multi-component cases (<1.65%), and constructs forward reaction sets leading to curated unimolecular (R→P) and bimolecular (R1+R2→P) datasets totaling 414k samples (used in backward/retrosynthesis training). Sequences are capped at 100 fragments. No atom-to-atom mapping is used in model training (AEs do not require it). For retrieval and coverage analyses, PubChem (111M molecules) and ChEMBL are used. Training: Data are split 90/10 into train/test; 10% of training is held out for validation. Models are trained with Adam optimizer and negative log-likelihood loss. Dropout rate is 0.1. Batch size is 300; training speed is ~12 minutes per epoch on a single Quadro RTX 8000 GPU. RDKit 2020.03.1 and PyTorch are used. The study also employs 10× data augmentation for uni- and bi-molecular reactions to improve stability and performance. Evaluation: Top-1 prediction accuracy is emphasized as a primary metric. Similarity-based metrics include Tanimoto (T) and Sørensen–Dice (S) coefficients (special cases of the Tversky index). Two accuracy regimes are reported: hard thresholds (single mutation, SM; double mutation, DM) counting predictions differing by one or two AE fragments; and soft thresholds using similarity cutoffs (e.g., T≥0.85, T≥0.80). Statistical significance of similarity scores is assessed using the ccbmlib package. Attention weights are analyzed to interpret learned focus on reaction-center AEs. Retrieval of predicted AE sets to real molecules is performed via database searches against PubChem, assessing exact and near-exact reconstruction under SM/DM allowances.

- Fragmentation comparison: Transformer with union AE0∪AE2 outperforms other schemes on unimolecular test set, achieving 55.4% exact matches; adding bioactively similar predictions (Tanimoto ≥0.85) increases to 68.1%. MACCS performs poorly (low resolution); ECFP2 is competitive; ECFP4 underperforms due to high bit-collision rates. - Optimal radius analysis: ECFP4 bits correspond to many unique AEs (often 100–160 per bit), whereas ECFP2 has far fewer (≤20), complicating learning and degrading performance; thus radius 1 (AE2) is optimal for translation. - RetroTRAE performance with 10× augmentation: • Unimolecular: top-1 exact 56.4%; SM 58.1%; DM 60.5%; T≥0.85 68.2%; T≥0.80 72.5%; mean T 0.88; mean S 0.94. • Bimolecular: top-1 exact 60.1%; SM 60.9%; DM 62.7%; T≥0.85 64.3%; T≥0.80 66.7%; mean T 0.79; mean S 0.88. • Total: top-1 exact 58.3%; with SM+DM 61.6%; T≥0.85 66.3%; T≥0.80 69.6%; mean T 0.84; mean S 0.91. - Statistical significance: Mean Tanimoto of unimolecular predictions is 0.88 (p<1e-5); SM/DM predictions are exceptionally similar to ground truth (p<1e-7). Soft threshold region T>0.8 has p≤1e-4. - Interpretability: Attention maps align high weights to AEs at reaction centers and to altered environments in ring-opening and dissociation examples, demonstrating chemically meaningful token focus. - Retrieval: From 1000 USPTO test products, 55.7% of predicted reactants are exactly retrieved from PubChem; allowing single mutations increases retrieval by ~30%; allowing double mutations retrieves all test molecules. AE-based searches are less degenerate than ECFP-based (ECFP returns on average 1.7× more candidates due to bit collisions and lack of stereochemical info). - Coverage of chemical space: USPTO-derived AE tokens are insufficient to span broad chemical space. Unique AE counts: USPTO 275 (AE0) and 15,982 (AE2); ChEMBL 386 (AE0) and 39,149 (AE2); PubChem 3,450 (AE0) and 533,276 (AE2). In PubChem, 38.5% (AE0) and 35.2% (AE2) are singletons. - Benchmark comparison: RetroTRAE achieves top-1 exact 58.3% (61.6% with SM/DM), outperforming template-free Transformer baselines (e.g., SCROP 41.5%, RetroPrime 44.1%, Augmented Transformer 46.2%, AutoSynRoute 54.1%). Template-based AiZynthFinder reports 43–72% depending on template library size; comparisons vary with datasets/templates.

The findings support the hypothesis that atom environments are effective, chemically meaningful tokens for template-free retrosynthesis. By focusing on local structural changes around reaction centers, RetroTRAE improves robustness (no SMILES grammar) and interpretability (attention on altered AEs). The AE0∪AE2 representation provides high resolution with manageable vocabulary, avoiding the collision issues seen with ECFP4 while retaining chemical specificity. High similarity of non-exact predictions and strong retrieval results indicate that most model outputs are practically useful reactant candidates differing only by minimal, chemically plausible modifications (e.g., peripheral positions, chain lengths, stereochemistry). Positional encodings, despite AE permutation invariance, further enhance performance, aligning with observations in general perception models. Compared with SMILES-based Transformers, the AE-based approach removes the need for syntax correction and atom mapping, simplifying pipelines and reducing error sources. Although this work focuses on single-step retrosynthesis due to dataset constraints, the approach is compatible with multi-step planning when combined with search strategies such as Monte Carlo Tree Search. The limited coverage of AE space in USPTO underscores the importance of broader, more diverse reaction datasets to further boost generality and accuracy.

The study introduces RetroTRAE, a template-free, Transformer-based retrosynthesis model operating on atom environment tokens, delivering state-of-the-art top-1 exact accuracy (58.3%) on USPTO and 61.6% when counting highly similar single/double-mutation predictions. The AE representation yields chemically interpretable attention focused on reaction centers, avoids SMILES grammar issues and atom-mapping dependencies, and enables effective retrieval of real molecules from large databases. The work also establishes optimal fragment radii for translation tasks and analyzes fingerprint collision effects. Future directions include integrating RetroTRAE with search algorithms for multi-step synthesis planning, expanding and diversifying training datasets to improve chemical space coverage, incorporating stereochemistry more comprehensively, and exploring AE-based inputs in other generative or predictive chemistry tasks.

- Dataset coverage: The USPTO dataset’s AE vocabulary does not span broad chemical space; limited diversity constrains generalization and coverage. Performance is expected to improve with larger, more diverse reaction corpora. - Representation degeneracy: AE sets do not encode full molecular connectivity, so multiple structures can share the same AE set; this can affect unique reconstruction and retrieval. - Fragment radius trade-offs: Larger-radius AEs (e.g., AE4) have enormous vocabularies unsuitable for translation; hashed ECFP4 suffers high bit collisions, degrading performance. - Structural nuances: AE-based predictions can err on aliphatic chain lengths and peripheral positions; single/double mutations often reflect stereochemical differences or positional isomers, partly due to limited stereochemical information in the dataset. - Scope: The model addresses single-step retrosynthesis; multi-step route planning requires coupling with an external search algorithm. - Retrieval degeneracy: ECFP-based retrieval is more degenerate than AE-based owing to bit collisions; absence of stereochemical annotations in the dataset can impact retrieval specificity.