Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models

Synthetic data has been proposed to alleviate high-quality data scarcity for training large language models (LLMs). While prior work shows synthetic data can improve downstream benchmark performance, the authors identify inherent flaws: uniform formats in synthetic data induce pattern overfitting and significant output distribution shifts that degrade instruction-following ability. Focusing on synthetic question–answer (Q-A) pairs, the paper analyzes these flaws and introduces an unlearning-based mitigation strategy using a controllable lower-bounded forgetting loss. Experiments show the method reverses instruction-following degradation caused by pattern overfitting without sacrificing benchmark performance and at relatively low cost, offering guidance for robust and efficient LLM training with synthetic data.