MD-HIT: Machine learning for material property prediction with dataset redundancy control

Materials datasets often contain many redundant (highly similar) materials due to historical tinkering in materials design. This redundancy biases performance evaluation under random splits, overestimating predictive accuracy and harming out-of-distribution (OOD) generalization. Inspired by CD-HIT in bioinformatics, we propose MD-HIT, a redundancy reduction algorithm for material datasets, with composition- (MD-HIT-composition) and structure-based (MD-HIT-structure) variants. On formation energy and band gap prediction tasks using composition-only and structure-based models, we show that enforcing redundancy control reduces apparent test performance compared to highly redundant settings but yields evaluations that better reflect true predictive capability.