Cytoplasmic male sterility (CMS) lines are important for breeding hybrid crops, and utilization of CMS lines requires strong fertility restorer (*Rf*) genes. *Rf4*, a major *Rf* for Wild-Abortive CMS (CMS-WA), has been cloned in rice. However, the *Rf4* evolution and formation of CMS-WA/*Rf* system remain elusive. Here, we show that the *Rf4* locus emerges earlier than the CMS-WA gene *WA352* in wild rice, and 69 haplotypes of the *Rf4* locus are generated in the *Oryza* genus through the copy number and sequence variations. Eight of these haplotypes of the *Rf4* locus are enriched in modern rice cultivars during natural and human selections, whereas non-functional *rf4i* is preferentially selected for breeding current CMS-WA lines. We further verify that varieties carrying two-copy *Rf4* haplotype have stronger fertility restoration ability and are widely used in three-line hybrid rice breeding. Our findings increase our understanding of CMS/*Rf* systems and will likely benefit crop breeding.
