The limits of fair medical imaging AI in real-world generalization

As artificial intelligence (AI) rapidly approaches human-level performance in medical imaging, it is crucial that it does not exacerbate healthcare disparities. Prior work showed AI can infer demographics from chest X-rays, prompting concern that models may use demographic shortcuts and behave unfairly across subpopulations. This study investigates the extent to which medical AI encodes demographic information and how that affects fairness in both in-distribution (ID) and out-of-distribution (OOD) settings. We examine radiology, dermatology and ophthalmology, including six global chest X-ray datasets. We confirm that AI leverages demographic shortcuts in disease classification. Algorithmic mitigation can create ‘locally optimal’ models that reduce ID fairness gaps, but such optimality often fails under OOD shifts. Models with less demographic encoding tend to be more ‘globally optimal’, exhibiting better fairness when evaluated in new environments. We propose best practices for selecting models that maintain performance and fairness beyond initial training contexts, highlighting critical considerations for AI clinical deployments across populations and sites.

Yuzhe Yang, Haoran Zhang, Judy W. Gichoya, Dina Katabi, Marzyeh Ghassemi