Lattice thermal conductivity (κ) is crucial for various applications but expensive to measure experimentally or calculate using first principles. This study presents a machine learning approach that predicts phonon scattering rates and thermal conductivity with accuracy comparable to experiments and first principles calculations. The approach mitigates computational challenges associated with the high skewness of phonon scattering rates and their complex contributions to thermal resistance. Transfer learning between different orders of phonon scattering further improves model performance, offering up to two orders of magnitude acceleration compared to first principles calculations, enabling large-scale thermal transport informatics.