Vision Language Model–Guided Semantic Distillation Network for Pedestrian Crossing Intention Prediction

Pedestrian crossing intention prediction is a critical function of autonomous vehicles, enabling safe pedestrian–vehicle interactions in complex urban environments. Accurate prediction remains difficult because intention depends on pedestrian behavior, vehicle dynamics, and surrounding context. Existing methods increasingly add modalities to improve performance, but this creates substantial computational overhead, leaving accurate prediction with few modalities unresolved. The paper proposes the vision language model-guided semantic distillation network (VLM-GSDN). During training, it uses pedestrian- and scene-level semantic descriptions generated by a vision language model to align local and global visual representations with high-level semantics. Efficient inference requires only RGB images and image-space pedestrian motion. Motion-guided modulation and predictive spatial modulation refine pedestrian representations with observed motion and predicted future displacement. Selective temporal attention integrates temporal information while emphasizing informative frames. A multi-task strategy uses pedestrian action-state transition prediction as auxiliary motion supervision. Experiments on JAAD and PIE show competitive state-of-the-art performance, promising cross-dataset generalization, and robustness over longer prediction horizons.