SLEAP: A deep learning system for multi-animal pose tracking

Studies of social behavior and naturalistic settings require accurate markerless pose estimation for multiple interacting animals. SLEAP (Social LEAP Estimates Animal Poses) is a general-purpose machine learning system for multi-animal pose tracking that provides an end-to-end workflow for labeling, training, inference, and proofreading via a GUI and APIs. It offers standardized data formats, reproducible configuration, 30+ network backbones, both bottom-up and top-down multi-instance strategies, and two identity-tracking approaches. Across seven datasets (flies, bees, mice, gerbils), SLEAP attains high accuracy and very high speed (>800 FPS with <3.5 ms latency at 1,024×1,024 resolution), enabling real-time applications. The system’s real-time capability is demonstrated by closed-loop optogenetic control based on online detection of social interactions.

Talmo D. Pereira, Nathaniel Tabris, Arie Matsliah, David M. Turner, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Edna Normand, David S. Deutsch, Z. Yan Wang, Grace C. McKenzie-Smith, Catalin C. Mitelut, Marielisa Diez Castro, John D’Uva, Mikhail Kislin, Dan H. Sanes, Sarah D. Kocher, Samuel S.-H. Wang, Annegret L. Falkner, Joshua W. Shaevitz, Mala Murthy