SLEAP: A deep learning system for multi-animal pose tracking.

TitleSLEAP: A deep learning system for multi-animal pose tracking.
Publication TypeJournal Article
Year of Publication2022
AuthorsPereira, TD, Tabris, N, Matsliah, A, Turner, DM, Li, J, Ravindranath, S, Papadoyannis, ES, Normand, E, Deutsch, DS, Z Wang, Y, McKenzie-Smith, GC, Mitelut, CC, Castro, MDiez, D'Uva, J, Kislin, M, Sanes, DH, Kocher, SD, Wang, SS-H, Falkner, AL, Shaevitz, JW, Murthy, M
JournalNat Methods
Volume19
Issue4
Pagination486-495
Date Published2022 Apr
ISSN1548-7105
KeywordsAlgorithms, Animals, Behavior, Animal, Deep Learning, Head, Machine Learning, Mice, Social Behavior
Abstract

<p>The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.</p>

DOI10.1038/s41592-022-01426-1
Alternate JournalNat Methods
PubMed ID35379947
PubMed Central IDPMC9007740
Grant ListR01 DC011284 / DC / NIDCD NIH HHS / United States
/ HHMI / Howard Hughes Medical Institute / United States
R01 NS104899 / NS / NINDS NIH HHS / United States
DP2 GM137424 / GM / NIGMS NIH HHS / United States
R35 NS111580 / NS / NINDS NIH HHS / United States
DP2 MH126375 / MH / NIMH NIH HHS / United States
P30 CA014195 / CA / NCI NIH HHS / United States
R00 MH109674 / MH / NIMH NIH HHS / United States
R01 NS045193 / NS / NINDS NIH HHS / United States