Neural-Behavioral Representation of
Natural Whole-body Movement in Monkeys
2 Tsinghua University
Abstract & Overview
In this work, we present a synchronized neural-behavioral recording and modeling framework for freely moving monkeys. We combine large-scale epidural cortical recordings across distributed sensory- and motor-related areas with synchronized 8-view markerless motion capture. By establishing an autoregressive neural-behavioral model, we compress natural movement kinematics and reconstruct whole-body actions conditioned on neural population signals. Our results show continuous, physically realistic decoding across multi-second timescales without explicit physics solvers, achieving the first proof-of-concept continuous whole-body decoding under naturalistic primate settings.
Demonstrations & Visualizations
Visualized skeletal sequences demonstrate that our Neural-Behavioral Model accurately reconstructs whole-body movements conditioned on neural signals while maintaining bio-physical continuity. In contrast, behavioral-only priors produce generic unconstrained sequences, and standard LSTM regressors exhibit substantial limb hyperextension and anatomical distortions.
Visual comparison across different initial conditions rollouts over time.








































Quantitative Benchmarks
Evaluations on continuous test sequences measuring Average Displacement Error (ADE), Final Displacement Error (FDE), and Pearson correlation coefficient (r) over 3-second horizons.
| Model | Horizontal Movement (cm / r) | Vertical Movement (cm / r) | ||||
|---|---|---|---|---|---|---|
| ADE ↓ | FDE ↓ | Pearson r ↑ | ADE ↓ | FDE ↓ | Pearson r ↑ | |
| Neural-Behavioral (Ours) | 7.84 | 13.27 | 0.43 | 5.95 | 10.17 | 0.33 |
| Behavioral | 10.36 | 19.19 | 0.21 | 9.20 | 19.74 | 0.25 |
| Label-Behavioral | 10.22 | 17.05 | 0.31 | 8.45 | 18.04 | 0.24 |
| LSTM | 15.29 | 9.38 | 0.24 | 7.78 | 15.20 | 0.23 |