Neural-Behavioral Representation of
Natural Whole-body Movement in Monkeys

Jieshi He1*, Puzhe Li1*, Yanan Sui2†, Mu-ming Poo1†
1 Center for Excellence in Brain Science and Intelligence Technology, CAS
2 Tsinghua University

Abstract & Overview

In this work, we present a synchronized neural-behavioral recording and modeling framework for freely moving monkeys. We combine large-scale epidural cortical recordings across distributed sensory- and motor-related areas with synchronized 8-view markerless motion capture. By establishing an autoregressive neural-behavioral model, we compress natural movement kinematics and reconstruct whole-body actions conditioned on neural population signals. Our results show continuous, physically realistic decoding across multi-second timescales without explicit physics solvers, achieving the first proof-of-concept continuous whole-body decoding under naturalistic primate settings.

Whole-Body Movement Decoding Epidural Cortical Recording (ECoG-like) Neural Representation Autoregressive Model Brain-Machine Interface
Decoding natural whole-body movement pipeline
Figure 1. Pipeline for decoding natural whole-body movement from epidural neural signals. System setup: 3D whole-body kinematics captured via an 8-camera array synchronized with 62-channel wireless epidural cortical recording. Computational framework: A condition-guided autoregressive model combines learned behavioral priors with continuous neural dynamics to decode bio-realistic continuous whole-body movements.

Quantitative Benchmarks

Evaluations on continuous test sequences measuring Average Displacement Error (ADE), Final Displacement Error (FDE), and Pearson correlation coefficient (r) over 3-second horizons.

Model Horizontal Movement (cm / r) Vertical Movement (cm / r)
ADE ↓ FDE ↓ Pearson r ↑ ADE ↓ FDE ↓ Pearson r ↑
Neural-Behavioral (Ours) 7.84 13.27 0.43 5.95 10.17 0.33
Behavioral 10.36 19.19 0.21 9.20 19.74 0.25
Label-Behavioral 10.22 17.05 0.31 8.45 18.04 0.24
LSTM 15.29 9.38 0.24 7.78 15.20 0.23

Citation

@inproceedings{he2026neuralbehavioral, title={Neural-Behavioral Representation of Natural Whole-body Movement in Monkeys}, author={He, Jieshi and Li, Puzhe and Sui, Yanan and Poo, Mu-ming}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2026} }