AI deciphers pianist’s muscle activity from video

AI and human movement research intersect to enable accurate estimation of hand muscle activity from standard video recordings. Researchers have introduced a system that accurately reconstructs muscle activation patterns without sensors using a deep learning framework trained on a large, comprehensive multimodal dataset of professional pianists. This advancement provides a low-cost, non-invasive method to analyze fine motor control, optimize rehabilitation strategies, enhance performance training, and inform future developments in human-machine interaction.

From poses to muscles: Inferring pianist’s hand muscles from video

Hand movements while playing the piano depend on precise coordination between tiny muscles hidden under the skin. Tracking these signals has traditionally required expensive, cumbersome, and technically complex electromyography (EMG) sensors. A research team led by Professor Hideki Koike of the Department of Computer Science, Faculty of Informatics, Tokyo University of Science (Science Tokyo) and Dr. Shinichi Furuya of Sony Computer Science Laboratories tackled this problem using artificial intelligence.

Their new framework, Piano Keystroke-Pose-Muscle Network (PianoKPM Net), estimates the activity of miniature hand muscles using only video recordings. Their findings will be published online on September 19, 2025, and will be presented at the 39th International Conference.^th Neural Information Processing Systems Conference (NeurIPS 2025) held on December 2, 2025 in San Diego, USA. The system is built on PianoKPM, a new dataset that captures with great precision how expert pianists move, press, and control their hands. It contains 12.6 hours of synchronized data from 20 professional pianists performing seven different musical tasks. Each performance was recorded with 60 frames/s multi-view video, 3D hand poses, 1 kHz keystroke data, audio, and 2 kHz EMG signals from six small hand muscles. The dataset includes more than 5 million pose frames and 28 million EMG samples, creating the first detailed map linking visible movement to internal muscle activity. “By leveraging this dataset, we propose PianoKPM Net, which estimates high-frequency electromyography from posture data,” exclaims Koike.

Using this foundation, PianoKPM Net learns to infer muscle movements from video data. By combining cues from hand pose and keystrokes, the model reconstructs the timing and strength of muscle signals. In comparative tests against advanced baselines such as NeuroPose and CodeTalker, PianoKPM Net was able to predict both the amplitude and timing of muscle activity with higher accuracy. Even with pianists and pieces not included in the training, the model maintains strong performance, confirming its adaptability and versatility.

This approach turns a simple camera into a non-invasive tool for studying muscle coordination. This allows you to watch as a master pianist controls subtle muscle movements to achieve speed, control, and accuracy. This allows detailed physiological assessment without attaching sensors to the body, reducing both cost and discomfort.

This technology has potential far beyond that of the piano. Sports science can track muscle activity to improve training accuracy and prevent overuse injuries. In rehabilitation, recovery progress can be monitored and continuous feedback provided to clinicians without physical attachment. Understanding a user’s muscular effort could also improve human-machine interaction systems, helping to improve robotic assistance and gesture-based interfaces. “The combination of PianoKPM Net and the PianoKPM dataset creates a foundation for affordable access to physiological and muscle activity signals within the body, supporting advances in human augmentation and advanced human-machine interaction,” Koike explains.

The research team plans to make both the dataset and model publicly available. This open release will enable scientists and developers to advance research in motor learning, embodied intelligence, and assistive robotics. Broad access will facilitate the standardization of benchmarks for motion and muscle activity estimation, accelerating development in multiple areas. PianoKPM Net provides a new way to study fine motor control by linking vision and physiology. This replaces complex EMG setups with easy-to-access video-based analysis, creating opportunities for performance research, clinical evaluation, and human technology design. This system represents a clear step toward affordable, AI-driven analysis of skilled movements, where invisible muscle patterns can finally be observed and understood using vision alone. In the future, this technology will be able to be used via a low-latency communication network even in environments where expensive bioinstrumentation equipment is not available, potentially contributing to remote skills education.

author:

Liu Ruofan^{1, 2}pen yicheng¹Takanori Oku²Liao Chenchi¹Erwin Wu¹Shinichi Furuya²and Hideki Koike¹

title:

From pose to muscle: Multimodal learning of piano hand electromyography

Affiliation:

¹Tokyo University of Science, School of Information Science and Technology, Department of Information Science

²Sony Computer Science Laboratories, Japan