Based on collaborative research by Meta AI Research, Tencent America, MIT CSAIL, and Carnegie Mellon, Meta has released Animated Drawings, an AI-based tool for creating animations from hand-drawn human-like characters.
Animated Drawings combine different techniques to achieve their goals. This includes object detection that predicts tight bounding boxes around people. Image segmentation to separate pixels belonging to people from those belonging to the background. Pose estimation that identifies a set of skeletal joints. This allows you to create a digital version of your drawing that can be animated using computer graphics techniques by applying motion data from pre-selected clips taken from human performers.
According to the researchers, the project is the first step towards a drawing-to-animation storytelling system.
We demonstrate the value of this method in several ways. First, we examine the accuracy and success of each stage of the system as a function of the size of the training dataset. Second, we do perceptual research,
The charm of twisted perspective retargeting when animating these characters.
As the researchers describe, after initial attempts to build a dataset from synthetic data failed, they set up a web-based tool called Animated Drawings Demo to collect data from their authors. The original goal was to collect about 10,000 drawings, but overwhelming user response led him to over 3 million drawings, of which about 200,000 made it into the final dataset.
The research results presented in a paper just approved for publication in ACM Transactions on Graphics are now available for experimental use thanks to Meta open sourcing an implementation of the algorithm described in this paper. rice field.
(Image credit: Meta)
The Animated Drawings published on GitHub are written in Python and tested on macOS Ventura 13.2.1 and Ubuntu 18.04. This tool makes it easy to create existing animations. annotated Draw using a command like this:
from animated_drawings import render
render.start('./examples/config/mvc/interactive_window_example.yaml')
To use additional drawings, the user must first annotate them. This can be done manually or more easily by using a set of scripts to run the humanoid figure detector and pose estimator. The script relies on TorchServe and can be easily deployed using the provided Dockerfile. Meta says the annotation files may need manual tweaking, and he also provides a web-based tool for that with the container.
As a final step, you can animate your character using motion clips in BVH format. Meta provides numerous BVH animations of her, including jumping, waving and greeting. Additional BVH animations of her can be created from his RGB video using tools such as rokoko.
As mentioned above, Animated Drawings is only the first attempt at an AI-based draw-to-animation system, and there are multiple potential improvements. Specifically, the researchers improved segmentation by better handling focus and distortion, lighting effects, or lack of shadows, and used pre-animation classification of people to improve pose estimation and animation. I am referring to using
