video Eggheads at Google’s DeepMind have developed a deep learning curriculum that can teach robots how to play soccer poorly.
In contrast to the sophisticated acrobatics of Boston Dynamics’ Atlas robots, a pair of Robotis OP3 robots under DeepMind’s tutelage look like exhausted toddlers on a sub-legal 5m x 4m football field, Or bumble and flop on the football pitch. Judge for yourself in the video below.
They do so with obvious purpose, and despite falling many times, they are able to right themselves and sometimes score goals. Even if it’s just a misguided anthropomorphic, it’s easy to see something akin to a determination that we value and encourage each other. It’s hard not to root for them, but if they’re scaled up and weaponized, they’ll provoke other emotions.
28 researchers involved in the project describe their work in papers [PDF] The title is “Learning Agile Soccer Skills for Biped Robots with Deep Reinforcement Learning”.
“used deep [Reinforcement Learning] Our goal is to train a humanoid robot with 20 working joints to play a simplified one-on-one (one-on-one) soccer game,” the authors explain. – Play settings.
“The resulting policy exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning and kicking, and transitions between them in a smooth, stable and efficient manner. goes far beyond what we intuitively expect from a robot.”
The DeepMind project isn’t as ambitious as the years-long effort to prepare machines for the RoboCup high-tech competition. But the latest iteration of RoboCup is decidedly less interesting to watch because of the restrained behavior of its participants. DeepMind players, on the other hand, wave their arms like maniacs.
Deep reinforcement learning is a method of training neural networks so that agents (software or hardware-based entities) learn how to do things (simulated or in the real world) through trial and error. And it has become a popular technique for teaching robots how to move around in different environments. As you can see from Cassie’s running acumen, it’s sort of a mechanical ostrich torso, and you’d never want to see it chasing after you.
The DeepMind team’s aim was to train agents to play soccer. This requires different skills such as walking, kicking, standing, scoring and defending, all of which must be coordinated in order to score goals and win the game.
To train an agent (in this case, the software that controls a robot), it was not enough to reward the system for scoring a goal. Instead, researchers approached skill sets on an individual basis and focused on developing what they called teacher policies. to manage goals scored, etc. Opponents fall to the ground immediately, which is different from real soccer diving.
Researchers had to be careful to stop goalscoring training when agents fell to the ground to prevent unwanted but apparently functional behavior. Instead of walking and kicking, go for the goal,” they explain in their paper.
The get-up and goalscoring policies were eventually combined. And through a process of deep reinforcement learning and rewards for achieving specific goals, the software developed passable soccer skills.
Transferring a trained software agent to a robot body was not too difficult. According to the authors, this was a zero-shot process, meaning no additional training was required.
“Reduce the gap between simulation and reality through simple system identification, improve policy robustness through domain randomization and perturbation during training, and improve reward conditions to acquire behaviors that are less likely to damage the robot.” We included formation,” they explain.
This means that simulator parameters are mapped to hardware actuator settings, randomized properties such as floor friction and joint orientation, masses of robot parts, control loop latencies, random perturbations, etc. We made sure that we could handle a wide variety of issues reliably. Forces acting on a robot’s body. In one tweak, we added a reward component that reduces the strain on knee joints that bots tend to damage.
Get-up and soccer teacher training took 14 and 158 hours (6.5 days) respectively, followed by 68 hours for distillation and self-play. And the results were better than intentionally trying to program those skills, Boffin said.
“Reinforcement learning policies outperformed professionally and manually designed skills, walking 156% faster and getting up 63% faster,” the paper states.
“When we initialized near the ball, we kicked the ball at 5% slower velocity. Both achieved ball velocities of about 2 m/s. The average kick velocity was 2.6 m/s (24% faster than the scripted skill), and the maximum kick velocity over the entire episode was 3.4 m/s.”
A research team at DeepMind has demonstrated that deep reinforcement learning can be applied to train humanoid robots effectively and at low cost. For better or worse, this is another stopping step towards a future where bipedal robots walk among us. ®
