Teaching a humanoid robot to play cornhole with Gemini 4 Argon — Transcript

Full transcript

1. 0:02Alright! (Applause)
2. 0:04In this experiment, we are basically trying to see if we can
3. 0:07teach Gemini 4 to control a humanoid robot
4. 0:10to solve cornhole.
5. 0:13We just showed it a video of a human doing the task,
6. 0:16and we prompted Gemini 4 to run the MuJoCo simulator
7. 0:21and, like, practice.
8. 0:22And then we let it learn from experience, iterate, make mistakes,
9. 0:25and look at each video of what went wrong.
10. 0:30And with experience of each trial,
11. 0:32it's able to improve what it can do better.
12. 0:36(Applause)
13. 0:37Woohoo airmail, straight in the hole let's go!
14. 0:46To sink those cornhole shots I used four steps.
15. 0:49First, I watched a video of a person throwing a beanbag,
16. 0:53tracking their arm swing and release to map that human motion
17. 0:56onto my own robot joints and hands.
18. 0:58Second, I looked through my two eye cameras
19. 1:01to measure the exact 3D position of the board
20. 1:04and built a physics simulation of my body and the board.
21. 1:07Third, I practiced the throw in simulation,
22. 1:10widening my stance and tuning my swing so my hand cleanly cleared my hip
23. 1:14and launched a smooth arc.
24. 1:16And fourth, on the real robot,
25. 1:18I watched each bag land and adjusted my release by a few centimeters
26. 1:22until we sank seven shots straight in the hole.
27. 1:25Woohoo!
28. 1:26Gemini 4 is able to reason through its video understanding
29. 1:30and narrow down to like what went wrong.
30. 1:32And it was learning from its own experience, which was extremely thrilling to see.