[2205.06175] A Generalist Agent

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

Published in Transactions on Machine Learning Research (11/2022) A Generalist Agent Scott Reed*,† , Konrad Żołna* , Emilio Parisotto* , Sergio Gómez Colmenarejo† , Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar…

related reading

• A Steerable Model with Emergent Capabilitiespi.website
• GEN-1.5: Embodied Foundation Models are One-Shot Learners - Generalist AIgeneralistai.com
• GPT-4openai.com
• LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
• Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
• Sporks of AGIsergeylevine.substack.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• To Understand Language is to Understand Generalization | Eric Jangevjang.com
• How Claude Performs on Robotics Tasks \ Anthropicanthropic.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• A VLA with Open-World Generalizationpi.website
• The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work