Demonstrations and natural language instructions are two common ways to specify and teach robots novel tasks. However, for many complex tasks, a demonstration or language instruction alone contains ambiguities, preventing tasks from being specified clearly. In such cases, a combination of both a demonstration and an instruction more concisely and effectively conveys the task to the robot than either modality alone. To instantiate this problem setting, we train a single multi-task policy on a few hundred challenging robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task Conditioning), a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction. By allowing these two modalities to mutually disambiguate and clarify each other during novel task specification, DeL-TaCo (1) substantially decreases the teacher effort needed to specify a new task and (2) achieves better generalization performance on novel objects and instructions over previous task-conditioning methods. To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone. See additional materials at https://sites.google.com/view/del-taco-learning

PDF, HTML, Arxiv

International Conference on Learning Representations (2023).

@inproceedings{yu_deltaco_23, title={Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks}, author={Albert Yu and Raymond J. Mooney}, booktitle={International Conference on Learning Representations}, journal={International Conference on Learning Representations}, month={May}, institution={UT Austin}, url="http://www.cs.utexas.edu/users/ai-lab?yu_deltaco_22", year={2023} }

Video