We develop an end-to-end model for learning to follow language instructions with compositional policies. Our model combines large language models with pretrained compositional value functions to generate policies for goal-reaching tasks specified in natural language. We evaluate our method in the BabyAI environment and demonstrate compositional generalization to novel combinations of task attributes. Notably our method generalizes to held-out combinations of attributes, and in some cases can accomplish those tasks with no additional learning samples.

PDF

Workshop on Language and Robot Learning at CoRL 2022 (2022).

@article{cohen:corl22langrob, title={End-to-End Learning to Follow Language Instructions with Compositional Policies}, author={Vanya Cohen and Geraud Nangue Tasse and Nakul Gopalan and Steven James and Ray Mooney and Benjamin Rosman}, booktitle={Workshop on Language and Robot Learning at CoRL 2022}, month={December}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127990", year={2022} }

Poster