UTCS Artificial Intelligence
courses
talks/events
demos
people
projects
publications
software/data
labs
areas
admin
Improving Grounded Natural Language Understanding through Human-Robot Dialog (2019)
Jesse Thomason
,
Aishwarya Padmakumar
,
Jivko Sinapov
,
Nick Walker
,
Yuqian Jiang
,
Harel Yedidsion
,
Justin Hart
,
Peter Stone
, and
Raymond J. Mooney
Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically -- continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.
View:
PDF
Citation:
In
IEEE International Conference on Robotics and Automation (ICRA)
, Montreal, Canada, May 2019.
Bibtex:
@inproceedings{thomason:icra19, title={Improving Grounded Natural Language Understanding through Human-Robot Dialog}, author={Jesse Thomason and Aishwarya Padmakumar and Jivko Sinapov and Nick Walker and Yuqian Jiang and Harel Yedidsion and Justin Hart and Peter Stone and Raymond J. Mooney}, booktitle={IEEE International Conference on Robotics and Automation (ICRA)}, month={May}, address={Montreal, Canada}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127746", year={2019} }
People
Justin Hart
Postdoctoral Fellow
hart [at] cs utexas edu
Yuqian Jiang
Ph.D. Student
Raymond J. Mooney
Faculty
mooney [at] cs utexas edu
Aishwarya Padmakumar
Ph.D. Alumni
aish [at] cs utexas edu
Jivko Sinapov
Postdoctoral Alumni
jsinapov [at] cs utexas edu
Peter Stone
Faculty
pstone [at] cs utexas edu
Jesse Thomason
Ph.D. Alumni
thomason DOT jesse AT gmail
Nick Walker
Undergraduate Alumni
nswalker [at] cs uw edu
Harel Yedidsion
Postdoctoral Fellow
harel [at] cs utexas edu
Harel Yedidsion
Postdoctoral Alumni
harel [at] cs utexas edu
Areas of Interest
Active Learning
Language and Robotics
Learning for Semantic Parsing
Labs
Machine Learning
Learning Agents