Description of the projectBack to TopCurrent state-of-the-art language learners require annotated corpora as training data. However, constructing such corpora is difficult and time-consuming. On the other hand, children acquire language through exposure to linguistic input in the context of a rich, relevant, perceptual environment. By connecting words and phrases to objects and events in the world, the semantics of language is grounded in perceptual experience (Harnad, 1990). Ideally, a machine learning system could learn language in a similar manner. Our ultimate goal is to build a system that can exploit the large amount of linguistic data available naturally in the world with minimal supervision. Although there has been some interesting computational work in grounded language learning (Roy, 2002; Bailey et al., 1997; Yu & Ballard, 2004), most of the focus has been on dealing with raw perceptual data and the complexity of the language involved has been very modest. To help make progress, we study the problem in a simulated environment that retains many of the important properties of a dynamic world with multiple agents and actions while avoiding many of the complexities of robotics and vision. Specifically, we use the Robocup simulator which provides a fairly detailed physical simulation of robot soccer. Our immediate goal is to build a system that learns to semantically interpret and generate language in the Robocup soccer domain by observing an on-going commentary of the game paired with the dynamic simulator state. While several groups have constructed Robocup commentator systems (Andre et al., 2000) that provide a textual natural-language (NL) transcript of the simulated game, their systems use manually-developed templates and are incapable of learning. |
DemoBack to TopBelow are sample sportscasts produced by our system. We also included commentaries made by our human commentators for comparison. [English Version]
[Korean Version]
Steps for creating the demo clip:
|
Publication and TalksBack to Top
Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision
[PDF]
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language
[Abstract]
[PDF]
[JAIR link]
Using Perceptual Context to Ground Language
[slides (PDF)]
Learning to Sportscast: A Test of Grounded Language Acquisition
[slides (PPT)]
[slides print-version (PPT)]
Learning to Sportscast: A Test of Grounded Language Acquisition
[Abstract]
[PDF]
[slides (PPT)]
[poster (PDF)]
[talk (video)]
|
DataBack to TopOverviewWe have collected human commentaries for the championship games of Robocup simulation league (www.robocup.org) from year 2001 to 2004. The commentators typed their comments into a text box, which were recorded with a timestamp. Together with the original game log files one can replay the games with these commentaries. We developed a symbolic representation of game events to simulate perception. Most of these events involve actions with the ball, such as kicking and passing, but also include other game information such as whether the current playmode is kickoff, offside, or corner kick. These events are automatically extracted from the game logs using a rule-based system. The events are represented as atomic formulas in predicate logic with timestamps. These constitute our MRs. We manually developed a context-free grammar for this formal semantic language. Note that the use of English words for predicates and constants in the MR is for human readability only, our system treats these as arbitrary conceptual tokens and must learn their connection to English words. Finally, we established a correspondence between the natural language commentaries and the game events. Each comment is paired with all of the events that occurred five seconds or less before the comment was made. Notice that this is a very coarse estimate and do not always capture the correct correspondences. Moreover, many sentences cannot be represented by our MR language. For evaluation purposes only, a gold-standard matching was produced by examining each comment manually and selecting the correct MR if it exists. Citations
Please use the following citations when referencing the sources of the data:
@InProceedings{chen:icml08,
title = "Learning to Sportscast: A Test of Grounded Language Acquisition",
author = "David L. Chen and Raymond J. Mooney",
booktitle = "Proceedings of 25th International Conference on Machine
Learning (ICML-2008)",
address = "Helsinki, Finland",
month = "July",
year = 2008
}
[Korean]
@Article{chen:jair10,
author = "David L. Chen, Joohyun Kim, Raymond J. Mooney",
title = "Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language",
journal = "Journal of Artificial Intelligence Research",
volume = "37",
pages = "397--435",
year = "2010",
}
DownloadsHere are compressed tarballs of all the data files:
Or you can browse and download individual components from the following directories: |
Contact InformationBack to TopIf you have any questions or comments, please contact David Chen or Joohyun Kim If you are interested in reading more literature in this area, check out our reading group CLAMP |