This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.

PDF

In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-16), pp. 1961--1966, Austin, Texas 2016.

@inproceedings{venugopalan:emnlp16, title={Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text}, author={Subhashini Venugopalan and Lisa Anne Hendricks and Raymond Mooney and Kate Saenko}, booktitle={Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-16)}, address={Austin, Texas}, pages={1961--1966}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127562", year={2016} }

Poster