We present a holistic data-driven technique that generates natural-language descriptions for videos. We combine the output of state-of-the-art object and activity detectors with ``real-world'' knowledge to select the most probable subject-verb-object triplet for describing a video. We show that this knowledge, automatically mined from web-scale text corpora, enhances the triplet selection algorithm by providing it contextual information and leads to a four-fold increase in activity identification. Unlike previous methods, our approach can annotate arbitrary videos without requiring the expensive collection and annotation of a similar training video corpus. We evaluate our technique against a baseline that does not use text-mined knowledge and show that humans prefer our descriptions 61% of the time.

PDF

Proceedings of the NAACL HLT Workshop on Vision and Language (WVL '13) (2013), pp. 10--19.

@article{krishnamoorthy:naacl-wvl13, title={Generating Natural-Language Video Descriptions Using Text-Mined Knowledge}, author={Niveda Krishnamoorthy and Girish Malkarnenkar and Raymond J. Mooney and Kate Saenko and Sergio Guadarrama}, booktitle={Proceedings of the NAACL HLT Workshop on Vision and Language (WVL '13)}, month={July}, address={Atlanta, Georgia}, pages={10--19}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127397", year={2013} }

Slides (PPT)