Deep Learning
Large Neural Networks with many layers have proven generally effective at a broad range of tasks and pose a unique set of challenges during learning and inference. We are investigating the properties and applications of such large Neural Nets.
People
Ishan Durugkar Ph.D. Student ishand [at] cs utexas edu
Wonjoon Goo Ph.D. Student wonjoon [at] cs utexas edu
Matthew Hausknecht Formerly affiliated Collaborator mhauskn [at] cs utexas edu
Stephen Roller Ph.D. Alumni roller [at] cs utexas edu
Wesley Tansey Formerly affiliated Collaborator tansey [at] cs utexas edu
Subhashini Venugopalan Ph.D. Alumni vsub [at] cs utexas edu
Chao-Yuan Wu Ph.D. Student cywu [at] cs utexas edu
Lemeng Wu Ph.D. Student lmwu [at] cs utexas edu
Jialin Wu Ph.D. Alumni jialinwu [at] utexas edu
Ruohan Zhang Ph.D. Student zharu [at] utexas edu
Publications
     [Expand to show all 53][Minimize]
CAPE: Corrective Actions from Precondition Errors using Large Language Models 2024
Shreyas Sundara Raman, Vanya Cohen, Ifrah Idrees, Eric Rosen, Raymond Mooney, Stefanie Tellex, and David Paulius, International Conference on Robotics and Automation (ICRA) (2024).
CAT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans 2024
Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, Raymond Mooney, Empirical Methods in Natural Language Processing (EMNLP) (2024).
CONTRADOC: Understanding Self-Contradictions in Documents with Large Language Models 2024
Jierui Li, Vipul Raheja, Dhruv Kumar, North American Chapter of the Association for Computational Linguistics (NAACL) (2024).
Measuring Sound Symbolism in Audio-visual Models 2024
Wei-Cheng Tseng, Yi-Jen Shih, David Harwath, Raymond Mooney, IEEE Spoken Language Technology (SLT) Workshop (2024).
Multimodal Contextualized Semantic Parsing from Speech 2024
Jordan Voas, Raymond Mooney, David Harwath, Association for Computational Linguistics (ACL) (2024).
Natural Language Can Help Bridge the Sim2Real Gap 2024
Albert Yu, Adeline Foote, Raymond Mooney, and Roberto Martín-Martín, Robotics, Science and Systems (RSS) (2024).
Sparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval 2024
Priyanka Mandikal, Raymond Mooney, The 4th Workshop on Scientific Document Understanding, AAAI (2024).
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks 2023
Garrett Bingham and Risto Miikkulainen, In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 2023. (also arXiv:2021.08958).
“Female Astronaut: Because sandwiches won’t make themselves up there!": Towards multi-modal misogyny detection in memes 2023
Smriti Singh, Amritha Haridasan, Raymond Mooney, Association of Computational Linguistics (ACL), Workshop on Online Abuse and Harms (WOAH) (2023).
Directly Optimizing Evaluation Metrics to Improve Text to Motion 2023
Yili Wang, Masters Thesis, Department of Computer Science, UT Austin.
Explaining Competitive-Level Programming Solutions using LLMs 2023
Jierui Li, Szymon Tworkowski, Yingying Wu, Raymond Mooney, Association of Computational Linguistics (ACL), Natural Language Reasoning and Structured Explanations Workshop (2023).
Learning Deep Semantics for Test Completion 2023
Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond Mooney and Milos Gligoric, International Conference on Software Engineering (2023).
Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks 2023
Albert Yu, Raymond J. Mooney, International Conference on Learning Representations (2023).
Using Planning to Improve Semantic Parsing of Instructional Texts 2023
Vanya Cohen, Raymond Mooney, Association of Computational Linguistics (ACL), Natural Language Reasoning and Structured Explanations Workshop (2023).
What is the Best Automated Metric for Text to Motion Generation? 2023
Jordan Voas, Masters Thesis, Department of Computer Science, UT Austin.
What is the Best Automated Metric for Text to Motion Generation? 2023
Jordan Voas, Yili Wang, Qixing Huang, Raymond Mooney, In ACM SIGGRAPH Asia, December 2023.
Discovering Parametric Activation Functions 2022
Garrett Bingham and Risto Miikkulainen, Neural Networks, Vol. 148 (2022), pp. 48-65.
End-to-End Learning to Follow Language Instructions with Compositional Policies 2022
Vanya Cohen, Geraud Nangue Tasse, Nakul Gopalan, Steven James, Ray Mooney, Benjamin Rosman, Workshop on Language and Robot Learning at CoRL 2022 (2022).
Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering 2022
Jialin Wu, Raymond Mooney, In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), December 2022.
Facilitating Software Evolution through Natural Language Comments and Dialogue 2022
Sheena Panthaplackel, PhD Thesis, Department of Computer Science, UT Austin.
Incorporating External Information for Visual Question Answering 2022
Jialin Wu, PhD Thesis, Department of Computer Science, UT Austin.
Planning with Large Language Models via Corrective Re-prompting 2022
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, Stefanie Tellex, Foundation Models for Decision Making Workshop at NeurIPS 2022 (2022).
Using Developer Discussions to Guide Fixing Bugs in Software 2022
Sheena Panthaplackel, Milos Gligoric, Junyi Jessy Li, Raymond J. Mooney, In Findings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), December 2022.
Using Natural Language to Aid Task Specification in Sequential Decision Making Problems 2022
Prasoon Goyal, PhD Thesis, Department of Computer Science, UT Austin.
Zero-shot Video Moment Retrieval With Off-the-Shelf Models 2022
Anuj Diwan, Puyuan Peng, Raymond J. Mooney, In Workshop on Transfer Learning for Natural Language Processing at NeurIPS 2022, December 2022.
Evaluating the Robustness of Natural Language Reward Shaping Models to Spatial Relations 2020
Antony Yun, Undergraduate Honors Thesis, Computer Science Department, University of Texas at Austin.
The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings 2020
Elliot Meyerson and Risto Miikkulainen, arxiv:2010.02354 (2020).
Evolutionary Neural AutoML for Deep Learning 2019
Jason Liang, Elliot Meyerson, Babak Hodjat, Dan Fink, Karl Mutch, and Risto Miikkulainen, In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2019), pp. 401–409 2019.
Evolving Deep Neural Networks 2019
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat, In Artificial Intelligence in the Age of Neural Networks and Brain Computing, Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito (Eds.), pp. 293-312 2019. Amsterdam: Elsev...
Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains 2019
Elliot Meyerson and Risto Miikkulainen, In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019.
Using Natural Language for Reward Shaping in Reinforcement Learning 2019
Prasoon Goyal, Scott Niekum, Raymond J. Mooney, In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, August 2019.
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering 2018
Elliot Meyerson and Risto Miikkulainen, In Proceedings of the Sixth International Conference on Learning Representations (ICLR), Vancouver, Canada 2018.
Generating Animated Videos of Human Activities from Natural Language Descriptions 2018
Angela S. Lin, Lemeng Wu, Rodolfo Corona , Kevin Tai , Qixing Huang , Raymond J. Mooney, In Proceedings of the Visually Grounded Interaction and Language Workshop at NeurIPS 2018, December 2018.
Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back 2018
Elliot Meyerson, Risto Miikkulainen, In Proceedings of the 35th International Conference on Machine Learning, pp. 739-748 2018.
Advances in Statistical Script Learning 2017
Karl Pichotta, PhD Thesis, Department of Computer Science, The University of Texas at Austin.
Captioning Images with Diverse Objects 2017
Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko, In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR-17), pp. 5753--5761 2017.
Natural-Language Video Description with Deep Recurrent Neural Networks 2017
Subhashini Venugopalan, PhD Thesis, Department of Computer Science, The University of Texas at Austin.
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data 2016
Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell, In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR-16), pp. 1--10 2016.
Deep Imitation Learning for Parameterized Action Spaces 2016
Matthew Hausknecht, Yilun Chen, and Peter Stone, In AAMAS Adaptive Learning Agents (ALA) Workshop, Singapore, May 2016.
Deep Reinforcement Learning in Parameterized Action Space 2016
Matthew Hausknecht and Peter Stone, In Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.
Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text 2016
Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, and Kate Saenko, In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-16), pp. 1961--1966, Austin, Texas 2016.
Learning Statistical Scripts with LSTM Recurrent Neural Networks 2016
Karl Pichotta and Raymond J. Mooney, In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16), Phoenix, Arizona, February 2016.
MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification 2016
Ye Zhang, Stephen Roller, and Byron Wallace., In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-16), pp. 1522--1527, San Diego, California 2016.
On-Policy vs. Off-Policy Updates for Deep Reinforcement Learning 2016
Matthew Hausknecht and Peter Stone, In Deep Reinforcement Learning: Frontiers and Challenges, IJCAI Workshop, New York, July 2016.
PIC a Different Word: A Simple Model for Lexical Substitution in Context 2016
Stephen Roller and Katrin Erk, In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-16), pp. 1121-1126, San Diego, California 2016.
Stacking With Auxiliary Features: Improved Ensembling for Natural Language and Vision 2016
Nazneen Fatema Rajani, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Statistical Script Learning with Recurrent Neural Networks 2016
Karl Pichotta and Raymond J. Mooney, In Proceedings of the Workshop on Uphill Battles in Language Processing (UBLP) at EMNLP 2016, Austin, TX, November 2016.
Using Sentence-Level LSTM Language Models for Script Inference 2016
Karl Pichotta and Raymond J. Mooney, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL-16), pp. 279--289, Berlin, Germany 2016.
Deep Recurrent Q-Learning for Partially Observable MDPs 2015
Matthew Hausknecht and Peter Stone, In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents (AAAI-SDMIA15), Arlington, Virginia, USA, November 2015.
Natural Language Video Description using Deep Recurrent Neural Networks 2015
Subhashini Venugopalan, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Sequence to Sequence -- Video to Text 2015
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko, In Proceedings of the 2015 International Conference on Computer Vision (ICCV-15), Santiago, Chile, December 2015.
Statistical Script Learning with Recurrent Neural Nets 2015
Karl Pichotta, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Translating Videos to Natural Language Using Deep Recurrent Neural Networks 2015
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko, In Proceedings the 2015 Conference of the North American Chapter of the Association for Computational Linguistics -- Human Language Technologies (NAACL HLT 2015), pp. 1494--1504, Denver, Colora...
Labs
Machine Learning Learning Agents