Lecture Course Overview

• Building Machines That Learn and Think Like People. Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman (2016)
• Intelligence without Reason. Rodney Brooks (1991)

Lecture Overview of Robot Perception

• The Limits and Potentials of Deep Learning for Robotics. Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, Peter Corke (2018)
• A Sensorimotor Account of Vision and Visual Consciousness. Kevin O'Regan and Alva Noë (2001)

2D Visual Recognition

• You Only Look Once: Unified, Real-Time Object Detection. Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi (2015)
• Segment Anything. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick (2023)

• End-to-End Object Detection with Transformers. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko (2020)
• Mask R-CNN. Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick (2017)

3D Data Processing

• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas (2017)
• Dynamic Graph CNN for Learning on Point Clouds. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, Justin M. Solomon (2018)

• PointPillars: Fast Encoders for Object Detection from Point Clouds (2019). Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, Oscar Beijbom (2019)
• PointCNN: Convolution On X-Transformed Points. Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen (2018)

Representation Learning for Robotics

• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. Peter R. Florence, Lucas Manuelli, Russ Tedrake (2018)
• R3M: A Universal Visual Representation for Robot Manipulation. Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, Abhinav Gupta (2022)

• Emerging Properties in Self-Supervised Vision Transformers. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin (2021)
• Unsupervised Learning of Object Keypoints for Perception and Control. Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, Volodymyr Mnih (2019)

Tutorial Tutorial on the robosuite simulation framework for robot learning

Neural Fields

• Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations. Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, Yuke Zhu (2021)
• iMAP: Implicit Mapping and Positioning in Real-Time. Edgar Sucar, Shikun Liu, Joseph Ortiz, Andrew J. Davison (2021)

• 3D Neural Scene Representations for Visuomotor Control. Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, Antonio Torralba (2021)
• NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys (2022)

Multimodal Perception

• Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks. Michelle A. Lee*, Yuke Zhu*, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg (2018)
• Audio-Visual Embodied Navigation. Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, Kristen Grauman (2019)

• More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch. Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine (2018)
• ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer. Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, Jiajun Wu (2022)

State Estimation

• Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Rico Jonschkowski, Divyam Rastogi, Oliver Brock (2018)
• DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, Silvio Savarese (2019)

• Particle Filter Networks with Application to Visual Localization. Peter Karkus, David Hsu, Wee Sun Lee (2018)
• Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield (2018)

Visual Tracking

• DART: Dense Articulated Real-Time Tracking. Tanner Schmidt, Richard Newcombe, Dieter Fox (2010)
• Tracking Everything Everywhere All at Once. Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely (2023)

• Category-Independent Articulated Object Tracking with Factor Graphs. Nick Heppert, Toki Migimatsu, Brent Yi, Claire Chen, Jeannette Bohg (2022)
• Tracking People by Predicting 3D Appearance, Location & Pose. Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Jitendra Malik (2021)

Lecture Overview of Robot Decision Making

• Reinforcement Learning in Robotics: A Survey. Jens Kober, J. Andrew Bagnell, Jan Peters (2013)
• Recent Advances in Robot Learning from Demonstration. Harish Ravichandar, Athanasios S. Polydoros, Sonia Chernova, Aude Billard (2020)

Model-free Reinforcement Learning

• Trust Region Policy Optimization. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel (2015)
• Soft Actor-Critic Algorithms and Applications. Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine (2018)

• Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017)
• Continuous Control with Deep Reinforcement Learning. Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra (2015)

Tutorial Tutorial on the PyTorch deep learning framework

Model-based Reinforcement Learning

• Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning. Marc Deisenroth, Carl Edward Rasmussen, Dieter Fox (2012)
• Dream to Control: Learning Behaviors by Latent Imagination. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi (2019)

• Deep Visual Foresight for Planning Robot Motion. Chelsea Finn, Sergey Levine (2016)
• Benchmarking Model-Based Reinforcement Learning. Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba (2019)

Imitation as Supervised Learning

• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell (2010)
• Agile Autonomous Driving using End-to-End Deep Imitation Learning. Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos A. Theodorou, and Byron Boots (2017)

• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, Pieter Abbeel (2017)
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song (2023)

Offline Reinforcement Learning

• Off-Policy Deep Reinforcement Learning without Exploration. Scott Fujimoto, David Meger, Doina Precup (2019)
• Offline Reinforcement Learning with Implicit Q-Learning. Ilya Kostrikov, Ashvin Nair, Sergey Levine (2021)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Sergey Levine, Aviral Kumar, George Tucker, Justin Fu (2020)
• MOPO: Model-based Offline Policy Optimization. Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma (2020)

Human-in-the-Loop Learning

• Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment. Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, Yuke Zhu (2022)
• Deep Reinforcement Learning from Human Preferences. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei (2017)

• ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning. Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S. Brown, Ken Goldberg (2021)
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. William Saunders, Girish Sastry, Andreas Stuhlmueller, Owain Evans (2017)

Learning to Learn

• RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel (2016)
• Meta-Learning with Memory-Augmented Neural Networks. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap (2016)

• Optimization as a Model for Few-Shot Learning. Sachin Ravi, Hugo Larochelle (2017)
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning. Xingyou Song, Yuxiang Yang, Krzysztof Choromanski, Ken Caluwaerts, Wenbo Gao, Chelsea Finn, Jie Tan (2020)

Task and Motion Planning

• Hierarchical Task and Motion Planning in the Now. Leslie Pack Kaelbling, Tomás Lozano-Pérez (2011)
• Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning. Marc Toussaint, Kelsey Allen, Kevin Smith, Joshua Tenenbaum (2014)

• Object-Centric Task and Motion Planning in Dynamic Environments. Toki Migimatsu, Jeannette Bohg (2019)
• PETLON: Planning Efficiently for Task-Level-Optimal Navigation. Shih-Yun Lo, Shiqi Zhang, Peter Stone (2018)

Attention Architectures

• VIMA: General Robot Manipulation with Multimodal Prompts. Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi Fan (2023)
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation. Mohit Shridhar, Lucas Manuelli, Dieter Fox (2022)

• RVT: Robotic View Transformer for 3D Object Manipulation. Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox (2023)
• Perceiver IO: A General Architecture for Structured Inputs & Outputs. Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, Joāo Carreira (2022)

Language in Robotics

• ConceptFusion: Open-set Multimodal 3D Mapping. Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, Antonio Torralba (2023)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich (2023)

• Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations. Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg (2020)
• TidyBot: Personalized Robot Assistance with Large Language Models. Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas Funkhouser (2023)

Sim-to-Real Gap

• Driving Policy Transfer via Modularity and Abstraction. Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, Vladlen Koltun (2018)
• Learning Agile and Dynamic Motor Skills for Legged Robots. Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter (2019)

• RMA: Rapid Motor Adaptation for Legged Robots. Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik (2021)
• IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Reality. Bingjie Tang, Michael A. Lin, Iretiayo Akinola, Ankur Handa, Gaurav S. Sukhatme, Fabio Ramos, Dieter Fox, Yashraj Narang (2023)

Program Synthesis for Embodied Agents

• Code as Policies: Language Model Programs for Embodied Control. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng (2022)
• Voyager: An Open-Ended Embodied Agent with Large Language Models. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (2023)

• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg (2022)
• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, Joshua B. Tenenbaum (2020)

Guest Lecture I: Ted Xiao and Yevgen Chebotar and Quan Vuong, Google DeepMind

Guest Lecture II: Kuan Fang, UC Berkeley / Cornell University

Lecture Lecture: Open Questions in Robot Learning

• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms. Oliver Kroemer, Scott Niekum, George Konidaris (2019)