CS395T: Robot Manipulation and Learning

Adaptive Planning and Control Methods to Operate in Unstructured Environments

 

Course Description

The ultimate goal of a robot is to manipulate its environment: change the environment's state through interactions in order to achieve a task. Over the decades, multiple techniques have been proposed for robot manipulation, including solutions for grasping, planning, motion, and interaction control. These techniques are successful in industrial settings where robots work in controlled and known environments. However, classical solutions for robot manipulation fail to generalize to less structured domains such as human homes, offices, and campuses. How can robots manipulate successfully in these general environments? How can they become dexterous active helpers in our homes? This course provides a deep overview of the most relevant techniques in robot manipulation with a special emphasis on those based on learning and how they have recently achieved higher levels of generalization and adaptation. The course covers a) classical foundational methodologies that provide a theoretical basis and help understand the challenges of robot manipulation, and b) modern learning methodologies that overcome the manipulation challenges by learning from experiences or from imitating (human) experts. The topics covered will include methods for sensorimotor control of manipulation and manipulation planning, providing a comprehensive overview of the elements necessary to design a manipulating robot. It will include solutions based on optimization, search, control, representation, reinforcement and imitation learning.

Course Time and Location
Time: Tue/Thu 9:30 to 11am
Location: GDC 1.406

Online Platforms
Ed Discussion (TODO: change link)
Canvas
Gradescope

Instructor

Roberto Martín-Martín
OH: Tue 11am - 12pm
Office: GDC 3.510
Email: robertomm [at] cs.utexas...

Teaching Assistant

Albert Yu
OH: Fri 12:30pm - 1:30pm
Location: GDC 4.416
Email: albertyu [at] utexas...

Learning Objective

This course is oriented toward graduate students and undergrads with strong backgrounds in robotics, machine learning, and artificial intelligence. The students should be excited about the recent advances in robot manipulation and be curious about the foundation elements that lead to these advances. They should be willing to learn to conduct research in the area. During the course, the students will:

  • learn the most relevant classical techniques in robot manipulation (grasping, sensorimotor control, planning, perception for manipulation), their theoretical grounding, their limitations, and the challenges that make these manipulation problems hard;
  • learn the most recent Robot Manipulation and Learning algorithms developed to overcome the limitations of classical techniques, their foundational elements, the current research trends, and the still-existing limitations in robot learning techniques for manipulation;
  • be able to evaluate, communicate, and apply advanced techniques to problems in robot manipulation.

Prerequisites

Students should have the following background:

  • Knowledge of basic data structures and algorithms as well as practical skills in computer programming. Proficiency in Python is required and high-level familiarity with C/C++ is a plus.
  • Familiarity with calculus, statistics, and linear algebra. Strong mathematical skills.
  • Coursework and/or equivalent experience in AI and Machine Learning (CS342, CS391L, and CS394R) are preferred. Practical experience with training machine learning models and/or developing robot manipulation solutions in the real world or simulation is strongly recommended.
  • Prior experience with robotics, robot learning and/or simulators is highly encouraged. The course has a project component that builds on some basic skills to program robot learning solutions, or the drive to learn those on your own to apply the techniques we see in the seminar.
  • Be excited, ambitious, and patient when working with robot manipulation and learning systems.

Note: This course is an advanced graduate-level course. If you are unclear whether you meet these requirements, please consult the instructor in advance. Undergraduates must obtain explicit approval from the instructor (email your CV and transcript) prior to enrollment.