CS395T: Robot Manipulation and Learning

Adaptive Planning and Control Methods to Operate in Unstructured Environments

 

Course Description

The ultimate goal of a robot is to manipulate its environment: change the environment's state through interactions in order to achieve a task. Over the decades, multiple techniques have been proposed for robot manipulation, including solutions for grasping, planning, motion, and interaction control. These techniques are successful in industrial settings where robots work in controlled and known environments. However, classical solutions for robot manipulation fail to generalize to less structured domains such as human homes, offices, and campuses. How can robots manipulate successfully in these general environments? How can they become dexterous active helpers in our homes? This course provides a deep overview of the most relevant techniques in robot manipulation with a special emphasis on those based on learning and how they have recently achieved higher levels of generalization and adaptation. The course covers a) classical foundational methodologies that provide a theoretical basis and help understand the challenges of robot manipulation, and b) modern learning methodologies that overcome the manipulation challenges by learning from experiences or from imitating (human) experts. The topics covered will include methods for sensorimotor control of manipulation and manipulation planning, providing a comprehensive overview of the elements necessary to design a manipulating robot. It will include solutions based on optimization, search, control, representation, reinforcement and imitation learning.

Course Time and Location
Time: Tue/Thu 3:30 to 5pm
Location: GDC 4.302

Online Platforms
Ed Discussion
Canvas
Gradescope

Instructor

Roberto Martín-Martín
OH: Tue 5-6 pm
Office: GDC 3.510
Email: robertom [at] cs.utexas.edu

Teaching Assistant

Arpit Bahety
OH: 1st & 3rd Thu of a month 2:30-3:30 pm
Location: GDC 3.504C
Email: abahety [at] cs.utexas.edu

Learning Objective

This course is oriented toward graduate students and undergrads with strong backgrounds in robotics, machine learning, and artificial intelligence. The students should be excited about the recent advances in robot manipulation and be curious about the foundation elements that lead to these advances. They should be willing to learn to conduct research in the area. During the course, the students will:

  • learn the most relevant classical techniques in robot manipulation (grasping, sensorimotor control, planning, perception for manipulation), their theoretical grounding, their limitations, and the challenges that make these manipulation problems hard;
  • learn the most recent Robot Manipulation and Learning algorithms developed to overcome the limitations of classical techniques, their foundational elements, the current research trends, and the still-existing limitations in robot learning techniques for manipulation;
  • be able to evaluate, communicate, and apply advanced techniques to problems in robot manipulation.

Prerequisites

Students should have the following background:

  • Knowledge of basic data structures and algorithms as well as practical skills in computer programming. Proficiency in Python is required and high-level familiarity with C/C++ is a plus.
  • Familiarity with calculus, statistics, and linear algebra. Strong mathematical skills.
  • Coursework and/or equivalent experience in AI and Machine Learning (CS342, CS391L, and CS394R) are preferred. Practical experience with training machine learning models and/or developing robot manipulation solutions in the real world or simulation is strongly recommended.
  • Be excited, ambitious, and patient when working with robot manipulation and learning systems.

Note: This course is an advanced graduate-level course. If you are unclear whether you meet these requirements, please consult the instructor in advance. Undergraduates must obtain explicit approval from the instructor (email your CV and transcript) prior to enrollment.