Learning Theory

CS395T, Fall 2011

BUR 212, Mon & Wed 2:00 - 3:30 pm

Instructors

Office Hours

Yi-Chao Chen (TA): TA Station 5, PAI 5.33, Thursdays 1:00 - 4:00 pm

Pradeep Ravikumar: ACES 2.434, Fridays 3:30-5:00 pm
Adam Klivans: TBD

Overview

A central problem in machine learning is to develop algorithms that have provable guarantees in terms of both running time and number of "training" observations required. Computational Learning Theory has traditionally focused on the first issue (the computational complexity of learning algorithms) while Statistical Learning Theory has focused on the second (their statistical efficiency). In this course we will cover both these aspects, and try to understand how learning is constrained given limited computation and limited data.

Grading

Four problem sets (3/4 of final grade), and a final paper presentation (1/4 of final grade).

(Optional) Textbooks/Papers

Statistical Learning Theory:
All of Statistics. Larry Wasserman.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Trevor Hastie, Robert Tibshirani, Jerome Friedman.
A Probabilistic Theory of Pattern Recognition. Luc Devroye, Laszlo Györfi, Gabor Lugosi.

Comp. Learning Theory:
An Introduction to Computational Learning Theory. Michael Kearns, Umesh Vazirani.

Background on Probability and Statistics:
Introduction to Probability and Statistics. Dimitri P. Bertsekas and John N. Tsitsiklis.
Statistical Inference. George Casella, Roger L. Berger.

Graphical Models:
Graphical models, exponential families, and variational inference. M. J. Wainwright and M. I. Jordan. Foundations and Trends in Machine Learning, Vol. 1, Numbers 1--2, pp. 1--305, December 2008.
Probabilistic Graphical Models: Principles and Techniques . D. Koller and N. Friedman.

High-dimensional Statistics:
Paper: Sharp thresholds for noisy and high-dimensional recovery of sparsity using $\ell_1$-constrained quadratic programming (Lasso). M. J. Wainwright. IEEE Transactions on Information Theory, 55:2183--2202, May 2009.
Paper: A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. S. Negahban, P. Ravikumar, M. J. Wainwright and B. Yu, 2010.

Homeworks

Homework 1 is out. Due Oct 10, 2010.

Schedule

Module	Date	Topic	Notes	Faculty
1: Statistical Models	08/24	Comptastical Thinking		PR
	08/29	Exponential Families, Generalized Linear Models	Scribe Notes	PR
	09/07	Statistical Models for Ranking	Scribe Notes	Ambuj Tewari
2: Graphical Models	08/31	Graphical Models : Introduction	Scribe Notes	PR
	09/12	Graphical Models : Inference I (Variable Elimination, Sum-product)	Scribe Notes	PR
	09/14	Graphical Models : Inference II (Junction Trees)	Scribe Notes	PR
	09/19	Graphical Models : Inference/Learning as Optimization I	Scribe Notes	PR
	09/21	Graphical Models : Inference/Learning as Optimization II	Scribe Notes	PR
3: High-dimensional Models	09/26	High-dimensional Statistics: Introduction (Sparsity, Group-Sparsity, Low-Rank)	Slides	PR
3: High-dimensional Models	09/28	High-dimensional Statistical Analysis (Sparsistency, Parameter Error Bounds)	Slides	PR
4: Risk Bounds	10/03	ERM, {Estimation, Approximation} Error	DGL; Chapter 12	AT
	10/05	Uniform Deviation Bounds, Glivenko-Cantelli Theorem	DGL; Chapter 12	AT
	10/10	Glivenko-Cantelli Theorem (contd.), VC Theorem	DGL; Chapter 12	AT