CS378: Natural Language Processing (Fall 2020)
NOTE: This page is for an old semester of this class
Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 9:30am - 11:00am, on Zoom (see Canvas for Zoom links)
Instructor Office Hours: Tuesday 11am-12pm, Wednesday 4pm-5pm (see Canvas for Zoom links)
TA: Tanya Goyal; Proctor: Shivang Singh
TA Office Hours (see Canvas for Zoom links):
- Monday 3pm-4pm (Tanya)
- Thursday 3pm-4pm (Tanya)
- Friday 2:30pm (Shivang)
Piazza 
Description
This course provides an introduction to modern natural language processing
using machine learning and deep learning approaches. Content includes
linguistics fundamentals (syntax, semantics, distributional properties of
language), machine learning models (classifiers, sequence taggers, deep
learning models), key algorithms for inference, and applications to a range of
problems. Students will get hands-on experience building systems to do tasks
including text classification, syntactic analysis, language modeling, and language generation.
Requirements
- CS 429
- Recommended: CS 331, familiarity with probability and linear algebra, programming experience in Python
- Helpful: Exposure to AI and machine learning (e.g., CS 342/343/363)
Detailed syllabus with course policies
Assignments: The assignments are largely similar to those from last year, but they are subject to changes in both specific
questions as well as framework code.
Assignment 0: Warmup [nyt dataset] [tokenizer.py]
Assignment 1: Sentiment Classification [code and dataset download]
Assignment 2: Feedforward Neural Networks and Optimization [code and dataset download]
Assignment 3: Sequence Modeling and Parsing [code and dataset on Canvas]
Take-home Midterm (topics) [midterm on Canvas] [spring 2020 midterm / solutions, spring 2019 midterm / solutions]
Assignment 4: Character Language Modeling with RNNs [code and dataset download]
Assignment 5: Machine Translation [code and dataset download]
Final Project: Independent Project (propose by Oct 27) or Question Answering [Github repo]
Readings: Textbook readings are assigned to complement the material discussed in lecture. You may find it useful
to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed
in the textbook if it isn't covered in lecture.
Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics.
Bold readings and videos are most central to the course content; it's recommended that you look at these.
The chief text in this course is Eisenstein: Natural Language Processing,
available as a free PDF online. For deep learning techniques, this text will be supplemented with selections from Goldberg: A Primer on Neural Network Models for Natural Language Processing.
(Another generally useful NLP book is Jurafsky and Martin: Speech and Language Processing (3rd ed. draft), with many draft chapters available for free online; however,
we will not be using it much for this course.)
Readings for future lectures are tentative and subject to change.
| Date |
Topics |
Readings |
Assignments |
| Aug 27 |
Introduction [4pp] |
|
A0 out (ungraded) |
| Sept 1 |
Classification 1: Features, Perceptron |
Classification lecture note
Perceptron Loss (VIDEO)
perc_lecture_plot.py
Eisenstein 2.0, 2.1, 2.3.1, 4.1, 4.3
|
|
| Sept 3 |
Classification 2: Logistic Regression |
Classification lecture note
Optimization (VIDEO)
Jurafsky and Martin 5.0-5.3
|
|
| Sept 8 |
Classification 3: Sentiment, Multiclass (slides: [1pp] [4pp]) |
Multiclass lecture note
Fairness (VIDEO)
Eisenstein 2.4.1, 2.5, 2.6, 4.2
Pang+02
Wang+Manning12
Socher+13 Sentiment
|
|
| Sept 10 |
Classification 4: Multiclass Log. Reg. / Neural 1: Feedforward, Backpropagation (slides: [1pp], [4pp]) |
Multiclass lecture note
Eisenstein 3.0-3.3
Goldberg 4
Schwartz+13 Authorship |
|
| Sept 15 |
Neural 2: FFNNs, Implementation, DANs [4pp] |
Neural Net Optimization (VIDEO)
ffnn_example.py
Eisenstein 3.3
Goldberg 3, 6
Iyyer+15 DANs
Init and backprop |
A1 due/A2 out |
| Sept 17 |
Neural 3: Word embeddings |
Eisenstein 14.5-14.6
Goldberg 5
Mikolov+13 word2vec
Pennington+14 GloVe
|
|
| Sept 22 |
Neural 4: Bias in embeddings (slides: [1pp], [4pp]) / Sequence 1: Tagging, POS |
Other embedding methods (VIDEO)
Bolukbasi+16 Bias
Eisenstein 8.1
|
|
| Sept 24 |
Sequence 2: Tagging as Classification, HMMs |
Viterbi lecture note
Eisenstein 7.1-7.4
|
|
| Sept 29 |
Sequence 3: Viterbi, Beam Search |
Viterbi lecture note
HMMs for POS Tagging (VIDEO)
Eisenstein 7.4.2
|
A2 due/A3 out |
| Oct 1 |
Sequence 4: CRFs / Trees 1: Constituency [4pp] |
Eisenstein 7.5-7.6
Manning POS
|
|
| Oct 6 |
Trees 2: PCFGs, CKY |
Eisenstein 10.1-3, 10.4.1
KleinManning03 Unlexicalized
|
|
| Oct 8 |
Trees 3 (notes): Finish constituency, Dependency (slides: [1pp], [4pp]) |
Refining grammars (VIDEO)
Eisenstein 11.3-4
Chen and Manning14
Andor+16
|
|
| Oct 13 |
Trees 4: Shift-Reduce; Midterm review |
State-of-the-art Parsers (VIDEO)
|
A3 due |
| Oct 15 |
Take-home Midterm (NO CLASS) |
|
Midterm out Oct 14 9am, due Oct 16 5pm |
| Oct 20 |
LM 1: N-grams, RNNs |
Eisenstein 6.1-6.2
|
|
| Oct 22 |
LM 2: LSTMs, Implementation (slides: [1pp] [4pp]) |
lstm_lecture.py
Eisenstein 6.3-6.5
Olah Understanding LSTMs
LSTM PyTorch documentation
RNNs with PyTorch
Karpathy Visualizing RNNs
|
A4 out |
| Oct 27 |
Finish LSTMs (slides from last time) / MT 1: Alignment, Phrase-based MT |
Eisenstein 18.1-18.2, 18.4
Michael Collins IBM Models 1+2
JHU slides
History of MT
|
Custom FP proposals due (NEW DUE DATE) |
| Oct 29 |
MT 2: Phrase-based, seq2seq (slides: [1pp], [4pp]) |
Eisenstein 18.2
|
|
| Nov 3 |
Generation: Dialogue, Summarization, Ethics [OPTIONAL] [4pp] |
Yu+19 Gunrock
Radford+19 GPT-2
Adiwardana+20 Google Meena
Roller+20 Facebook Blender
Gehman+20 Toxicity
|
|
| Nov 5 |
MT 3: Attention, Neural MT (slides: [1pp], [4pp]) |
Eisenstein 18.3
Luong+15 Attention
Wu+16 Google
SennrichZhang19 Low-resource
|
A4 due / A5 out |
| Nov 10 |
QA 1: Semantic representations, semantic parsing (slides: [1pp], [4pp]) |
Eisenstein 12
ZettlemoyerCollins05
JiaLiang16
|
|
| Nov 12 |
QA 2: Reading comprehension |
Eisenstein 17.5
Stanford Attentive Reader
SQuAD
DrQA
QA span visualization |
A5 due / FP out |
| Nov 17 |
QA 3 / Pre-training 1: ELMo (slides: [1pp], [4pp]) |
ELMo |
|
| Nov 19 |
Pre-training 2: Transformers / BERT (slides: [1pp], [4pp]) |
Transformers
Illustrated Transformer
BERT
|
|
| Nov 24 |
NLP in other languages, cross-lingual models [4pp] |
Xlingual POS
Xlingual parsing
Xlingual embeddings
Multilingual sent embs
XLM-R
How multilingual is mBERT? |
FP check-in due November 24 |
| Nov 26 |
NO CLASS |
|
|
| Dec 1 |
Interpreting Neural Models [4pp] |
Ribeiro+16 LIME
Simonyan+13 Visualizing
Nguyen18 Evaluating Explanations
Rudin+18 Stop Explaining
Bansal+20 Whole Exceed Its Parts
EMNLP 2020 Interpretability tutorial
|
|
| Dec 3 |
Wrapup + Ethics [4pp] |
HovySpruit2016 Social Impact of NLP
Zhao+17 Bias Amplification
Rudinger+18 Gender Bias in Coref
Gebru+18 Datasheets for Datasets
Raji+20 Auditing
|
FP due Dec 9 |