The map-reduce programming paradigm is a fundamental tool used in processing large data sets, and is supported in current tools such as Hadoop and MongoDB. In this course you will gain an understanding of the concepts embodied in map-reduce, and will investigate how map-reduce is used to address various problems in processing and analyzing large data sets. This course will explore map-reduce as implemented in Hadoop, as well as the associated distributed file system (HDFS). This course will also survey some of the tools built on top of Hadoop and HDFS.

Upon completing this course, the student will be able to write map-reduce programs for various large data set processing tasks, and understand how various tools in the Hadoop ecosystem utilize map-reduce and the Hadoop distributed file system (HDFS).

Data structures, Java programming experience.

• Required: MapReduce Design Patterns, by Donald Miner and Adam Shook
  • O'Reilly Media
  • Print ISBN: 978-1-4493-2717-0 | ISBN 10: 1-4493-2717-6
  • Ebook ISBN: 978-1-4493-4197-8 | ISBN 10: 1-4493-4197-7
• Recommended: Hadoop: The Definitive Guide, 3rd Edition, by Tom White
  • O'Reilly Media/Yahoo Press
  • Print ISBN: 978-1-4493-1152-0 | ISBN 10: 1-4493-1152-0
  • Ebook ISBN: 978-1-4493-1151-3 | ISBN 10: 1-4493-1151-2

David Franke
Email: dfranke@cs.utexas.edu
Office: GDC 4.706
Office Hours:

• T 11:00 AM - 12:00 PM
• Th 8:15 AM - 9:15 AM
• By appointment

Swadhin Pradhan
Email: swadhin@utexas.edu
Office: GDC 6.802D
Office Hours: M-W 3:00 - 4:30 PM, Desk 1, GDC TA Station 1.310

This course will focus on writing code to solve various problems, so assignments will be programming assignments. These programs will be cumulative in that subsequent assignments will build on previous programs you have written, so it is important to complete assignments on time so you can move on to the next assignment.

Small datasets will be provided for each assignment so that you will not consume too much computing resource (time and space) while developing your solution. Some assignments will also offer a large dataset so that you can measure how your map-reduce solution scales with the dataset size and the computing resources available.

You are free to discuss approaches to solving the assigned problems with your classmates, but each student is expected to write their own code. Source code must be submitted for each assignment, in addition to the results you obtained when running your program against the datasets provided. If duplicate work is detected, all parties involved will be penalized. All students should read and be familiar with the UTCS Rules to Live By.

Grading Rubric:

• Does the program run
• Does the program produce the correct output
• Does the program contain the required element(s)
• Code quality: structure and documentation

Late Policy

Required artifacts for each programming assignment are due at the start of class (9:30 AM) on the due date, as we will be discussing the solution during that class period. Penalty for late submission is 25%.

Special Notes on Assignment Submission and Grading:

• If you submit your assignment before the due date/time, that is what will be graded (the last submission before the due date/time). Additional late submissions will not be considered. You must decide between submitting a partially working program before the deadline versus a (more complete) program after the deadline.
• Late submissions will only be accepted for one week following the due date.

• Sep. 2: Lecture Notes - 2
• Sep. 4: Lecture Notes - 3
• Sep. 9: Lecture Notes - 4
• Sep. 11: Lecture Notes - 5
• Sep. 16: Lecture Notes - 6
• Sep. 18: Lecture Notes - 7
• Sep. 23: Lecture Notes - 8
• Sep. 25: Lecture Notes - 9
• Sep. 30: Lecture Notes - 10
• Oct. 2: Lecture Notes - 11
• Oct. 7: Lecture Notes - 12
• Oct. 9: Lecture Notes - 13
• Oct. 14: Lecture Notes - 14
• Oct. 16: Lecture Notes - 15
• Oct. 21: Lecture Notes - 16
• Oct. 23: Lecture Notes - 17
• Oct. 28: Lecture Notes - 18
• Oct. 30: Lecture Notes - 19
• Nov. 4: Lecture Notes - 20
• Nov. 6: Lecture Notes - 21
• Nov. 11: Lecture Notes - 22
• Nov. 13: Lecture Notes - 23
• Nov. 18: Lecture Notes - 24
• Nov. 25: Lecture Notes - 25
• Dec. 2: Lecture Notes - 26
• Dec. 4: Lecture Notes - 27

Further Reading