The map-reduce programming paradigm is a fundamental tool used in processing large data sets, and is supported in current tools such as Hadoop. Apache Spark offers another programming paradigm for processing large data sets. In this course you will gain an understanding of the concepts embodied in map-reduce, and will investigate how map-reduce is used to address various problems in processing and analyzing large data sets. This course will explore map-reduce as implemented in Hadoop, as well as the associated distributed file system (HDFS). In this course you will gain an understanding of the concepts offered and supported in Spark, and will investigate how to apply these concepts to address various problems including those you addressed using map-reduce.

Upon completing this course, the student will be able to design and implement map-reduce programs for various large data set processing tasks, and will be able to design and implement programs using Apache Spark.

Data structures, Java programming experience.

• Required: MapReduce Design Patterns, by Donald Miner and Adam Shook
  • O'Reilly Media
  • Print ISBN: 978-1-4493-2717-0 | ISBN 10: 1-4493-2717-6
• Required: Learning Spark, by Holden Karau, Andy Konwinsky, Patrick Wendell, Matei Zaharia
  • O'Reilly Media
  • Print ISBN: 978-1-4493-5862-4 | ISBN 10: 1-4493-5862-4
• Recommended: Hadoop: The Definitive Guide, 4th Edition, by Tom White
  • O'Reilly Media/Yahoo Press
  • Print ISBN: 978-1-4919-0163-2 | ISBN 10: 1-4919-0163-2

David Franke
Email: dfranke at cs.utexas.edu
Office: GDC 6.402
Office Hours:

• T 11:00 AM - 12:00 PM
• Th 12:30 PM - 1:30 PM
• Or by appointment

Vivek Pradhan
Email: vivek.k.pradhan at utexas.edu
Office: GDC 1.302, Desk 2
Office Hours:

• M 2:15 PM - 3:30 PM

This course will focus on writing code to solve various problems, so assignments will be programming assignments. These programs will be cumulative in that subsequent assignments will build on previous programs you have written, so it is important to complete assignments on time so you can move on to the next assignment.

Small datasets will be provided for each assignment so that you will not consume too much computing resource (time and space) while developing your solution. Some assignments will also offer a large dataset so that you can measure how your map-reduce solution scales with the dataset size and the computing resources available.

You are free to discuss approaches to solving the assigned problems with your classmates, but each student is expected to write their own code. Source code must be submitted for each assignment, in addition to the results you obtained when running your program against the datasets provided. If duplicate work is detected, all parties involved will be penalized. All students should read and be familiar with the UTCS Rules to Live By.

Grading Rubric:

• Does the program run
• Does the program produce the correct output
• Does the program contain the required element(s)
• Code quality: structure and documentation

Percentages for each element may be different for each assignment.

Late Policy

Required artifacts for each programming assignment are due at the start of class (9:30 AM) on the due date, as we will be discussing the solution during that class period. Penalty for late submission is 25%.

Special Notes on Assignment Submission and Grading:

• If you submit your assignment before the due date/time, that is what will be graded (the last submission before the due date/time). Additional late submissions will not be considered. You must decide between submitting a partially working program before the deadline versus a (more complete) program after the deadline.
• Late submissions will only be accepted for one week following the due date for points.
• Submissions made after this time (one week after the due date/time) will get some consideration when final grades are determined, so I encourage you to turn in your work even after the extended late deadline.

Other Considerations

In using cloud-based services such as AWS, you will have an account that is charged for the various resources used. You are responsible for shutting down any services you start-up in the course of doing your assignments. It is possible to start services (like Elastic Map Reduce) and leave them running even though you are not doing any work. I encourage you to shut down any services you have started at the end of a work session. If you repeatedly leave services running that you are not using, your account will be charged and you may exhaust the free credits of your account.

Final Grades

Final grade will be determined on the cumulative percentage score over all assignments and the exams. Assignment and exam percentages are:

• Assignments: 75%
• Exam 1: 10%
• Exam 2: 15%

• A: 92% to 100%
• A-: 90% to 92%
• B+: 88% to 90%
• B: 82% to 88%
• B-: 80% to 82%
• C+: 78% to 80%
• C: 72% to 78%
• C-: 70% to 72%
• D+: 68% to 70%
• D: 62% to 68%
• D-: 60% to 62%

• Aug. 31: Lecture Notes - 1
• Sep. 5: Lecture Notes - 2
• Sep. 7: Lecture Notes - 3
• Sep. 12: Lecture Notes - 4
• Sep. 14: Lecture Notes - 5
• Sep. 19: Lecture Notes - 6
• Sep. 21: Lecture Notes - 7
• Sep. 26: Lecture Notes - 8
• Sep. 28: Lecture Notes - 9
• Oct. 3: Lecture Notes - 10
• Oct. 5: Lecture Notes - 11
• Oct. 10: Lecture Notes - 12
• Oct. 12: Lecture Notes - 13
• Oct. 17: Lecture Notes - 14
• Oct. 19: Lecture Notes - 15
• Oct. 24: Lecture Notes - 16
• Oct. 26: Lecture Notes - 17 , 17A
• Oct. 31: Lecture Notes - 18
• Nov. 2: Lecture Notes - 19
• Nov. 7: Lecture Notes - 20
• Nov. 9: Lecture Notes - 21
• Nov. 14: Lecture Notes - 22
• Nov. 16: Lecture Notes - 23
• Nov. 21: Lecture Notes - 24
• Nov. 28: Lecture Notes - 25
• Nov. 30: Lecture Notes - 26 , Spark Streaming
• Dec. 5: Lecture Notes - 27

Further Reading