Machine Learning Systems (Spring 2022)

• When: Mondays from 1:00 to 4:00
• Where: Soda 405 (and on zoom with with link posted on Slack).
• Instructor: Joseph E. Gonzalez
• Co-Instructor: Amir Gholami
• Office Hours:: Arrange by email.
• Announcements: Slack (please send us an email if you are not added yet)
• Sign-up to Present: Every student should sign-up to present in at least three rows and as different roles each time. Note that the Backup/Scribe presenter may be asked to fill in for one of the other roles with little notice.
• If you have reading suggestions please send a pull request to this course website on Github by modifying the index.md file.

Course Description

The recent success of AI has been in large part due in part to advances in hardware and software systems. These systems have enabled training increasingly complex models on ever larger datasets. In the process, these systems have also simplified model development, enabling the rapid growth in the machine learning community. These new hardware and software systems include a new generation of GPUs and hardware accelerators (e.g., TPU), open source frameworks such as Theano, TensorFlow, PyTorch, MXNet, Apache Spark, Clipper, Horovod, and Ray, and a myriad of systems deployed internally at companies just to name a few. At the same time, we are witnessing a flurry of ML/RL applications to improve hardware and system designs, job scheduling, program synthesis, and circuit layouts.

In this course, we will describe the latest trends in systems designs to better support the next generation of AI applications, and applications of AI to optimize the architecture and the performance of systems. The format of this course will be a mix of lectures, seminar-style discussions, and student presentations. Students will be responsible for paper readings, and completing a hands-on project. For projects, we will strongly encourage teams that contains both AI and systems students.

New Course Format

Two previous versions of this course were offered in Spring 2019, and Fall 2019. The format of this third offering is slightly different. Each week will cover a different research area in AI-Systems. The lecture will be organized around a mini program committee meeting for the weeks readings. Students will be required to submit detailed reviews for a subset of the papers and lead the paper review discussions. For some of the topics, we have also invited prominent researchers for each area and they will present an overview of the field, followed by discussions raised during the “committee meeting”. The goal of this new format is to both build a mastery of the material and also to develop a deeper understanding of how to evaluate and review research and hopefully provide insight into how to write better papers.

Course Syllabus

This is a tentative schedule. Specific readings are subject to change as new material is published.

Jump to Today

Week	Date	Topic
1	1/24/22	Introduction and Course Overview This lecture will be an overview of the class, requirements, and an introduction to the history of machine learning and systems research. Lecture slides: [pdf, pptx] SysML: The New Frontier of Machine Learning Systems Read Chapter 1 of Principles of Computer System Design. You will need to be on campus or use the Library VPN to obtain a free PDF. A Few Useful Things to Know About Machine Learning How to read a paper provides some pretty good advice on how to read papers effectively. Timothy Roscoe’s writing reviews for systems conferences will also help you in the reviewing process.
2	1/31/22	Big Data Systems Guest Speaker: Reynold Xin (Databricks) Submit your review before 1:00PM. Lecture slides: PDF, PPTX Guest Lecture slides: PDF Towards a Unified Architecture for in-RDBMS Analytics Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores Spark SQL: Relational Data Processing in Spark The MADlib Analytics Library or MAD Skills, the SQL
3	2/07/22	Hardware for Machine Learning Guest Speaker: Prof. Sophia Shao (UC Berkeley) Submit your review before 1:00PM. Lecture slides: PDF, PPTX Guest Lecture slides: PDF Mixed precision training. [ICLR’18] Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks [SIGRAPH, 2016] Interstellar: Using Halide’s Scheduling Language to Analyze DNN Accelerators (formerly: DNN Dataflow Choice Is Overrated) Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration [Best Paper Award, DAC’21] A New Golden Age for Computer Architecture Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures [CACM’08]
4	2/14/22	Distributed deep learning, Part I: Systems Guest Speaker: Microsoft DeepSpeed Team Submit your review before 1:00PM. Lecture slides: PDF, PPTX Guest Lecture slides: PDF Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines [SC’21, Best Paper finalist] Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM [SC’21, Best Student Paper] ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning [SC’21] DeepSpeed: Advancing MoE inference and training to power next-generation AI scale [Blog post] Large Scale Distributed Deep Networks [NeurIPS’12] Gpipe: Efficient training of giant neural networks using pipeline parallelism [NeurIPS’19] PipeDream: Fast and Efficient Pipeline Parallel DNN Training [SOSP’19]
5	2/21/22	Holiday (Presidents Day)
6	2/28/22	Distributed deep learning, Part II: Scaling Constraints Guest Speaker: Michael Houston (Nvidia) Submit your review before 1:00PM. Lecture slides: PDF, PPTX Guest Lecture slides: PDF Measuring the Effects of Data Parallelism on Neural Network Training Scaling Laws for Neural Language Models [OpenAI, 2020] Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima [ICLR’16] Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers [ICML’20] Large Batch Optimization for Deep Learning: Training BERT in 76 minutes [ICLR’20] Scaling Vision Transformers
7	03/07/22	Project Proposals Lecture slides: PDF, PPTX
8	03/14/22	Machine learning Applied to Systems Guest Speaker: Prof. Tim Kraska (MIT) Submit your review before 1:00PM. Lecture slides: [PDF], [PPTX] The Case for Learned Index Structures [ICMD’18] Device Placement Optimization with Reinforcement Learning [ICML’17] Neural Adaptive Video Streaming with Pensieve [SIGCOMM’17]
9	03/21/22	Spring Break
10	03/28/22	Machine Learning Frameworks and Automatic Differentiation Guest Speaker: Prof. Tianqi Chen (OctoML and CMU) Submit your review before 1:00PM. Lecture slides: PDF, PPTX Joey’s Lecture slides: PDF, PPTX Automatic differentiation in ML: Where we are and where we should be going TensorFlow: A System for Large-Scale Machine Learning TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
11	04/04/22	Efficient Machine Learning Guest Speaker: Vikas Chandra (Facebook) Submit your review before 1:00PM. Lecture slides: PDF, PPTX Linear Mode Connectivity and the Lottery Ticket Hypothesis Integer-only Quantization of Neural Networks for Efficient Integer-Arithmetic-Only Inference FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search Hessian Aware trace-Weighted Quantization of Neural Networks The state of sparsity in deep neural networks Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
12	04/11/22	Fundamentals of Machine Learning in the Cloud, the Modern Data Stack Guest Speaker: Prof. Matei Zaharia (Databricks and Stanford) Submit your review before 1:00PM. Lecture slides: PDF The Sky Above The Clouds FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning RubberBand: cloud-based hyperparameter tuning
13	04/18/22	Benchmarking Machine Learning Workloads Guest Speaker: Prof. Vijay Reddi (Harvard) Submit your review before 1:00PM. Lecture slides: PDF, PPTX MLPerf Training Benchmark MLPerf Inference Benchmark Benchmark Analysis of Representative Deep Neural Network Architectures
14	04/25/22	Machine learning and Security Submit your review before 1:00PM. Lecture slides: PDF Communication-Efficient Learning of Deep Networks from Decentralized Data Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform Robust Physical-World Attacks on Deep Learning Models Helen: Maliciously Secure Coopetitive Learning for Linear Models Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference Rendered Insecure: GPU Side Channel Attacks are Practical The Algorithmic Foundations of Differential Privacy Federated Learning: Collaborative Machine Learning without Centralized Training Data Federated Learning at Google … A comic strip? SecureML: A System for Scalable Privacy-Preserving Machine Learning More reading coming soon …
15	05/02/22	RRR Week
16	05/09/22	Project Presentations

Projects

Detailed candidate project descriptions will be posted shortly. However, students are encourage to find projects that relate to their ongoing research.

Grading

Grades will be largely based on class participation and projects. In addition, we will require weekly paper summaries submitted before class.

• Projects: 60%
• Weekly Summaries: 20%
• Class Participation: 20%