A new version of this course is being offered in Fall 2019

AI-Sys Spring 2019

Course Description

The recent success of AI has been in large part due in part to advances in hardware and software systems. These systems have enabled training increasingly complex models on ever larger datasets. In the process, these systems have also simplified model development, enabling the rapid growth in the machine learning community. These new hardware and software systems include a new generation of GPUs and hardware accelerators (e.g., TPU and Nervana), open source frameworks such as Theano, TensorFlow, PyTorch, MXNet, Apache Spark, Clipper, Horovod, and Ray, and a myriad of systems deployed internally at companies just to name a few. At the same time, we are witnessing a flurry of ML/RL applications to improve hardware and system designs, job scheduling, program synthesis, and circuit layouts.

In this course, we will describe the latest trends in systems designs to better support the next generation of AI applications, and applications of AI to optimize the architecture and the performance of systems. The format of this course will be a mix of lectures, seminar-style discussions, and student presentations. Students will be responsible for paper readings, and completing a hands-on project. Readings will be selected from recent conference proceedings and journals. For projects, we will strongly encourage teams that contains both AI and systems students.

Course Syllabus

This is a tentative schedule. Specific readings are subject to change as new material is published.

Jump to Today

Week Date (Lec.) Topic
1
 1/23/19
( 1 )

Introduction and Course Overview

This lecture will be an overview of the class, requirements, and an introduction to what makes great AI-Systems research.
2
 1/28/19
( 2 )

Convolutional Neural Network Architectures

Minor Update: We have moved the reading on auto-encoders to Wednesday.

Reading notes for the two required readings below must be submitted using this google form by Monday the 28th at 9:30AM. We have asked that for each reading you answer the following questions:

  1. What is the problem that is being solved?
  2. What are the metrics of success?
  3. What are the key innovations over prior work?
  4. What are the key results?
  5. What are some of the limitations and how might this work be improved?
  6. How might this work have long term impact?

If you find some of the reading confusing and want a more gentle introduction, the optional reading contains some useful explanatory blog posts that may help.

Convolutional Networks
1/30/19
( 3 )

More Neural Network Architectures

Auto-Encoders

Graph Networks
3
 2/4/19
( 4 )

Deep Learning Frameworks
2/6/19
( 5 )

RL Systems & Algorithms
4
 2/11/19
( 6 )

Application: Data Structure and Algorithms
2/13/19
( 7 )

Distributed Systems for ML
5
 2/18/19
( 8 )

Administrative Holiday (Feb 18th)
2/20/19
( 9 )
  • Reading Quiz due before class. There was a mix-up in updating the reading and the wrong paper was swapped. You may either read the Hyperband paper (preferred) or the Vizer paper (see optional reading) for the second reading.
  • A Generalized Framework for Population Based Training [pdf]
6
 2/25/19
( 10 )

Auto ML & Neural Architecture Search (1/2)
2/27/19
( 11 )

Auto ML & Neural Architecture Search (2/2)
7
 3/4/19
( 12 )

Autonomous Vehicles

  • Reading Quiz due before class.
  • Autonomous Vehicles Overview [pdf, pptx]
  • Presentation: The Architectural Implications of Autonomous Driving[pdf]
3/6/19
( 13 )

Deep Learning Compilers
8
 3/11/19
( 14 )

Project Presentation Checkpoints
3/13/19
( 15 )

Application: Program synthesis
9
 3/18/19
( 16 )

Distributed Deep Learning (Part 1)
3/20/19
( 17 )

Distributed Deep Learning (Part 2)
10
 3/25/19
( 18 )

Spring Break (March 25th)
3/27/19
( 19 )

Spring Break (March 27th)
11
 4/1/19
( 20 )

Application: Networking
4/3/19
( 21 )

Dynamic Neural Networks
12
 4/8/19
( 22 )

Model Compression
4/10/19
( 23 )

Applications: Security
13
 4/15/19
( 24 )

Application: Prediction Serving
4/17/19
( 25 )

Natural Language Processing Systems
14
 4/22/19
( 26 )

Explanability & Interpretability
4/24/19
( 27 )

Scheduling for DL Workloads

  • Reading Quiz due before class.
  • DL Scheduling slides [pdf]
  • Dominant Resource Fairness (DRF) slides [pdf]
15
 4/29/19
( 28 )

Cortical Learning and Stoica Course Summary
5/1/19
( 29 )

Neural Modular Networks and Gonzalez Course Summary

  • Reading Quiz due before class.
  • Neural Modular Networks Slides [pdf, pptx]
  • Gonzalez Course Summary (Reflections on the Field of AI-Systems) [pdf, pptx]
16
 5/6/19
( 30 )

RRR Week (May 6th)
5/8/19
( 31 )

Poster Session from 9:00 to 11:00
17
 5/13/19
( 32 )

Final Reports Due

  • Due at 11:59 PM
  • Format: 8 pages (Google Doc)
  • Email link to jegonzal@berkeley.edu and istoica@berkeley.edu

Projects

Detailed candidate project descriptions will be posted shortly. However, students are encourage to find projects that relate to their ongoing research.

Grading

Grades will be largely based on class participation and projects. In addition, we will require weekly paper summaries submitted before class.