CS 4803 / 7643 Deep Learning
Spring 2020, TR 1:30  2:45 pm, Clough Commons 152
Course Information
This is an exciting time to be studying (Deep) Machine Learning, or Representation Learning, or for lack of a better term, simply Deep Learning!
Deep Learning is rapidly emerging as one of the most successful and widely applicable set of techniques across a range of domains (vision, language, speech, reasoning, robotics, AI in general), leading to some pretty significant commercial success and exciting new directions that may previously have seemed out of reach.
This course will introduce students to the basics of Neural Networks (NNs) and expose them to some cuttingedge research. It is structured in modules (background, Convolutional NNs, Recurrent NNs, Deep Reinforcement Learning, Deep Structured Prediction). Modules will be presented via instructor lectures and reinforced with homeworks that teach theoretical and practical aspects. The course will also include a project which will allow students to explore an area of Deep Learning that interests them in more depth.
Instructor:
Zsolt Kira: Mondays 10:00am  11:00am  C1106 Briarcliff
TAs: Yihao Chen: Wednesdays 11:00am – 12:00pm  C0908 Home Park
 Sameer Dharur: Tuesdays 11:30am  12:30pm  C1303 Glenwood
 Rahul Duggal: Thursdays 11:30am  12:30pm  C0908 Home Park
 Patrick Grady: Mondays 11:30am  12:30pm  C0908 Home Park
 Harish Kamath: Thursdays 3:00pm  4:00pm  C1303 Glenwood
 Anishi Mehta: Wednesdays 2:30pm  3:30pm  C1303 Glenwood
 Manas Sahni: Mondays 3pm  4pm  C1303 Glenwood
 Yinquan Lu: Thursdays 4pm  5pm  C1303 Glenwood
 Jiachen Yang: Tuesdays 4pm  5pm  C1303 Glenwood
 Zhuoran Yu: Fridays 11:30am  12:30pm  C1303 Glenwood
Schedule
Mar 16: HW4 outDate  Topic  Optional Reading 
W1: Jan 7 
Class Administrativia.
PS0 out (w/ LaTeX template) Slides (pdf). 

W1: Jan 9 
Image Classification and kNN.
Slides (pdf), Supervised Learning notes, kNN notes. 

W2: Jan 14 
Linear Classifiers, Loss Functions.
Slides (pdf) 

W2: Jan 16 
Regularization, Neural Networks.
PS/HW1 out Slides (pdf), Manual differentiation derivation. 

W3: Jan 21  Optimization, Backprop.
Slides (pdf) Gradients notes. 

W3: Jan 23 
Backprop continued. Computational Flow Graphs. Matrices, vectors, tensors, Jacobians.
Slides (pdf), 

W4: Jan 28  Forward/backward autodiff, Convolutional Neural Networks (CNNs) part 1: Convolutions, FC vs Conv Layers.
Slides (pdf) Forward/Reverse Autodiff Examples CNNs notes. 

W4: Jan 30  CNNs part 2. Slides (pdf) 

W5: Feb 4  CNNs part 3 Slides (pdf) CNNs Backprop notes. 

W5: Feb 6 
CNNS part 4, CNN Architectures Slides (pdf) 

W6: Feb 11  FB Lecture (Kristen M. Altenburger and Sam Pepose): Data Wrangling Slides (pdf) PS/HW1 due, PS/HW2 out 

W6: Feb 13  CNNs and Visualization. Slides (pdf) 

W7: Feb 18  FB Lecture (Ledell Wu): Embeddings and World2Vec.
Slides (pdf) 
Optional: 
W7: Feb 20  Visualization (Cont.)
Slides (pdf) 

W8: Feb 25  Training neural networks
Slides (pdf) Feb 26th: HW2 due, Feb 27th: HW3 out 

W8: Feb 27  Recurrent Neural Networks (RNNs) part 1.
Slides (pdf) RNNs notes. 

W9: Mar 3  RNNs 2: RNNs, LSTMs.
Slides (pdf) 

W9: Mar 5  FB Lecture: RNNs 3: SelfAttention, Transformers.
Slides (pdf) (slides from last semester) 

W10: Mar 10  RL background.
Slides (pdf). 
MDP Notes (courtesy Byron Boots). 
W10: Mar 12  RL Continued and Deep RL
Slides (pdf). Project Proposals Due! 
Notes on Qlearning (courtesy Byron Boots).

W11: Mar 17  Spring break


W11: Mar 19  Spring break


W12: Mar 24  Test Week: Deep RL continued (recorded)
Slides (pdf). 
Policy iteration notes (courtesy Byron Boots). Policy gradient notes (courtesy Byron Boots). 
W12: Mar 26  Test Week: Deep RL Live Session (Friday 03/27 10am)


W13: Mar 31  FB Lecture: Language modeling, translation, etc.
Slides (pdf). Mar 31: HW3 due 

W13: Apr 2  Generative models Part 1
Slides (pdf). Apr 3: HW4 due (optional) 

W14: Apr 7  Generative models Part 2. Slides (pdf). 

W14: Apr 9  FB Lecture: Largescale systems
pytorch scaling and visual cortex (pdf). 

W15: Apr 14  Guest Lecture (Argo AI): Visual embeddings best practices
See piazza post/canvas for presentation. Project Checkin Due April 15th! 

W15: Apr 16  FB Lecture: Fairness, privacy, ethics
Slides 

W16: Apr 21  (poster presentation cancelled)
Unsupervised, selfsupervised, semisupervised, fewshot learning Slides (pdf). Final project due: April 30th 
Grading
 2% PS0
 78% Homework (4 homeworks)
 20% Final Project
 Bonus participation points for top Piazza posters (up to 3%)
Late policy for deliverables
 No penalties for medical reasons or emergencies. Please see GT Catalog for rules about contacting the office of the Dean of Students. NOTE: Please do not send us any sensitive medical information! Send all such proof/materials to the Dean of Students.
 Every student has 7 free late days (7 x 24hour chunks) for this course.
 After all free late days are used up, penalty is 25% for each additional late day.
Prerequisites
CS 4803/7643 should not be your first exposure to machine learning. Ideally, you need:
 Introlevel Machine Learning
 CS 3600 for the undergraduate section and CS 7641/ISYE 6740/CSE 6740 or equivalent for the graduate section.
 Algorithms
 Dynamic programming, basic data structures, complexity (NPhardness)
 Calculus and Linear Algebra
 positive semidefiniteness, multivariate derivates (be prepared for lots and lots of gradients!)
 Programming
 This is a demanding class in terms of programming skills.
 HWs will involve a mix of languages (Python, C++) and libraries (PyTorch).
 Your language of choice for project.
 Ability to deal with abstract mathematical concepts
Project Details (20% of course grade)
The class project is meant for students to (1) gain experience implementing deep models and (2) try Deep Learning on problems that interest them. The amount of effort should be at the level of one homework assignment per group member (24 people per group). A PDF writeup describing the project in a selfcontained manner will be the sole deliverable. Your final writeup is required to be between 4  6 pages (template will be released), structured like a paper from a computer vision conference (CVPR, ECCV, ICCV, etc.). Please use this template so we can fairly judge all student projects without worrying about altered font sizes, margins, etc. After the class, we will post all the final reports online so that you can read about each others’ work. Additionally, we will allow people to upload additional code, videos and other supplementary material as zip file similar to code upload for assignments. While the PDF may link to supplementary material, external documents and code, such resources may or may not be used to evaluate the project. The final PDF should completely address all of the points in the rubrik described below.
Potential Project Ideas
Final Report Template
Rubrik (60 points)
We are not looking to see if you succeeded or failed at accomplishing what you set out to do. It’s ok if your results are not “good”. What matters is that you put in a reasonable effort, understand the project and how it related to Deep Learning in detail, and are able to clearly communicate that understanding. Note that you must justify your design, implementation, and experimentation decisions using your knowledge and data. You should make claims about why you think the results turned out the way they did, and perform specific experimentation or gather relevant data to justify it.
A former DARPA director named George H. Heilmeier came up with a list of questions for evaluating research projects. We’ve adapted that list for our rubrik. Introduction / Background / Motivation:
 (5 points) What did you try to do? What problem did you try to solve? Articulate your objectives using absolutely no jargon.
 (5 points) How is it done today, and what are the limits of current practice?
 (5 points) Who cares? If you are successful, what difference will it make? Data
 (5 points) What data did you use? Provide details about your data, specifically choose the most important aspects of your data mentioned here: https://arxiv.org/abs/1803.09010. You don’t have to choose all of them, just the most relevant.
Approach:
 (10 points) What did you do exactly? How did you solve the problem? Why did you think it would be successful? Is anything new in your approach?
 (5 points) What problems did you anticipate? What problems did you encounter? Did the very first thing you tried work? Experiments and Results:
 (10 points) How did you measure success? What experiments were used? What were the results, both quantitative and qualitative? Did you succeed? Did you fail? Why? Justify your reasons with arguments supported by evidence and data.
In addition, 15 more points will be distributed based on:
 (5 points) Appropriate use of figures / tables / visualizations. Are the ideas presented with appropriate illustration? Are the results presented clearly; are the important differences illustrated?
 (5 points) Overall clarity. Is the manuscript selfcontained? Can a peer who has also taken Deep Learning understand all of the points addressed above? Is sufficient detail provided?
 (5 points) Finally, points will be distributed based on your understanding of how your project relates to Deep Learning. Here are some questions to think about:
 What was the structure of your problem? How did the structure of your model reflect the structure of your problem?
 What parts of your model had learned parameters (e.g., convolution layers) and what parts did not (e.g., postprocessing classifier probabilities into decisions)?
 What representations of input and output did the neural network expect? How was the data pre/postprocessed?
 What was the loss function?
 Did the model overfit? How well did the approach generalize?
 What hyperparameters did the model have? How were they chosen? How did they affect performance? What optimizer was used?
 What Deep Learning framework did you use?
 What existing code or models did you start with and what did those starting points provide?
 At least some of these questions and others should be relevant to your project and should be addressed in the PDF. You do not need to address all of them in full detail. Some may be irrelevant to your project and others may be standard and thus require only a brief mention. For example, it is sufficient to simply mention the crossentropy loss was used and not provide a full description of what that is. Generally, provide enough detail that someone with an appropriate background (in both Deep Learning and your domain of choice) could replicate the main parts of your project somewhat accurately, probably missing a few less important details.
Submit the final report by uploading to “Final Project” assignment on Gradescope. There will be a group assignment corresponding to the project submission. Every group should submit the report once and all group member names should be listed through the gradescope interface. Instructions coming soon. The supplementary material should also be uploaded as zip file to the Final Project (Supplementary Material) assignment.
FAQs

The class is full. Can I still get in?
Sorry. The course admins in CoC control this process. Please talk to them.

Unregistered Students who intend to register:
If you are not registered for this course, you will not have access to gradescope for submission of PS0. Please email the course staff address for access (cs48037643staff@lists.gatech.edu).
Students who individually emailed us and have not been added yet  you may have left out the details of which course instance you are planning to take (either CS 7643 or CS 4803). Please fill the above form in order to provide us with this information.

Registered students who are not able to access gradescope:
This will happen if you were registered to the course very recently. Gradescope rosters are synced periodically and it may take some time for you to receive a gradescope signup notification. If you still face problems with accessing gradescope, please post a comment below.

I am graduating this semester and I need this class to complete my degree requirements. What should I do?
Talk to the advisor or graduate coordinator for your academic program. They are keeping track of your degree requirements and will work with you if you need a specific course.

Can I audit this class or take it pass/fail?
No. Due to the large demand for this class, we will not be allowing audits or pass/fail. Letter grades only. This is to make sure students who want to take the class for credit can.

Can I simply sit in the class (no credits)?
In general, we welcome members of the Georgia Tech community (students, staff, and/or faculty) to sitin. Out of courtesy, we would appreciate if you let us know beforehand (via email or in person). If the classroom is full, we would ask that you please allow registered students to attend.

I have a question. What is the best way to reach the course staff?
Registered students – your first point of contact is Piazza (so that other students may benefit from your questions and our answers). If you have a personal matter, email us at the class mailing list cs48037643staff@lists.gatech.edu
Related Classes / Online Resources
 CS231n Convolutional Neural Networks for Visual Recognition, Stanford
 Machine Learning, Oxford
 Deep Learning, New York University
 Deep Learning, CMU
 Deep Learning, University of Maryland
 Hugo Larochelle’s Neural Networks class
Book
Overviews
Note to people outside Georgia Tech
Feel free to use the slides and materials available online here. If you use our slides, an appropriate attribution is requested. Please email the instructor with any corrections or improvements.