CS 4644 / 7643 Deep Learning

Spring 2023, Wed/Fri 12:30pm - 1:45 pm, Clough UG Learning Commons 152

Course Information

This is an exciting time to be studying (Deep) Machine Learning, or Representation Learning, or for lack of a better term, simply Deep Learning!

Deep Learning is rapidly emerging as one of the most successful and widely applicable set of techniques across a range of domains (vision, language, speech, reasoning, robotics, AI in general), leading to some pretty significant commercial success and exciting new directions that may previously have seemed out of reach.

This course will introduce students to the basics of Neural Networks (NNs) and expose them to some cutting-edge research. It is structured in modules (background, Convolutional NNs, Recurrent NNs, Deep Reinforcement Learning, Deep Structured Prediction). Modules will be presented via instructor lectures and reinforced with homeworks that teach theoretical and practical aspects. The course will also include a project which will allow students to explore an area of Deep Learning that interests them in more depth.

Instructor

Zsolt Kira

Teaching Assistants

Ting-Yu Lan

Anshul Ahluwalia

Ahmed Shaikh

Aaditya Singh

Yash Jain

Yash Jakhotiya

Hoon Lee

Zach Minot

Sumedh Vijay

Ningyuan Yang

Jan Vijay Singh

Aaditya Singh

Class Info & Links

Lectures: Wed/Fri 12:30 pm - 1:45 pm, Clough UG Learning Commons 152

Zoom: https://gatech.zoom.us/j/91625047484

Piazza: https://piazza.com/gatech/spring2023/cs46447643/home (code DLSPR2023, available via Canvas)

Canvas

CS4644: https://gatech.instructure.com/courses/294004

CS7643: https://gatech.instructure.com/courses/293996

Gradescope

CS4644: https://www.gradescope.com/courses/484319

CS7643: https://www.gradescope.com/courses/484320

</dl> </div>

Tentative Schedule (subject to changes)

Date	Topic	Optional Reading
W1: Jan 11	Intro lecture + class logistics. Slides (PDF) PS0/HW0 is due 11:59pm 01/15 (NO grace period). IMPORTANT: All students MUST complete PS0! This is true even if you are currently on the waitlist and likely to get in! PS0 is on Gradescope, HW0 is here. See FAQ and qualtrics survey below for gradescope access if you are not registered.	LeCun et al., Nature '15 Shannon, 1956 DL book: Linear Algebra background DL book: Probability background DL book: ML Background
W1: Jan 13	Machine learning intro, applications (CV, NLP, etc.), parametric models and their components Slides (PDF) PS0/HW0 due 01/15 11:59pm EST. HW1/PS1 released
W2: Jan 18	Supervised Learning, Linear Classification, Loss functions, Gradient Descent Slides (PDF) Supervised Learning notes,	DL book: Linear Algebra background DL book: Probability background DL book: ML Background
W2: Jan 20	Gradient Descent, Linear Algebra View Slides (PDF)
W3: Jan 25	Neural Networks, Backpropagation, Linear Algebra View Slides (PDF) Gradients notes MLP/ReLU Jacobian notes	DL book: Deep Feedforward Nets Matrix calculus for deep learning
W3: Jan 27	Backpropagation, Jacobians, Autodiff Slides (PDF) Forward/Reverse Autodiff Examples	Automatic Differentiation Survey, Baydin et al.
W4: Feb 1	Automatic Differentiation recap, activation functions, optimization Slides (PDF)
W4: Feb 3	Optimization, Regularization, Data Augmentation Slides (PDF) PS1/HW1 due Feb 3rd 11:59pm (grace period until Feb 5th 11:59pm), PS/HW2 out	DL book: Optimization for Training Deep Models DL book: Regularization for DL General preprocessing: Preprocessing for deep learning: from covariance matrix to image whitening General preprocessing: cs231n on preprocessing Optional: Khetarpal, Khimya, et al. Re-evaluate: Reproducibility in evaluating reinforcement learning algorithms." (2018). See related blog post
W5: Feb 8	Convolutions, Convolution Gradients Slides (PDF) CNNs notes, CNNs Backprop notes.	DL book: Convolutional Networks
W5: Feb 10	Convolution Gradients Slides (PDF) Note: From last year so ignore any administrative info
W6: Feb 15	Pooling, Convolutional Neural Networks (CNNs) Slides (PDF)
W6: Feb 17	Visualizing Neural Networks Slides (PDF)	Understanding Neural Networks Through Deep Visualization Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
W7: Feb 22	Visualizing Cont., Advanced CNN Architectures Slides (PDF) PS/HW2 due Feb 24th 11:59pm (Grace period Feb 26th), PS/HW3 out
W7: Feb 24	Advanced CNN architectures. Slides (PDF)	Fully Convolutional Networks for Semantic Segmentation
W8: Mar 1	Intro to Structured Neural Representations, Recurrent Neural Networks Slides (PDF)	DL Book: Sequential Modeling and Recurrent Neural Networks (RNNs)
W8: Mar 3	Recurrent Neural Networks, Long Short-Term Memory Slides (PDF)
W9: Mar 8	Project Planning Session
W9: Mar 10	Guest Lecture: Attentions and Transformers Slides (PDF)	Attention is all you need BERT Paper The Illustrated Transformer
W10: Mar 15	Guest Lecture: Michael Auli (Meta) Slides (PDF) PS/HW3 due Mar 16th 11:59pm (Grace period Mar 18th), PS/HW3 out
W10: Mar 17	Meta Guest Lecture: Armen Aghajanyan - Large Language Models, Scaling Laws, and the Future of AI Research Project Proposal Due Mar 17th 11:59pm
W11: Mar 22	Spring Break
W11: Mar 24	Spring Break
W12: Mar 29	RL background. Slides (PDF)	Sutton & Bartow Chapter 1 Survey paper on Deep RL MDP Notes (courtesy Byron Boots)
W12: Mar 31	RL: RL Part 2 - Q-Learning, DQN, Policy Gradient. Slides (PDF)	Notes on Q-learning (courtesy Byron Boots)
W13: Apr 5	RL: Policy Gradients, REINFORCE, Actor-Critic. Slides (PDF)	Policy iteration notes (courtesy Byron Boots) Policy gradient notes (courtesy Byron Boots)
W13: Apr 7	Guest Lecture: Zubair Irshad - Learning Object-Centric Neural 3D Scene Representations Slides (PDF) PS/HW4 due Apr 8th 11:59pm (grace period Apr 10th)
W14: Apr 12	Guest Lecture: Ishan Misra (Meta) - Self-Supervised Vision Learning Slides (PDF)
W14: Apr 14	Variational Autoencoders Slides (PDF)	Tutorial on Variational Autoencoders
W15: Apr 19	Diffusion Models Slides (PDF)
W15: Apr 21	Emerging trends, wrap-up. Slides (PDF) Final Project Due May 1st 11:59pm Note: We will encourage projects to be posted to piazza

Grading

• 72% Homework (4 homeworks)
• 28% Final Project
• 1% (potential bonus) Class Participation: top endorsed answers/questions/comments on Piazza

Late policy for deliverables

• There will be no make-up work provided for missed assignments. Of course, emergencies (illness, family emergencies) will happen. In those instances, please contact the Dean of Students office (see here for rules). The Dean of Students is equipped to verify emergencies and pass confirmation on to all your classes. For consistency, we ask all students to do this in the event of an emergency. Do not send any personal/medical information to the instructor or TAs; all such information should go through the Dean of Students.
• Every homework deliverable and project deliverable will have a 48-hour grace period during which no penalty will apply. This is intended to allow you time to verify that your submission has been submitted (we recommend you re-download it and look it over to make sure all questions/deliverables have been answered). Canvas will show your submission as late, but you do not have to ask for this grace period. Deliverables after the grace period will receive a grade of 0.

Prerequisites

CS 4644/7643 should NOT be your first exposure to machine learning. Ideally, you need:

• Intro-level Machine Learning
  • CS 3600 for the undergraduate section and CS 7641/ISYE 6740/CSE 6740 or equivalent for the graduate section.
• Algorithms
  • Dynamic programming, basic data structures, complexity (NP-hardness)
• Calculus and Linear Algebra
  • positive semi-definiteness, multivariate derivates (be prepared for lots and lots of gradients!)
• Programming
  • This is a demanding class in terms of programming skills.
  • HWs will involve Python and PyTorch.
  • Your library of choice for project.
• Ability to deal with abstract mathematical concepts

Online Student Conduct and (N)etiquette

Communicating appropriately in the online classroom can be challenging. All communication, whether by email, Piazza, Canvas, or otherwise, must be professional and respectful. In order to minimize this challenge, it is important to remember several points of “internet etiquette” that will smooth communication for both students and instructors

1. Read first, Write later. Read the ENTIRE set of posts/comments on a discussion board before posting your reply, in order to prevent repeating commentary or asking questions that have already been answered.
2. Avoid language that may come across as strong or offensive. Language can be easily misinterpreted in written electronic communication. Review email and discussion board posts BEFORE submitting. Humor and sarcasm may be easily misinterpreted by your reader(s). Try to be as matter of fact and as professional as possible.
3. Follow the language rules of the Internet. Do not write using all capital letters, because it will appear as shouting. Also, the use of emoticons can be helpful when used to convey nonverbal feelings. ☺
4. Consider the privacy of others. Ask permission prior to giving out a classmate’s email address or other information.
5. Keep attachments small. If it is necessary to send pictures, change the size to an acceptable 250kb or less (one free, web-based tool to try is picresize.com).
6. No inappropriate material. Do not forward virus warnings, chain letters, jokes, etc. to classmates or instructors. The sharing of pornographic material is forbidden.

NOTE: The instructor reserves the right to remove posts that are not collegial in nature and/or do not meet the Online Student Conduct and Etiquette guidelines listed above.

Plagiarism & Academic Integrity

Georgia Tech aims to cultivate a community based on trust, academic integrity, and honor. Students are expected to act according to the highest ethical standards. All students enrolled at Georgia Tech, and all its campuses, are to perform their academic work according to standards set by faculty members, departments, schools and colleges of the university; and cheating and plagiarism constitute fraudulent misrepresentation for which no credit can be given and for which appropriate sanctions are warranted and will be applied. For information on Georgia Tech’s Academic Honor Code, please visit http://www.catalog.gatech.edu/policies/honor-code/ or http://www.catalog.gatech.edu/rules/18/.

You are encouraged to discuss problems and papers with others as long as this does not involve the copying of code or solutions. After discussions, all materials that are part of a submission should be wholly your own. Do NOT search for code directly implementing the assignment and submit snippets or variations of them. You can search for conceptual information but NOT code solutions. Any public material that you use (open-source software, help from a textbook, or substantial help from a friend, etc.) should be acknowledged explicitly in anything you submit to us. Our current policy is that you CANNOT use any AI-assisted (e.g. based on large language models such as Github Copilot) text or coding responses in part or whole; again, every piece of text or code you submit must be wholly your own.

If you have any doubts about whether something is legal or not, please do check with the class Instructor or the TA. We will actively check for cheating, and any act of dishonesty will result in a Fail grade. Any student suspected of cheating or plagiarizing on any deliverable including assignments will be reported to the Office of Student Integrity, who will investigate the incident and identify the appropriate penalty for violations.

Students with Disabilities

If you are a student with learning needs that require special accommodation, contact the Office of Disability Services at 404.894.2563 or http://disabilityservices.gatech.edu/, as soon as possible, to make an appointment to discuss your special needs and to obtain an accommodations letter. Please also e-mail me as soon as possible in order to set up a time to discuss your learning needs.

Subject to Change Statement

The syllabus and course schedule may be subject to change. Changes will be communicated via the Canvas announcement tool. It is the responsibility of students to check Piazza, email messages, and course announcements to stay current in their online courses.

Campus Resources

Community Resources

Project Details (28% of course grade)

The class project is meant for students to (1) gain experience implementing deep models and (2) try Deep Learning on problems that interest them. The amount of effort should be at least the level of 1.5 homework assignment per group member (2-4 people per group). The deliverables are

- Project Proposal (2%): Due Mar 17
- Final Report (26%): Due May 1st

The final report is a PDF write-up describing the project in a self-contained manner will be the sole deliverable. Your final write-up is required to be between 4 - 6 pages using a template provided with structured like a paper from a computer vision conference (CVPR, ECCV, ICCV, etc.). Please use this template so we can fairly judge all student projects without worrying about altered font sizes, margins, etc. After the class, we will post all the final reports online so that you can read about each others’ work. Additionally, we will allow people to upload additional code, videos, and other supplementary material as zip file similar to code upload for assignments. While the PDF may link to supplementary material, external documents, and code, such resources may or may not be used to evaluate the project. The final PDF should completely address all of the points in the rubric described below.

Rubric

We will release a detailed project rubric format as the semester progresses.

FAQs

• The class is full. Can I still get in?

  Sorry. The course admins in CoC control this process. Please talk to them.

• Unregistered Students who intend to register:

  If you are not registered for this course, you will not have access to Gradescope for submission of PS0. Please fill the following form in order to be added to Gradescope and be able to submit PS0: https://gatech.co1.qualtrics.com/jfe/form/SV_3ggz8OrRB7wXqIu

  Students who individually emailed us and have not been added yet - you may have left out the details of which course instance you are planning to take (either CS 7643 or CS 4644). There are two separate Gradescope courses for the two instances. Please fill the above form in order to provide us with this information.

• Registered students who are not able to access Gradescope:

  This will happen if you were registered to the course very recently. Gradescope rosters are synced periodically and it may take some time for you to receive a Gradescope sign-up notification. If you still face problems with accessing Gradescope, please email us.

• I am graduating this Spring and I need this class to complete my degree requirements. What should I do?

  Talk to the advisor or graduate coordinator for your academic program. They are keeping track of your degree requirements and will work with you if you need a specific course.

• Can I audit this class or take it pass/fail?

  No. Due to the large demand for this class, we will not be allowing audits or pass/fail. Letter grades only. This is to make sure students who want to take the class for credit can.

• I have a question. What is the best way to reach the course staff?

  Registered students – your first point of contact is Piazza (so that other students may benefit from your questions and our answers). If you have a personal matter, create a private piazza post.

Related Classes / Online Resources

• CS231n Convolutional Neural Networks for Visual Recognition, Stanford
• Machine Learning, Oxford
• Deep Learning, New York University
• Deep Learning, CMU
• Deep Learning, University of Maryland
• Hugo Larochelle’s Neural Networks class

Book

• Deep Learning, Ian Goodfellow, Aaron Courville, and Yoshua Bengio, MIT Press

Overviews

• Deep Learning, Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, Nature
• Representation Learning: A Review and New Perspectives, Yoshua Bengio, Aaron Courville, and Pascal Vincent

Note to people outside Georgia Tech

Feel free to use the slides and materials available online here. If you use our slides, an appropriate attribution is requested. Please email the instructor with any corrections or improvements.