Table of Contents

Course Topics and Calendar

Below is a list of the topics I am planning to cover in this course. Since course topics are among the most often requested information about this course, I am placing this on top of this website. More information about this course can be found in the sections that follow the course content below.

Part 1: Introduction

  • L01: Course overview, introduction to deep learning
  • L02: The brief history of deep learning
  • L03: Single-layer neural networks: The perceptron algorithm

Part 2: Mathematical and computational foundations

  • L04: Linear algebra and calculus for deep learning
  • L05: Parameter optimization with gradient descent
  • L06: Automatic differentiation with PyTorch
  • L07: Cluster and cloud computing resources

Part 3: Introduction to neural networks

  • L08: Multinomial logistic regression
  • L09: Multilayer perceptrons and backpropration
  • L10: Regularization to avoid overfitting
  • L11: Input normalization and weight initialization
  • L12: Learning rates and advanced optimization algorithms
  • Project proposal (student submission)

Part 4: Deep learning for computer vision and language modeling

  • L13: Introduction to convolutional neural networks
  • L14: Convolutional neural networks architectures
  • L15: Introduction to recurrent neural networks

Part 5: Deep generative models

  • L16: Autoencoders
  • L17: Variational autoencoders
  • L18: Introduction to generative adversarial networks
  • L19: Evaluating generative adversarial networks
  • L20: Recurrent neural networks for seq-to-seq modeling
  • L21: Self-attention and transformer networks

Course Description

Credits: 3

Course Description:

Deep learning is a field that specializes in discovering and extracting intricate structures in large, unstructured datasets for parameterizing artificial neural networks with many layers. Since deep learning has pushed the state-of-the-art in many research and application areas, it’s become indispensable for modern technology.

The focus of this course will be on understanding deep, artificial neural networks by connecting it to related concepts in statistics, such as generalized linear models and maximum likelihood estimation. Beyond covering deep learning models for predictive modeling, the latter portion of this course will focus on deep generative models and models based on stochastic variational inference, which allows for learning directed probabilistic models.

Besides covering and explaining deep learning and generative models on a mathematical and conceptual level, this course emphasizes the practical aspects of deep learning. Open-source libraries from the Python open-source ecosystem for scientific computing will be used to provide students with hands-on experience for implementing deep neural nets, working on supervised learning tasks, and applying generative models for dataset synthesis. Regarding the class project, students will form teams of three and collaboratively work on a project proposal to outline the planned scope of the project and meet with the lecturer for further discussion and feedback. After receiving feedback on the proposal, students will work independently towards the final project report, which will be submitted in the form of a conference paper for peer-review by other students and the lecturer. Finally, the students will give an 8-10-minute talk at the end of the semester to formally present their projects in class.

Learning Outcomes:

  • Developing an advanced understanding of deep learning and generative models, which represent state-of-the-art approaches for predictive modeling in today’s data-driven world.
  • Identifying scenarios where it makes sense to deep learning for real-world problem-solving.
  • Building a repertoire of different algorithms and approaches to deep learning and understanding their various strengths and weaknesses.
  • Learning how to use the Python programming language and Python’s scientific computing stack for implementing deep learning algorithms to 1) enhance the learning experience, 2) conduct research and be able to develop nzvel algorithms, and 3) apply deep learning to problem-solving in various fields and application areas.
  • Combining both the theoretical and practical concepts taught in this class to creative, real-world problem solving and communicating the outcome professionally in form of a scientific paper and a formal oral presentation.

Course Rerequisites:

Along with introducing of the concepts of deep learning and generative models, the in-class lectures will provide a refresher on relevant concepts from calculus and linear algebra; however, a calculus background (e.g., Math 221) and a linear algebra background (e.g., Math 340) is recommended. While this course will also provide an introduction to the basics of the Python programming language for machine learning, it is highly recommended that students are familiar with basic programming and have completed an introductory programming class.

The official requisites are

  • MATH 320, 321, 340, 341, graduate/professional standing, or member of the Statistics Visiting International Scholars program

Course Audience:

Students majoring in math or statistics or those wishing to take additional statistics courses.

Course Information, Resources, and Communication

During this online semester, we will mainly be using Canvas for this course so that everything is conveniently organized in one place. This includes the course material, announcements, homework submissions, exam, and discussions. I highly recommend using Firefox or Chrome for using Canvas because it seems some features are not well supported in Safari, yet.

  • Questions: Outside the virtual office hours, I set up a [Piazza forum](https://canvas.wisc.edu/courses/244897/external_tools/65 for the course (you can access it through a link on Canvas or the direct link here). This is most efficient in case multiple students have the same or similar questions. Students are also encouraged to help other students on Piazza.

For personal questions (course accommodations due to illness, missed assignments, etc.), please choose the option “This private post is only visible to Instructors” before posting.

Course Logistics

When

Lecture material will usually be posted every Tuesday and Thursday morning.

Where

Online.

Instructors

  • Instructor: Sebastian Raschka
  • Teaching Assistant: Zhongjie Yu

Office Hours

Prof. Sebastian Raschka (Instructor)

  • To accommodate for most common time zones, my office hours will be in the evening this semester, Thursdays from 6 - 7 pm CT (Madison time). Zoom links will be shared on Canvas.

Zhongjie Yu (Teaching Assistant)

  • Tuesday 7:45 pm - 8:45 pm. Zoom links will be shared on Canvas.

Overall Format and Participation

  • By the end of each week, yous should have watched all lecture contents shared (Tuesdays and Fridays) this week.
  • A short self-assessment quiz will be posted at the end of each lecture week (Fridays) asking conceptual questions about the lecture’s contents. It quiz will be due on the Friday of the following week.
  • Hands-on homework assignments involving code will be posted approximately every two to 3 weeks.
  • Starting after the first few weeks of the semester, students will form teams of three to work and collaborate on an individual class project throughout the semester. Students are expected to meet (virtually) on a regular and weekly basis to make progress towards their project goals.

Resources and Useful Material

Python Material

Regarding the use of Python in this course, we will mainly focus on two libraries: NumPy and PyTorch*. You can think of NumPy as a linear algebra library that provides utilities similar to MatLab (if you are familiar with MatLab). It’s a library that is used in almost any scientific computing task and other libraries in Python and is generally useful. PyTorch is the main deep learning library we will be using.

In any case, you don’t need to be an expert Python programmer to follow the content in this class, (and I will teach you about PyTorch in this course, so no worries about learning it beforehand). However, some basic familiarity with Python will be necessary in order to use these libraries.

Below are two beginner-friendly resources that I think will be useful to learning the Python basics:

Python for Beginners (A collection of short video Lectures)

A great video series by educators at Microsoft, which was recently made available for free on YouTube: Python for Beginners.

Python Like You Mean It

A more comprehensive free introduction for getting started with Python and its main scientific computing libraries: https://www.pythonlikeyoumeanit.com.

NumPy Resources

We won’t be using NumPy (a linear algebra and array library) extensively in this course. However, the NumPy API is very similar PyTorch’s tensor API, so understanding NumPy will be very useful for understanding and working with PyTorch. Below is tutorial that I wrote based on my lecture notes. I also added videos from last semester’s Stat 451: Machine Learning:

Scientific Computing in Python: Introduction to NumPy and Matplotlib – Including Video Tutorials.

Deep Learning Books

Deep learning is a new and very fast moving field, and many of the knowledge is contained in freely available research articles and other articles shared freely on the internet. As of today, there is also no nice textbook available that would suit the structure of this course.

Thus, I will link free resources, including internet articles and research articles that are relevant for the course. The book suggestions are recommendations but not requirements. I will not use any chapters directly for this course, but you can use them as a personal reference.

Deep Learning with PyTorch

The “Deep Learning with PyTorch” offers a nice overview for beginners. I have written a more detailed review here.

A link direct link to the free PDF version is available at: https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf. A print version can be obtained from https://www.manning.com/books/deep-learning-with-pytorch.

Deep Learning – Goodfellow, Bengio, Courville

The “Deep Learning” book by Goodfellow et al. is nice for deeper theoretical coverage of deep learning. The book is also officially and freely available as web version at http://www.deeplearningbook.org

Python Machine Learning, 3rd Edition – Raschka & Mirjalili

My Python Machine Learning book provides intro to general machine learning that could be useful as background information on the general concepts. The deep learning chapters are in TensorFlow though, and we will be using PyTorch in this class. However, the explanations of convolutional networks, recurrent neural networks, etc. are still useful, and I will be using several figures from this book.

Grading

The final grade will be computed using the following weighted grading scheme:

  • 30% Problem Sets (Homeworks and quizzes)
  • 20% Midterm Exam
  • 50% Class Project:
    • 5% Project proposal
    • 20% Project presentation
    • 25% Project report

The final letter grade will be based on the percent of the total points accumulated in the course. The proposed grade cut-offs are as follows:

  • A: >= 93%
  • AB: >= 90%
  • B: >= 85%
  • BC: >= 80%
  • C: >= 70%
  • D: >= 50%
  • F: < 50%

However, please note that the grades are subject to curving and the cut-offs may be adjusted such that a grade distribution similar to previous semester can be achieved.

Exam

The midterm exam will be the only exam in this class. There is no final exam. The midterm exam will take place on Tuesday, March 30th, 6:00 - 7:30 pm. Please block off this time on your calendar.

Please note that the exam is taking place in the evening to accomodate for different time zones. More details about the exam (format and types of questions) will be shared at a later time in this semester.

Class Project

The goal of working on a class project is three-fold. First, it will provide you with the opportunity to apply the concepts learned in this class creatively, which helps you with understanding material more deeply. Second, designing and working on a unique project in a team which is something that you will encounter, if you haven’t already, rather sooner than later in life, and this course project helps with preparing for that. Third, along with the opportunity to practice and the satisfaction of working creatively, students can use this project to enhance their portfolio or resume.

The project consists of 3 parts:

  1. a project proposal,
  2. a short project presentation,
  3. and a project report.

For the class project, you will be working in teams of three.

The project proposal is a short 2-3 page report outlining your plans; it will be due on March 1st. The goal of this proposal is to share your plans with me (the instructor) so that I can provide constructive feedback.

The project presentation will be a (pre-recorded) oral presentation (approx. 8-10 min) that is due at the end of the semester. The goal of this presentation is to practice summarizing and communicating your project to an audience.

The final report is a 8-page report in a conference paper format. It will be submitted at the end of the semester. The purpose of this report is to provide a comprehensive and professional description of your project.

As part of this project experience, each student will also be peer-reviewing 3 talks and 3 reports. More details about the report format and procedures will be shared later in this semester. Examples will be provided.

Other Important Course Information

Rules, Rights & Responsibilities

See the Guides’s Rules, Rights and Responsibilities

Academic Integrity

By enrolling in this course, each student assumes the responsibilities of an active participant in UW-Madison’s community of scholars in which everyone’s academic work and behavior are held to the highest academic integrity standards. Academic misconduct compromises the integrity of the university. Cheating, fabrication, plagiarism, unauthorized collaboration, and helping others commit these acts are examples of academic misconduct, which can result in disciplinary action. This includes but is not limited to failure on the assignment/course, disciplinary probation, or suspension. Substantial or repeated cases of misconduct will be forwarded to the Office of Student Conduct & Community Standards for additional review. For more information, refer to studentconduct.wiscweb.wisc.edu/academic-integrity/.

Accommodations for Students with Disabilities

McBurney Disability Resource Center syllabus statement: “The University of Wisconsin-Madison supports the right of all enrolled students to a full and equal educational opportunity. The Americans with Disabilities Act (ADA), Wisconsin State Statute (36.12), and UW-Madison policy (Faculty Document 1071) require that students with disabilities be reasonably accommodated in instruction and campus life. Reasonable accommodations for students with disabilities is a shared faculty and student responsibility. Students are expected to inform faculty [me] of their need for instructional accommodations by the end of the third week of the semester, or as soon as possible after a disability has been incurred or recognized. Faculty [I], will work either directly with the student [you] or in coordination with the McBurney Center to identify and provide reasonable instructional accommodations. Disability information, including instructional accommodations as part of a student’s educational record, is confidential and protected under FERPA.” http://mcburney.wisc.edu/facstaffother/faculty/syllabus.php

Diversity and Inclusion

Institutional statement on diversity: “Diversity is a source of strength, creativity, and innovation for UW-Madison. We value the contributions of each person and respect the profound ways their identity, culture, background, experience, status, abilities, and opinion enrich the university community. We commit ourselves to the pursuit of excellence in teaching, research, outreach, and diversity as inextricably linked goals.

The University of Wisconsin-Madison fulfills its public mission by creating a welcoming and inclusive community for people from every background – people who as students, faculty, and staff serve Wisconsin and the world.” https://diversity.wisc.edu/

COVID-19 Context

During the global COVID-19 pandemic, we must prioritize our collective health and safety to keep ourselves, our campus, and our community safe. As a university community, we must work together to prevent the spread of the virus and to promote the collective health and welfare of our campus and surrounding community.

Information on COVID-19 is constantly changing. Students should be attentive to University communications regarding COVID-19 that may alter instruction and supersede parts of this syllabus.

COVID-19 Updates

Please see https://covidresponse.wisc.edu for the recent updates and developments.