Deep Learning Resources

Neural Networks and Deep Learning Model Zoo

Python 3.7

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

Multilayer Perceptrons

Convolutional Neural Networks

Basic

Concepts

  • Replacing Fully-Connnected by Equivalent Convolutional Layers [PyTorch]

All-Convolutional

  • All-Convolutional Neural Network [PyTorch]

AlexNet

VGG

ResNet

  • ResNet and Residual Blocks [PyTorch]
  • ResNet-18 Digit Classifier Trained on MNIST [PyTorch]
  • ResNet-18 Gender Classifier Trained on CelebA [PyTorch]
  • ResNet-34 Digit Classifier Trained on MNIST [PyTorch]
  • ResNet-34 Gender Classifier Trained on CelebA [PyTorch]
  • ResNet-50 Digit Classifier Trained on MNIST [PyTorch]
  • ResNet-50 Gender Classifier Trained on CelebA [PyTorch]
  • ResNet-101 Gender Classifier Trained on CelebA [PyTorch]
  • ResNet-152 Gender Classifier Trained on CelebA [PyTorch]

Network in Network

  • Network in Network CIFAR-10 Classifier [PyTorch]

Metric Learning

  • Siamese Network with Multilayer Perceptrons [TensorFlow 1]

Autoencoders

Fully-connected Autoencoders

Convolutional Autoencoders

  • Convolutional Autoencoder with Deconvolutions / Transposed Convolutions[TensorFlow 1] [PyTorch]
  • Convolutional Autoencoder with Deconvolutions (without pooling operations) [PyTorch]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation [TensorFlow 1] [PyTorch]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation – Trained on CelebA [PyTorch]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation – Trained on Quickdraw [PyTorch]

Variational Autoencoders

  • Variational Autoencoder [PyTorch]
  • Convolutional Variational Autoencoder [PyTorch]

Conditional Variational Autoencoders

  • Conditional Variational Autoencoder (with labels in reconstruction loss) [PyTorch]
  • Conditional Variational Autoencoder (without labels in reconstruction loss) [PyTorch]
  • Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) [PyTorch]
  • Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) [PyTorch]

General Adversarial Networks (GANs)

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

  • A simple single-layer RNN (IMDB) [PyTorch]
  • A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) [PyTorch]
  • RNN with LSTM cells (IMDB) [PyTorch]
  • RNN with LSTM cells and Own Dataset in CSV Format (IMDB) [PyTorch]
  • RNN with GRU cells (IMDB) [PyTorch]
  • Multilayer bi-directional RNN (IMDB) [PyTorch]

Many-to-Many / Sequence-to-Sequence

  • A simple character RNN to generate new text (Charles Dickens) [PyTorch]

Ordinal Regression

  • Ordinal Regression CNN – CORAL w. ResNet34 on AFAD-Lite [PyTorch]
  • Ordinal Regression CNN – Niu et al. 2016 w. ResNet34 on AFAD-Lite [PyTorch]
  • Ordinal Regression CNN – Beckham and Pal 2016 w. ResNet34 on AFAD-Lite [PyTorch]

Tips and Tricks

PyTorch Workflows and Mechanics

Custom Datasets

  • Using PyTorch Dataset Loading Utilities for Custom Datasets – CSV files converted to HDF5 [PyTorch]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets – Face Images from CelebA [PyTorch]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets – Drawings from Quickdraw [PyTorch]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets – Drawings from the Street View House Number (SVHN) Dataset [PyTorch]

Training and Preprocessing

  • Dataloading with Pinned Memory [PyTorch]
  • Standardizing Images [PyTorch]
  • Image Transformation Examples [PyTorch]
  • Char-RNN with Own Text File [PyTorch]
  • Sentiment Classification RNN with Own CSV File [PyTorch]

Parallel Computing

  • Using Multiple GPUs with DataParallel – VGG-16 Gender Classifier on CelebA [PyTorch]

Other

  • Sequential API and hooks [PyTorch]
  • Weight Sharing Within a Layer [PyTorch]
  • Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib [PyTorch]

Autograd

  • Getting Gradients of an Intermediate Variable in PyTorch [PyTorch]

TensorFlow Workflows and Mechanics

Custom Datasets

  • Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives [TensorFlow 1]
  • Storing an Image Dataset for Minibatch Training using HDF5 [TensorFlow 1]
  • Using Input Pipelines to Read Data from TFRecords Files [TensorFlow 1]
  • Using Queue Runners to Feed Images Directly from Disk [TensorFlow 1]
  • Using TensorFlow’s Dataset API [TensorFlow 1]

Training and Preprocessing

  • Saving and Loading Trained Models – from TensorFlow Checkpoint Files and NumPy NPZ Archives [TensorFlow 1]





Free Chapters from Introduction to Artificial Neural Networks and Deep Learning: A Practical Guide with Applications in Python

  • Introduction

  • The Perceptron [Code Notebook]

  • Optimizing Cost Functions with Gradient Descent

  • Logistic Regression and Softmax Regression

  • From Softmax Regression to Multi-layer Perceptrons

  • Cross Validation and Performance Metrics

  • Regularization in Neural Networks

  • Learning Rates and Weight Initialization

  • Convolutional Neural Networks

  • Recurrent Neural Networks

  • Echostate Networks

  • Autoencoders

  • General Adverserial Neural Networks

  • Deep Generative Models

  • Reinforcement Learning

  • Appendix A: Mathematical Notation [PDF]

  • Appendix B: Algebra Basics [PDF]

  • Appendix C: Linear Algebra Essentials

  • Appendix D: Calculus and Differentiation Primer [PDF]

  • Appendix E: Python Setup

  • Appendix F: Introduction to NumPy [PDF] [Code Notebook]

  • Appendix G: TensorFlow Basics [PDF] [Code Notebook]

  • Appendix H: Cloud Computing [PDF]