Some Techniques To Make Your PyTorch Models Train (Much) Faster

Practical training optimizations: compilation, mixed precision, and distributed strategies.

Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch

Gradient checkpointing, quantization, and other memory-saving techniques.

Accelerating Large Language Models with Mixed-Precision Techniques

How float16 and bfloat16 speed up LLM training and inference.

Comparing Different Automatic Image Augmentation Methods in PyTorch

AutoAugment, RandAugment, and AugMix compared with PyTorch code.

DGX Spark and Mac Mini for Local PyTorch Development

Hands-on impressions of local GPU hardware for LLM and PyTorch work.

PyTorch — Tutorials & Practical Guides

Practical tutorials and guides focused on PyTorch — from memory management and training speed to GPU setup and data pipelines. Complements my Build a Large Language Model (From Scratch) book.

2025

Oct 29 DGX Spark and Mac Mini for Local PyTorch Development The DGX Spark for local LLM inferencing and fine-tuning was a pretty popular discussion topic recently. I got to play with one myself, primarily working...

2023

Jul 1 Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch Peak memory consumption is a common bottleneck when training deep learning models such as vision transformers and LLMs. This article provides a series of...
May 11 Accelerating Large Language Models with Mixed-Precision Techniques Training and using large language models (LLMs) is expensive due to their large compute requirements and memory footprints. This article will explore how...
Feb 23 Some Techniques To Make Your PyTorch Models Train (Much) Faster This blog post outlines techniques for improving the training performance of your PyTorch model without compromising its accuracy. To do so, we will wrap a...
Jan 29 Comparing Different Automatic Image Augmentation Methods in PyTorch Data augmentation is a key tool in reducing overfitting, whether it's for images or text. This article compares three Auto Image Data Augmentation...
Jan 15 Training an XGBoost Classifier Using Cloud GPUs Without Worrying About Infrastructure Imagine you want to quickly train a few machine learning or deep learning models on the cloud but don't want to deal with cloud infrastructure. This short...

2022

Jul 5 No, We Don't Have to Choose Batch Sizes As Powers Of 2 Regarding neural network training, I think we are all guilty of doing this: we choose our batch sizes as powers of 2, that is, 64, 128, 256, 512, 1024, and...
Jun 30 Sharing Deep Learning Research Models with Lightning Part 2: Leveraging the Cloud Deploys a Super Resolution GAN (SRGAN) app to the cloud with Lightning.ai — covers containerization, model serving, and production deployment steps.
Jun 17 Sharing Deep Learning Research Models with Lightning Part 1: Building A Super Resolution App Build a Super Resolution GAN app with Lightning.ai: interactive UI, model serving, and a modern workflow for sharing deep learning research.
Jun 12 Taking Datasets, DataLoaders, and PyTorch’s New DataPipes for a Spin The PyTorch team recently announced TorchData, a prototype library focused on implementing composable and reusable data loading utilities for PyTorch. In...
May 18 Running PyTorch on the M1 GPU Today, PyTorch officially introduced GPU support for Apple's ARM M1 chips. This is an exciting day for Mac users out there, so I spent a few minutes trying...
Mar 24 TorchMetrics TorchMetrics is a really nice and convenient library that lets us compute the performance of models in an iterative fashion. It's designed with PyTorch (and...
Feb 25 Machine Learning with PyTorch and Scikit-Learn Machine Learning with PyTorch and Scikit-Learn has been a long time in the making, and I am excited to finally get to talk about the release of my new book...

2021

Jan 21 Book Review: Deep Learning With PyTorch After its release in August 2020, Deep Learning with PyTorch has been sitting on my shelf before I finally got a chance to read it during this winter break...