Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub!

This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people.

Thanks also to everyone who shared ideas and opened PRs with improvements!

Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories).

I am also currently working on a larger applied custom “small” LLM project. It has been keeping me super busy this month, but I will share more on that soon in an upcoming Substack mega-article! It’s my longest one yet!

If you are new to it, some of the highlights in the repo include

1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.)

2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts).

3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts

4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA

So, if you don’t have any weekend plans yet, happy tinkering!

Figures from the LLMs-from-scratch repo.

Source: website version of my Substack note.