# NanoGPT Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/nanogpt
> Markdown URL: https://aitinkerers.org/technologies/nanogpt.md
> Technology record last updated: 2026-02-23T01:19:44Z
> Generated: 2026-09-21T15:53:29Z

NanoGPT is Andrej Karpathy's minimalist, PyTorch-based implementation of the GPT-2 transformer: a streamlined, ~300-line codebase designed for rapid training and educational clarity.

NanoGPT, developed by Andrej Karpathy, is the definitive minimalist PyTorch implementation of the GPT-2 transformer architecture. It prioritizes "teeth over education," delivering a streamlined, efficient codebase: `model.py` and `train.py` are each approximately 300 lines of Python. This simplicity allows users to quickly train or fine-tune medium-sized GPTs; for example, it can reproduce the 124M parameter GPT-2 model on OpenWebText. The project is a core resource for researchers and practitioners seeking clarity, speed, and a highly hackable foundation for large language model experimentation.

- Official technology site: https://github.com/karpathy/nanoGPT
- Public AI Tinkerers demos and talks: 4
- Result page: 1 of 1

## Recent Public Talks and Demos

### [NanoGPT-inference: How to build LLM inference from scratch](https://belgium.aitinkerers.org/talks/rsvp_YjYVlZRBYJY)

LLM inference, the engineering behind serving LLMs efficiently and economically, is becoming increasingly important. In this post, I'll show you how to speed up LLM inference with various techniques. I also release the code of each inference engine as a simple extension to Karpathy's NanoGPT.

- Event context: AI Tinkerers Ghent Meetup - February 11 — 2026-02-11 — Belgium
- Public talk page: https://belgium.aitinkerers.org/talks/rsvp_YjYVlZRBYJY

### [Too Big to Think](https://austin.aitinkerers.org/talks/rsvp_v17yXVtbeFY)

This is an shortened version of the Oral Presentation of my paper Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers https://arxiv.org/abs/2506.09099 I gave an oral presentation of this paper at the TTODLer-FM workshop at ICML 2025: https://icml.cc/virtual/2025/workshop/39957 If you're interested in seeing the recording of that presentation, you can click the SlidesLive Video Button for the 9:30am timestamp (Under Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers) Here is the paper's abstract: The relationship between memorization and generalization in large language models (LLMs) remains an open area of research, with growing evidence that the two are deeply intertwined. In this work, we investigate this relationship by pre-training a series of capacity-limited Transformer models from scratch on two synthetic character-level tasks designed to separately probe generalization (via arithmetic extrapolation) and memorization (via factual recall). We observe a consistent trade-off: small models extrapolate to unseen arithmetic cases but fail to memorize facts, while larger models memorize but fail to extrapolate. An intermediate-capacity model exhibits a similar shift toward memorization. When trained on both tasks jointly, no model (regardless of size) succeeds at extrapolation. These findings suggest that pre-training may intrinsically favor one learning mode over the other. By isolating these dynamics in a controlled setting, our study offers insight into how model capacity shapes learning behavior and offers broader implications for the design and deployment of small language models.

- Event context: AI Tinkerers Austin AI Demo Night w/ Google & Comet — 2025-10-09 — Austin
- Public talk page: https://austin.aitinkerers.org/talks/rsvp_v17yXVtbeFY

### [nanoDiffusion](https://zurich.aitinkerers.org/talks/rsvp_iPlQoUZ8s9U)

Nano sized image generator than can be trained in a couple of CPU minutes on your personal laptop.

- Event context: AI Tinkerers Zurich - February 6 — 2025-02-06 — Zürich
- Public talk page: https://zurich.aitinkerers.org/talks/rsvp_iPlQoUZ8s9U

### [Speedrunning NanoGPT training runs](https://portland.aitinkerers.org/talks/rsvp_rFDus1GRZYc)

NanoGPT is a small scale (124m parameter) transformer architecture initially made by Andrei Karpathy for learning how to build/train LLMs. There is an active, ongoing effort to integrate various architectural changes and optimizer improvements into NanoGPT in order to break training speed records. These speedruns are tracked by Keller Jordan (who started the speedrunning effort) here: https://github.com/KellerJordan/modded-nanogpt/tree/master. As of last week, with PyTorch 2.5 and the new Muon optimizer, NanoGPT speedruns have reached parity with LLM.c in training speeds for 120m scale models. My current experiment/attempt at speedrunning is taking Keller Jordan's speedrun and integrating differential attention layers (https://arxiv.org/abs/2410.05258) into it. DIfferential attention splits the attention heads into excitatory and inhibitory attention heads, allowing the attention layer to focus more clearly on context which it deems relevant. The paper showed a 25-30% improvement in loss for models of the same parameter count, so I am hoping that combining it with the Muon optimizer will break the latest speedrunning record. My fork is here: https://github.com/RyanPersson/modded-nanogpt/tree/differential-flash-attention (Differential-flash-attention branch, master is in sync with Keller Jordan's repo.) I've been training/testing on Lambda Cloud H100 instances. I got my varient running on a single H100 node last night, but it still segfaults on an 8xH100 DGX cluster. I am hoping to get that resolved and get it working by the meetup next Tuesday, but figured I would go ahead and put in a request to present because it'd be fun to talk about either way.

- Event context: AI Tinkerers Portland @Nedspace - October 29th 2024, 6pm — 2024-10-30 — Portland
- Public talk page: https://portland.aitinkerers.org/talks/rsvp_rFDus1GRZYc

## Related Technologies

- [PyTorch](https://aitinkerers.org/technologies/pytorch) ([Markdown](https://aitinkerers.org/technologies/pytorch.md)) — 273 public demos
- [AMD 9800X3D](https://aitinkerers.org/technologies/amd-9800x3d) ([Markdown](https://aitinkerers.org/technologies/amd-9800x3d.md)) — 1 public demo
- [CPU](https://aitinkerers.org/technologies/cpu) ([Markdown](https://aitinkerers.org/technologies/cpu.md)) — 3 public demos
- [diffusion transformers](https://aitinkerers.org/technologies/diffusion-transformers) ([Markdown](https://aitinkerers.org/technologies/diffusion-transformers.md)) — 1 public demo
- [GeForce RTX 5090](https://aitinkerers.org/technologies/geforce-rtx-5090) ([Markdown](https://aitinkerers.org/technologies/geforce-rtx-5090.md)) — 1 public demo
- [LLM](https://aitinkerers.org/technologies/llm) ([Markdown](https://aitinkerers.org/technologies/llm.md)) — 123 public demos
- [Muon optimizer](https://aitinkerers.org/technologies/muon-optimizer) ([Markdown](https://aitinkerers.org/technologies/muon-optimizer.md)) — 1 public demo
- [nanoDiffusion](https://aitinkerers.org/technologies/nanodiffusion) ([Markdown](https://aitinkerers.org/technologies/nanodiffusion.md)) — 1 public demo
- [NVIDIA H100](https://aitinkerers.org/technologies/nvidia-h100) ([Markdown](https://aitinkerers.org/technologies/nvidia-h100.md)) — 4 public demos
- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 662 public demos
- [Stable Diffusion](https://aitinkerers.org/technologies/stable-diffusion) ([Markdown](https://aitinkerers.org/technologies/stable-diffusion.md)) — 32 public demos
- [Transformer](https://aitinkerers.org/technologies/transformer) ([Markdown](https://aitinkerers.org/technologies/transformer.md)) — 11 public demos