# GPT-2 Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/gpt-2
> Markdown URL: https://aitinkerers.org/technologies/gpt-2.md
> Technology record last updated: 2026-03-05T10:13:23Z
> Generated: 2026-09-21T00:46:21Z

GPT-2 is a 1.5 billion-parameter, transformer-based language model from OpenAI (2019), trained on 40GB of internet text (WebText) to predict the next word, demonstrating strong zero-shot performance across diverse tasks.

GPT-2 (Generative Pre-trained Transformer 2) is a large, transformer-based language model developed by OpenAI, first released in a staged manner starting in 2019. The largest version features 1.5 billion parameters and was trained on a massive 40GB dataset called WebText, sourced from 8 million web pages. Its core objective was simple: predict the next word in a sequence. This simple training goal resulted in a powerful model capable of generating high-quality, coherent conditional synthetic text. Critically, GPT-2 demonstrated a remarkable ability to perform multiple downstream tasks—including summarization, translation, and question answering—in a 'zero-shot' setting, meaning it required no task-specific training data to achieve state-of-the-art results at the time.

- Official technology site: https://openai.com/blog/better-language-models-and-their-implications
- Public AI Tinkerers demos and talks: 4
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Building a GPT-2 level LLM from scratch](https://dubai.aitinkerers.org/talks/rsvp_LfaYKjax6j4)

This project trains a GPT-2-level large language model (LLM) from scratch, allowing you to explore how modern LLMs are built and trained. By experimenting with smaller or larger models, as well as small (public-domain text) or large (English Wikipedia) datasets, you can better understand the inner workings and trade-offs involved in training such models.

- Event context: AI Tinkerers - Dubai Meetup #4 (January) — 2025-01-05 — Dubai
- Public talk page: https://dubai.aitinkerers.org/talks/rsvp_LfaYKjax6j4

### [Spreadsheets (in Browsers) are all you need](https://seattle.aitinkerers.org/talks/rsvp_jwzkT1bwU4A)

I've ported my Spreadsheets-are-all-you-need Excel spreadsheet that implements GPT2 to the browser. In the process, I've created an interface and web component library that is a cross between a Jupyter notebook and a spreadsheet that runs locally, offline-first in the browser in JavaScript rather than Python. It's especially ideal for web developers who feel more comfortable with JavaScript to dissect and understand how a real LLM works. It also provides a gentle stepping stone to doing AI/ML in Jupyter/Python notebooks.

- Event context: AI Tinkerers Seattle - December 2024 Meetup — 2024-12-12 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_jwzkT1bwU4A

### [Enhancing AI with RAG - Techniques to improve accuracy](https://bogota.aitinkerers.org/talks/rsvp_QjI40TNRTXw)

This talk explores how to optimize AI companions for project management using RAG. Standard RAG models struggle with context, leading to incomplete or inaccurate data retrieval, which is important when managing resources and making decisions. We’ll explore GraphRAG and context-aware RAG, which use knowledge graphs and enhanced contextual embeddings to improve the precision and relevance of information retrieval.

- Event context: AI Tinkerers Octubre en Colombia 4.0 — 2024-10-30 — Bogotá
- Public talk page: https://bogota.aitinkerers.org/talks/rsvp_QjI40TNRTXw

### [Spreadsheets are all you need](https://seattle.aitinkerers.org/talks/rsvp_8v5-XzScCnw)

Spreadsheets-are-all-you-need implements the forward pass of GPT2 entirely in Excel using standard spreadsheet functions. By using a spreadsheet anyone can explore and play directly with how a “real” transformer works under the hood with minimal abstractions to get in the way.

- Event context: AI Tinkerers Seattle - October Meetup — 2023-10-26 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_8v5-XzScCnw

## Related Technologies

- [alBERT](https://aitinkerers.org/technologies/albert) ([Markdown](https://aitinkerers.org/technologies/albert.md)) — 4 public demos
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [ELMo](https://aitinkerers.org/technologies/elmo) ([Markdown](https://aitinkerers.org/technologies/elmo.md)) — 1 public demo
- [English Wikipedia](https://aitinkerers.org/technologies/english-wikipedia) ([Markdown](https://aitinkerers.org/technologies/english-wikipedia.md)) — 1 public demo
- [Excel](https://aitinkerers.org/technologies/excel) ([Markdown](https://aitinkerers.org/technologies/excel.md)) — 3 public demos
- [GPT-3](https://aitinkerers.org/technologies/gpt-3) ([Markdown](https://aitinkerers.org/technologies/gpt-3.md)) — 191 public demos
- [GPT models](https://aitinkerers.org/technologies/gpt-models) ([Markdown](https://aitinkerers.org/technologies/gpt-models.md)) — 1 public demo
- [Graph database](https://aitinkerers.org/technologies/graph-database) ([Markdown](https://aitinkerers.org/technologies/graph-database.md)) — 2 public demos
- [GraphRAG](https://aitinkerers.org/technologies/graphrag) ([Markdown](https://aitinkerers.org/technologies/graphrag.md)) — 13 public demos
- [JavaScript](https://aitinkerers.org/technologies/javascript) ([Markdown](https://aitinkerers.org/technologies/javascript.md)) — 40 public demos
- [Jupyter](https://aitinkerers.org/technologies/jupyter) ([Markdown](https://aitinkerers.org/technologies/jupyter.md)) — 8 public demos
- [Knowledge Graph](https://aitinkerers.org/technologies/knowledge-graph) ([Markdown](https://aitinkerers.org/technologies/knowledge-graph.md)) — 9 public demos
- [Public-domain text](https://aitinkerers.org/technologies/public-domain-text) ([Markdown](https://aitinkerers.org/technologies/public-domain-text.md)) — 1 public demo
- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 662 public demos
- [RAG](https://aitinkerers.org/technologies/rag) ([Markdown](https://aitinkerers.org/technologies/rag.md)) — 147 public demos
- [RoBERTa](https://aitinkerers.org/technologies/roberta) ([Markdown](https://aitinkerers.org/technologies/roberta.md)) — 118 public demos
- [T5](https://aitinkerers.org/technologies/t5) ([Markdown](https://aitinkerers.org/technologies/t5.md)) — 5 public demos
- [Transformer](https://aitinkerers.org/technologies/transformer) ([Markdown](https://aitinkerers.org/technologies/transformer.md)) — 11 public demos