# decoder-only Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/decoder-only
> Markdown URL: https://aitinkerers.org/technologies/decoder-only.md
> Technology record last updated: 2026-05-05T02:41:29Z
> Generated: 2026-09-21T01:41:07Z

This architecture is the workhorse of generative Large Language Models (LLMs), specializing in autoregressive text generation via causal (masked) multi-head self-attention.

The decoder-only model is a streamlined Transformer variant, removing the original encoder component to focus exclusively on sequence generation (Source 1.3, 1.5). It operates by predicting the next token based on all preceding tokens in the input sequence, a process called autoregression (Source 1.8). The core mechanism is a stack of decoder blocks, each utilizing masked self-attention to ensure the model cannot ‘look ahead’ at future tokens, maintaining causal integrity (Source 1.2, 1.9). This design powers industry-leading models like OpenAI's GPT series (GPT-3, GPT-4) and Meta's Llama family (Llama-2, Llama-3), making it the standard for tasks requiring fluent, context-aware content creation (Source 1.3, 1.6).

- Official technology site: https://machinelearningmastery.com/building-a-decoder-only-transformer-model-like-llama-2-and-llama-3/
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Molecular structure elucidation, Latent test-time compute and a different way of doing Entropix.](https://milan.aitinkerers.org/talks/rsvp_NmpNgyS3NKQ)

Entropix took the twitter ML world by storm just a few months ago. But now that test-time compute is hot in token space via RL-finetuning, entropix may look like arelic from the past. I present a specific instance of autoregressive transformers where the output space is fragile aenough not to allow exploration in token space, an hence utilize model output distribution uncertainty, in the shape of entropy and varentropy, to allocate test-time compute and significantly improve performance on a moleculare structure elucidation task via SMILES prediction.

- Event context: AI Tinkerers Milan - May 8, 2025 — 2025-05-08 — Milan
- Public talk page: https://milan.aitinkerers.org/talks/rsvp_NmpNgyS3NKQ

## Related Technologies

- [entropix](https://aitinkerers.org/technologies/entropix) ([Markdown](https://aitinkerers.org/technologies/entropix.md)) — 1 public demo
- [layer looping](https://aitinkerers.org/technologies/layer-looping) ([Markdown](https://aitinkerers.org/technologies/layer-looping.md)) — 1 public demo
- [Muon](https://aitinkerers.org/technologies/muon) ([Markdown](https://aitinkerers.org/technologies/muon.md)) — 1 public demo
- [Transformer](https://aitinkerers.org/technologies/transformer) ([Markdown](https://aitinkerers.org/technologies/transformer.md)) — 11 public demos