# Transformer Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/transformer
> Markdown URL: https://aitinkerers.org/technologies/transformer.md
> Technology record last updated: 2026-02-25T18:58:43Z
> Generated: 2026-09-21T19:48:14Z

The Transformer is a neural network architecture that uses a multi-head self-attention mechanism to process sequences in parallel, replacing slower recurrent (RNN) and convolutional (CNN) layers.

The Transformer architecture, introduced in the landmark 2017 paper 'Attention Is All You Need' by Vaswani et al. (Google), revolutionized sequence-to-sequence modeling. It operates entirely on an attention mechanism (multi-head self-attention), eliminating the need for sequential processing via Recurrent Neural Networks (RNNs). This design allows for massive parallelization, drastically reducing training time and enabling the scale-up of models to billions of parameters. It is the foundational technology for all modern Large Language Models (LLMs), including BERT and the Generative Pre-trained Transformer (GPT) series, driving state-of-the-art performance across Natural Language Processing (NLP) and computer vision tasks.

- Official technology site: https://arxiv.org/abs/1706.03762
- Public AI Tinkerers demos and talks: 11
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Leigh Fair-Smiley - Attention Based Context Unlocking Indefinite Context Memory](https://portland.aitinkerers.org/talks/rsvp_ciS8kfufDe0)

ML Researcher Specializing in machine learning experimentation, model behavior, and advancing practical ML systems.

- Event context: Portland: Agentic Systems Deep Dive — Building the Next Generation of AGI — 2025-12-04 — Portland
- Public talk page: https://portland.aitinkerers.org/talks/rsvp_ciS8kfufDe0

### [Molecular structure elucidation, Latent test-time compute and a different way of doing Entropix.](https://milan.aitinkerers.org/talks/rsvp_NmpNgyS3NKQ)

Entropix took the twitter ML world by storm just a few months ago. But now that test-time compute is hot in token space via RL-finetuning, entropix may look like arelic from the past. I present a specific instance of autoregressive transformers where the output space is fragile aenough not to allow exploration in token space, an hence utilize model output distribution uncertainty, in the shape of entropy and varentropy, to allocate test-time compute and significantly improve performance on a moleculare structure elucidation task via SMILES prediction.

- Event context: AI Tinkerers Milan - May 8, 2025 — 2025-05-08 — Milan
- Public talk page: https://milan.aitinkerers.org/talks/rsvp_NmpNgyS3NKQ

### [nanoDiffusion](https://zurich.aitinkerers.org/talks/rsvp_iPlQoUZ8s9U)

Nano sized image generator than can be trained in a couple of CPU minutes on your personal laptop.

- Event context: AI Tinkerers Zurich - February 6 — 2025-02-06 — Zürich
- Public talk page: https://zurich.aitinkerers.org/talks/rsvp_iPlQoUZ8s9U

### [Multi task audio models using auto regressive transformers](https://bengaluru.aitinkerers.org/talks/rsvp_2eKJ-4Lkk5E)

We have pretrained and finetuned a single model that can take in audio or text and output audio or text. This single model can be used for multiple audio-related tasks, like TTS, ASR, and text-to-voice completion. We will demo the TTS part and talk about the overall architecture of the model. We have hosted the model with ultra-fast inference and low latency.

- Event context: AI Tinkerers Bangalore - November - RSVP REQUIRED — 2024-11-12 — Bengaluru
- Public talk page: https://bengaluru.aitinkerers.org/talks/rsvp_2eKJ-4Lkk5E

### [toby](https://nyc.aitinkerers.org/talks/rsvp_ufDluDL80Gg)

toby enables you to speak in other languages on any video call. it’s a desktop app that translates audio on your computer mid-sentence. just download our desktop app, set it up on any video call platform, & speak other languages with minimum delay 🚀

- Event context: AI Tinkerers August Meetup — 2024-08-28 — New York City
- Public talk page: https://nyc.aitinkerers.org/talks/rsvp_ufDluDL80Gg

### [Attention in 4 lines](https://berlin.aitinkerers.org/talks/rsvp_yaSBWEmfNVA)

Everybody is using attention but do you really understand how does it click? Today our speaker Dr. Ünsal Gökdağ will explain the attention mechanics in 4 lines of code with a bit of a backstory.

- Event context: AI Tinkerers Berlin - August 22 — 2024-08-22 — Berlin
- Public talk page: https://berlin.aitinkerers.org/talks/rsvp_yaSBWEmfNVA

### [Virtual avatar generation models as world navigators](https://sf.aitinkerers.org/talks/rsvp_VKgGo_Qx060)

A novel video model capable of simulating human movement within a given environment by assuming the parameters of a virtual avatar.

- Event context: AI Tinkerers - San Francisco - Summer Edition - July 2024 — 2024-07-12 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_VKgGo_Qx060

### [A transformer from scratch in go](https://toronto.aitinkerers.org/talks/rsvp_qjjZ9YhPLb8)

This is a port of karpathy's llm.c in Golang. The demo will feature an overview of the llm.c project, the go fork, and show a quick overview of the code structure, especially on the unit tests that allow for tweaking of parameters to see what's actually going on inside the transformer architecture.

- Event context: AI Tinkerers - Toronto - June 2024 Meetup — 2024-06-27 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_qjjZ9YhPLb8

### [David vs Goliath - the power of small models](https://amsterdam.aitinkerers.org/talks/rsvp_YiNA7j4I9TU)

With the trend to larger and larger models, we might forget that a specialized, domain-specific small model might be a better solution to a given problem, not only on raw performance but also when accounting for all factors like run-time requirements and costs. At Chimnie, we needed specialised parsing of postal addresses in the UK. After using off-the-shelf larger models, we decided to train our own transformer model from scratch, specialized in UK addresses. It turned out to be more performant while requiring a fraction of the resources of other more general models. And it was not hard to do! In this 5 minute demo I'll show how we built the dataset, the training pipeline and the final working model.

- Event context: AI Tinkerers Amsterdam - Inaugural Meetup - May — 2024-05-23 — Amsterdam
- Public talk page: https://amsterdam.aitinkerers.org/talks/rsvp_YiNA7j4I9TU

### [Training Diffusion Transformers for Style Transfer (for the GPU poor)](https://la.aitinkerers.org/talks/rsvp_z4k8pNn_y6w)

We trained diffusion transformers to generate images from text in the style of the Simpsons and Legos on a single commodity GPU.

- Event context: May 21st - LA AI Tinkerers Meetup & Demos — 2024-05-22 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_z4k8pNn_y6w

### [Spreadsheets are all you need](https://seattle.aitinkerers.org/talks/rsvp_8v5-XzScCnw)

Spreadsheets-are-all-you-need implements the forward pass of GPT2 entirely in Excel using standard spreadsheet functions. By using a spreadsheet anyone can explore and play directly with how a “real” transformer works under the hood with minimal abstractions to get in the way.

- Event context: AI Tinkerers Seattle - October Meetup — 2023-10-26 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_8v5-XzScCnw

## Related Technologies

- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 662 public demos
- [Stable Diffusion](https://aitinkerers.org/technologies/stable-diffusion) ([Markdown](https://aitinkerers.org/technologies/stable-diffusion.md)) — 32 public demos
- [Amazon Translate](https://aitinkerers.org/technologies/amazon-translate) ([Markdown](https://aitinkerers.org/technologies/amazon-translate.md)) — 1 public demo
- [Attention](https://aitinkerers.org/technologies/attention) ([Markdown](https://aitinkerers.org/technologies/attention.md)) — 1 public demo
- [Audio Tokenization](https://aitinkerers.org/technologies/audio-tokenization) ([Markdown](https://aitinkerers.org/technologies/audio-tokenization.md)) — 1 public demo
- [Autodesk MotionBuilder](https://aitinkerers.org/technologies/autodesk-motionbuilder) ([Markdown](https://aitinkerers.org/technologies/autodesk-motionbuilder.md)) — 1 public demo
- [Avatar Generation](https://aitinkerers.org/technologies/avatar-generation) ([Markdown](https://aitinkerers.org/technologies/avatar-generation.md)) — 1 public demo
- [C](https://aitinkerers.org/technologies/c-2) ([Markdown](https://aitinkerers.org/technologies/c-2.md)) — 6 public demos
- [CPU](https://aitinkerers.org/technologies/cpu) ([Markdown](https://aitinkerers.org/technologies/cpu.md)) — 3 public demos
- [DALL-E 2](https://aitinkerers.org/technologies/dall-e-2) ([Markdown](https://aitinkerers.org/technologies/dall-e-2.md)) — 7 public demos
- [DALL-E 3](https://aitinkerers.org/technologies/dall-e-3) ([Markdown](https://aitinkerers.org/technologies/dall-e-3.md)) — 12 public demos
- [decoder-only](https://aitinkerers.org/technologies/decoder-only) ([Markdown](https://aitinkerers.org/technologies/decoder-only.md)) — 1 public demo
- [Desktop Application](https://aitinkerers.org/technologies/desktop-application) ([Markdown](https://aitinkerers.org/technologies/desktop-application.md)) — 1 public demo
- [Diffusion model](https://aitinkerers.org/technologies/diffusion-model) ([Markdown](https://aitinkerers.org/technologies/diffusion-model.md)) — 3 public demos
- [diffusion transformers](https://aitinkerers.org/technologies/diffusion-transformers) ([Markdown](https://aitinkerers.org/technologies/diffusion-transformers.md)) — 1 public demo
- [entropix](https://aitinkerers.org/technologies/entropix) ([Markdown](https://aitinkerers.org/technologies/entropix.md)) — 1 public demo
- [Excel](https://aitinkerers.org/technologies/excel) ([Markdown](https://aitinkerers.org/technologies/excel.md)) — 3 public demos
- [Fairseq](https://aitinkerers.org/technologies/fairseq) ([Markdown](https://aitinkerers.org/technologies/fairseq.md)) — 1 public demo