# Mixture of Experts (MoE) Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/mixture-of-experts-moe
> Markdown URL: https://aitinkerers.org/technologies/mixture-of-experts-moe.md
> Technology record last updated: 2026-04-07T04:24:52Z
> Generated: 2026-10-23T13:54:08Z

MoE scales model capacity by activating only a fraction of total parameters per token via specialized sub-networks.

Mixture of Experts (MoE) replaces dense layers with sparse expert layers, using a router to direct input tokens to the most relevant sub-networks. This architecture allows models like Mixtral 8x7B to maintain the inference speed of a 12B parameter model while leveraging the knowledge of a 45B parameter system. By decoupling total parameter count from compute cost, MoE enables massive scaling (seen in Google's Switch Transformer hitting 1.6 trillion parameters) without a linear increase in FLOPs. It is the core mechanism behind GPT-4's efficiency, balancing high-tier reasoning with manageable latency.

- Official technology site: https://huggingface.co/blog/moe
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [MoE parameters, MoE problems: visualizing Mixture of Experts Routing Layers](https://manchester-nh.aitinkerers.org/talks/rsvp_vIT84Fso8c4)

- Event context: AI Tinkerers Manchester (Bedford), NH - April 2026 Meetup — 2026-04-15 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_vIT84Fso8c4

## Related Technologies

- [Forward hook](https://aitinkerers.org/technologies/forward-hook) ([Markdown](https://aitinkerers.org/technologies/forward-hook.md)) — 1 public demo
- [IBM Granite](https://aitinkerers.org/technologies/ibm-granite) ([Markdown](https://aitinkerers.org/technologies/ibm-granite.md)) — 4 public demos
- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 686 public demos
- [PyTorch](https://aitinkerers.org/technologies/pytorch) ([Markdown](https://aitinkerers.org/technologies/pytorch.md)) — 274 public demos