# MMLU Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/mmlu
> Markdown URL: https://aitinkerers.org/technologies/mmlu.md
> Technology record last updated: 2026-03-13T04:40:34Z
> Generated: 2026-10-25T15:38:13Z

MMLU (Massive Multitask Language Understanding) is the definitive 57-subject benchmark for rigorously testing a large language model's (LLM) general knowledge and reasoning depth.

MMLU is the premier benchmark for evaluating Large Language Models (LLMs): Massive Multitask Language Understanding. It features 15,908 multiple-choice questions spanning 57 diverse academic and professional subjects (e.g., law, computer science, US history). This test measures a model's breadth of world knowledge and problem-solving ability, pushing performance past simple conversational tasks. When released in 2020, the top model (GPT-3 175B) scored 43.9%; today's leaders like GPT-4o consistently hit 88% accuracy. We use MMLU to track model generalization and identify critical shortcomings across specialized domains.

- Official technology site: https://arxiv.org/abs/2009.03300
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Automating LLM as a judge with EvalForge and Weave](https://seattle.aitinkerers.org/talks/rsvp_5bPYaF3NHAo)

- Event context: AI Tinkerers Seattle - September 2024 Meetup — 2024-09-27 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_5bPYaF3NHAo

## Related Technologies

- [EvalForge](https://aitinkerers.org/technologies/evalforge) ([Markdown](https://aitinkerers.org/technologies/evalforge.md)) — 2 public demos
- [HumanEval](https://aitinkerers.org/technologies/humaneval) ([Markdown](https://aitinkerers.org/technologies/humaneval.md)) — 1 public demo
- [Weave](https://aitinkerers.org/technologies/weave) ([Markdown](https://aitinkerers.org/technologies/weave.md)) — 3 public demos
- [Weights & Biases](https://aitinkerers.org/technologies/weights-biases) ([Markdown](https://aitinkerers.org/technologies/weights-biases.md)) — 9 public demos