# Evals Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/evals
> Markdown URL: https://aitinkerers.org/technologies/evals.md
> Technology record last updated: 2026-04-25T08:15:13Z
> Generated: 2026-10-25T02:01:36Z

Evals is OpenAI's open-source framework: systematically benchmark Large Language Models (LLMs) and LLM-powered systems for performance, accuracy, and stability.

This is OpenAI's open-source framework for rigorous LLM evaluation, providing a structured, reproducible method to test models like GPT-4 against specific criteria (accuracy, reasoning, instruction-following). The platform includes a public registry of benchmarks and allows developers to create custom evals: use proprietary data to match application needs without public exposure. Evals is essential for continuous quality assurance (QA), catching regressions, and ensuring stability before any production deployment.

- Official technology site: https://github.com/openai/evals
- Public AI Tinkerers demos and talks: 4
- Result page: 1 of 1

## Recent Public Talks and Demos

### [NeverTwice: cost optimized API's for LLM inference tailored to your personal use case.](https://lausanne.aitinkerers.org/talks/rsvp_oYss-NBmvTQ)

- Event context: AI Tinkerers Lausanne June 2026 Meetup — 2026-06-25 — Lausanne
- Public talk page: https://lausanne.aitinkerers.org/talks/rsvp_oYss-NBmvTQ

### [Agentic AI Evaluation](https://singapore.aitinkerers.org/talks/rsvp_WUfjRFh_OmY)

- Event context: AIT Singapore: scalable agentic workflows - 12th August 2025 — 2025-08-12 — Singapore
- Public talk page: https://singapore.aitinkerers.org/talks/rsvp_WUfjRFh_OmY

### [Using LLMs for finetuning humans to recognize art](https://nyc.aitinkerers.org/talks/rsvp_3cErAke0iuQ)

- Event context: Building with Claude | An evening with Anthropic — 2024-10-21 — New York City
- Public talk page: https://nyc.aitinkerers.org/talks/rsvp_3cErAke0iuQ

### [A new CLI/Library for evals](https://austin.aitinkerers.org/talks/rsvp_tmBFag9qz7M)

- Event context: July Meetup: Community AI Demos — 2024-07-11 — Austin
- Public talk page: https://austin.aitinkerers.org/talks/rsvp_tmBFag9qz7M

## Related Technologies

- [LLMs](https://aitinkerers.org/technologies/llms) ([Markdown](https://aitinkerers.org/technologies/llms.md)) — 84 public demos
- [Benchmark dataset](https://aitinkerers.org/technologies/benchmark-dataset) ([Markdown](https://aitinkerers.org/technologies/benchmark-dataset.md)) — 1 public demo
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [Claude](https://aitinkerers.org/technologies/claude) ([Markdown](https://aitinkerers.org/technologies/claude.md)) — 180 public demos
- [CLI](https://aitinkerers.org/technologies/cli) ([Markdown](https://aitinkerers.org/technologies/cli.md)) — 9 public demos
- [datasets](https://aitinkerers.org/technologies/datasets) ([Markdown](https://aitinkerers.org/technologies/datasets.md)) — 6 public demos
- [GPT-3](https://aitinkerers.org/technologies/gpt-3) ([Markdown](https://aitinkerers.org/technologies/gpt-3.md)) — 191 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [Keras](https://aitinkerers.org/technologies/keras) ([Markdown](https://aitinkerers.org/technologies/keras.md)) — 74 public demos
- [LangChain](https://aitinkerers.org/technologies/langchain) ([Markdown](https://aitinkerers.org/technologies/langchain.md)) — 448 public demos
- [Langfuse](https://aitinkerers.org/technologies/langfuse) ([Markdown](https://aitinkerers.org/technologies/langfuse.md)) — 13 public demos
- [Library](https://aitinkerers.org/technologies/library) ([Markdown](https://aitinkerers.org/technologies/library.md)) — 4 public demos
- [LiteLLM](https://aitinkerers.org/technologies/litellm) ([Markdown](https://aitinkerers.org/technologies/litellm.md)) — 21 public demos
- [LLMstats](https://aitinkerers.org/technologies/llmstats) ([Markdown](https://aitinkerers.org/technologies/llmstats.md)) — 1 public demo
- [MetGuessr](https://aitinkerers.org/technologies/metguessr) ([Markdown](https://aitinkerers.org/technologies/metguessr.md)) — 1 public demo
- [NotDiamond](https://aitinkerers.org/technologies/notdiamond) ([Markdown](https://aitinkerers.org/technologies/notdiamond.md)) — 1 public demo
- [ONNX](https://aitinkerers.org/technologies/onnx) ([Markdown](https://aitinkerers.org/technologies/onnx.md)) — 84 public demos
- [OpenRouter](https://aitinkerers.org/technologies/openrouter) ([Markdown](https://aitinkerers.org/technologies/openrouter.md)) — 36 public demos