Evals

Evals is OpenAI's open-source framework: systematically benchmark Large Language Models (LLMs) and LLM-powered systems for performance, accuracy, and stability.

This is OpenAI's open-source framework for rigorous LLM evaluation, providing a structured, reproducible method to test models like GPT-4 against specific criteria (accuracy, reasoning, instruction-following). The platform includes a public registry of benchmarks and allows developers to create custom evals: use proprietary data to match application needs without public exposure. Evals is essential for continuous quality assurance (QA), catching regressions, and ensuring stability before any production deployment.

https://github.com/openai/evals