MMLU is a benchmark of multiple-choice questions spanning 57 academic subjects — from high-school biology to graduate-level professional medicine — designed to measure the breadth of knowledge a language model has absorbed during pre-training. The benchmark became the standard "general knowledge" comparison metric for large language models through 2022-2024.
By 2025 MMLU had been largely saturated. Top models routinely scored above 90 percent, and further progress on the benchmark became difficult to interpret because the ceiling was dominated by ambiguous or mislabeled questions. The community migrated to harder benchmarks, including Humanity's Last Exam, GPQA Diamond, and task-based benchmarks like OSWorld.
MMLU remains useful for comparing smaller open-source models that have not yet saturated it, and for regression testing during model training. It is no longer a meaningful differentiator at the frontier.