We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE) question collection, following a year-long process of cleaning and refinement with input from research communities.
HLE-Diamond consists of 1,000 questions.
Main results. We compare the performance of current models on HLE-Diamond without tools.
All models are evaluated with reasoning high, capped at 128k output tokens.
ClaudeFable 5.2
74.3%GPT-6.2Astra
66.6%Gemini 4Neon
65.2%Gemini 4Carbon
63.6%GPT-6.2Sol
58.3%ClaudeOpus 5.5
54.6%MuseWonder
34.7%ClaudeSonnet 5.5
33.1%KimiK3.1 Pro
29.5%Grok4.7
22.8%
Token Usage
Reasoning high
GPT-6.2Astra
3.3kGPT-6.2Sol
3.7kClaudeSonnet 5.5
8.6kClaudeFable 5.2
8.8kClaudeOpus 5.5
9.6kKimiK3.1 Pro
9.8kMuseWonder
16.1kGemini 4Neon
38.3kGrok4.7
43.2kGemini 4Carbon
59.2k
Dataset. HLE-Diamond consists of 500 reasoning and 500 knowledge questions, measuring reasoning and expert knowledge, respectively.
ClaudeFable 5.2
79.4%69.2%GPT-6.2Astra
78.8%54.4%Gemini 4Neon
76.4%54.0%Gemini 4Carbon
72.0%55.2%GPT-6.2Sol
69.0%47.6%ClaudeOpus 5.5
62.4%46.8%MuseWonder
43.2%26.2%ClaudeSonnet 5.5
42.4%23.8%KimiK3.1 Pro
33.6%25.4%Grok4.7
31.2%14.4%
All models are evaluated with reasoning high, capped at 128k output tokens.
Evaluation with tools. HLE-Diamond questions are designed to be answerable in a closed-book setting, testing both reasoning and expert knowledge. Since HLE is also used to evaluate the capabilities of agentic systems, we outline our recommended settings for evaluating HLE-Diamond with tools here.
GPT-6Astra
59.9%82.9%ClaudeOpus 5.5
54.6%73.9%GPT-6.1Sol
53.2%69.1%ClaudeFable 5.1
51.6%72.3%ClaudeOpus 5
38.1%68.9%GPT-6Sol
32.8%65.0%MuseSpark 1.3
24.6%55.4%
All models are evaluated with reasoning high.
Acknowledgement. We extend our deepest gratitude to all participating question contributors and experts involved in creating and refining the dataset, and to the researchers whose inputs across HLE-Rolling shaped HLE-Diamond.
For any inquiries, please contact agibenchmark@safe.ai.
Citation
@article{phan2025lastexam,
title = {A benchmark of expert-level academic questions to assess {AI} capabilities},
author = {{Center for AI Safety} and {Scale AI} and {HLE Contributors Consortium}},
journal = {Nature},
volume = {649},
pages = {1139--1146},
year = {2026},
doi = {10.1038/s41586-025-09962-4},
eprint = {2501.14249},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2501.14249}
}