Introducing HLE-Diamond

We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE) question collection, following a year-long process of cleaning and refinement with input from research communities.

HLE-Diamond consists of 1,000 questions.

Main results. We compare the performance of current models on HLE-Diamond without tools.

All models are evaluated with reasoning high. See reasoning max

HLE-Diamond

• GPT-6Astra
  59.9%
• Gemini4 Argon
  56.9%
• ClaudeOpus 5.5
  54.6%
• GPT-6.1Sol
  53.2%
• ClaudeFable 5.1
  51.6%
• MuseWonder
  34.7%
• ClaudeSonnet 5.5
  33.1%
• Grok4.7
  22.8%

Token Usage

Reasoning high

• GPT-6.1Sol
  4.5k
• GPT-6Astra
  5.2k
• ClaudeSonnet 5.5
  8.6k
• ClaudeOpus 5.5
  9.6k
• ClaudeFable 5.1
  14.8k
• MuseWonder
  16.1k
• Gemini4 Argon
  26.1k
• Grok4.7
  43.2k

Dataset. HLE-Diamond consists of 500 reasoning and 500 knowledge questions, measuring reasoning and expert knowledge, respectively.

Reasoning and Knowledge Partitions

ReasoningKnowledge

• GPT-6Astra
  74.2%

  45.6%
• Gemini4 Argon
  61.8%

  52.0%
• ClaudeOpus 5.5
  62.4%

  46.8%
• GPT-6.1Sol
  67.4%

  39.0%
• ClaudeFable 5.1
  61.8%

  41.4%
• MuseWonder
  43.2%

  26.2%
• ClaudeSonnet 5.5
  42.4%

  23.8%
• Grok4.7
  31.2%

  14.4%

All models are evaluated with reasoning high. See reasoning max

Evaluation with tools. HLE-Diamond questions are designed to be answerable in a closed-book setting, testing both reasoning and expert knowledge. Since HLE is also used to evaluate the capabilities of agentic systems, we outline our recommended settings for evaluating HLE-Diamond with tools here.

HLE-Diamond with tools

Without toolsWith tools (web+code)

• GPT-6Astra
  59.9%

  82.9%
• ClaudeOpus 5.5
  54.6%

  73.9%
• GPT-6.1Sol
  53.2%

  69.1%
• ClaudeFable 5.1
  51.6%

  72.3%
• ClaudeOpus 5
  38.1%

  68.9%
• GPT-6Sol
  32.8%

  65.0%
• MuseSpark 1.3
  24.6%

  55.4%

All models are evaluated with reasoning high.

Acknowledgement. We extend our deepest gratitude to all participating question contributors and experts involved in creating and refining the dataset, and to the researchers whose inputs across HLE-Rolling shaped HLE-Diamond.

For any inquiries, please contact agibenchmark@safe.ai.

@article{phan2025lastexam,
      title = {A benchmark of expert-level academic questions to assess {AI} capabilities},
      author = {{Center for AI Safety} and {Scale AI} and {HLE Contributors Consortium}},
      journal = {Nature},
      volume = {649},
      pages = {1139--1146},
      year = {2026},
      doi = {10.1038/s41586-025-09962-4},
      eprint = {2501.14249},
      archivePrefix = {arXiv},
      primaryClass = {cs.LG},
      url = {https://arxiv.org/abs/2501.14249}
}