Humanity's Last Exam
HLE-Diamond Logo

Introducing HLE-Diamond

Center for AI Safety&Scale AI
Hugging FaceDatasetload_dataset("cais/hle-diamond")

Center for AI Safety and Scale AI

We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE) question collection, following a year-long process of cleaning and refinement with input from research communities.

HLE-Diamond consists of 1,000 questions.

Main results. We compare the performance of current models on HLE-Diamond without tools.

All models are evaluated with reasoning high, capped at 128k output tokens.

HLE-Diamond
  • ClaudeFable 5.2
    74.3%
  • GPT-6.2Astra
    66.6%
  • Gemini 4Neon
    65.2%
  • Gemini 4Carbon
    63.6%
  • GPT-6.2Sol
    58.3%
  • ClaudeOpus 5.5
    54.6%
  • MuseWonder
    34.7%
  • ClaudeSonnet 5.5
    33.1%
  • KimiK3.1 Pro
    29.5%
  • Grok4.7
    22.8%

Token Usage

Reasoning high

  • GPT-6.2Astra
    3.3k
  • GPT-6.2Sol
    3.7k
  • ClaudeSonnet 5.5
    8.6k
  • ClaudeFable 5.2
    8.8k
  • ClaudeOpus 5.5
    9.6k
  • KimiK3.1 Pro
    9.8k
  • MuseWonder
    16.1k
  • Gemini 4Neon
    38.3k
  • Grok4.7
    43.2k
  • Gemini 4Carbon
    59.2k

Dataset. HLE-Diamond consists of 500 reasoning and 500 knowledge questions, measuring reasoning and expert knowledge, respectively.

Reasoning and Knowledge Partitions
ReasoningKnowledge
  • ClaudeFable 5.2
    79.4%
    69.2%
  • GPT-6.2Astra
    78.8%
    54.4%
  • Gemini 4Neon
    76.4%
    54.0%
  • Gemini 4Carbon
    72.0%
    55.2%
  • GPT-6.2Sol
    69.0%
    47.6%
  • ClaudeOpus 5.5
    62.4%
    46.8%
  • MuseWonder
    43.2%
    26.2%
  • ClaudeSonnet 5.5
    42.4%
    23.8%
  • KimiK3.1 Pro
    33.6%
    25.4%
  • Grok4.7
    31.2%
    14.4%

All models are evaluated with reasoning high, capped at 128k output tokens.

Evaluation with tools. HLE-Diamond questions are designed to be answerable in a closed-book setting, testing both reasoning and expert knowledge. Since HLE is also used to evaluate the capabilities of agentic systems, we outline our recommended settings for evaluating HLE-Diamond with tools here.

HLE-Diamond with tools
Without toolsWith tools (web+code)
  • GPT-6Astra
    59.9%
    82.9%
  • ClaudeOpus 5.5
    54.6%
    73.9%
  • GPT-6.1Sol
    53.2%
    69.1%
  • ClaudeFable 5.1
    51.6%
    72.3%
  • ClaudeOpus 5
    38.1%
    68.9%
  • GPT-6Sol
    32.8%
    65.0%
  • MuseSpark 1.3
    24.6%
    55.4%

All models are evaluated with reasoning high.

Acknowledgement. We extend our deepest gratitude to all participating question contributors and experts involved in creating and refining the dataset, and to the researchers whose inputs across HLE-Rolling shaped HLE-Diamond.

For any inquiries, please contact agibenchmark@safe.ai.

Citation

@article{phan2025lastexam,
      title = {A benchmark of expert-level academic questions to assess {AI} capabilities},
      author = {{Center for AI Safety} and {Scale AI} and {HLE Contributors Consortium}},
      journal = {Nature},
      volume = {649},
      pages = {1139--1146},
      year = {2026},
      doi = {10.1038/s41586-025-09962-4},
      eprint = {2501.14249},
      archivePrefix = {arXiv},
      primaryClass = {cs.LG},
      url = {https://arxiv.org/abs/2501.14249}
}