How UK AISI And EvalEval Support Repeatable AI Benchmark Testing
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: How UK AISI And EvalEval Support Repeatable AI Benchmark Testing on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

The UK AI Security Institute is publishing selected benchmark results through EvalEval’s Evaluation Cards, which pair results with verification, context and configuration information. The release covers five benchmarks across six frontier models, plus two cyber evaluations with a different, partly overlapping model set.

The UK AI Security Institute (AISI) is publishing selected AI benchmark results through EvalEval’s Evaluation Cards, adding verification, evaluation context and configuration details to help readers understand how scores were produced, as detailed in the original analysis. The release accompanies AISI’s paper on inference-time compute and evaluation protocols and includes results for five benchmarks across six frontier models.

The paper, How Inference Compute Shapes Frontier LLM Evaluation, reports results for HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0. The six models named for these main experiments are Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4. AISI has also shared results from Cyber CTFs and The Last Ones, two cyber evaluations that use a different model set, overlapping only partly with the main experiments.

EvalEval says the cards include verified results, evaluation context and configuration information. Its format brings benchmark metadata, evaluation-run data and model metadata into a common structure. The release covers publicly reported AISI methods and findings where appropriate; the announcement does not say that it includes every AISI evaluation or all underlying transcripts.

AISI’s paper examines how scores vary with inference-time compute and evaluation protocol. In its Humanity’s Last Exam analysis, the researchers track the cumulative share of attempted tasks solved within a given token count, counting each task’s earliest observed success. In runs where models received correctness feedback from an oracle after each attempt, they solved additional tasks as token use increased. This result illustrates that a reported score can depend on the conditions of the run.

At a glance
reportWhen: Current release; no record-by-record pu…
The developmentAISI has begun sharing selected results from its evaluation research in EvalEval’s structured Evaluation Cards format.
At a glance
reportWhen: Announced in the EvalEval Coalition’s r…
The developmentAISI is using EvalEval’s open Evaluation Cards platform to publish evaluation results with details intended to make them easier to inspect and reproduce.

Why Evaluation Setup Matters

Benchmark scores are often compared across models as though they measure performance under identical conditions. But compute budgets, feedback rules and other protocol choices can change what a score represents. A result shown alongside its setup gives researchers, model developers and policy teams more information for interpreting comparisons and identifying runs that only appear equivalent.

The cards make selected details easier to inspect and may help others reproduce or compare published evaluations. They do not, on their own, establish which benchmark or protocol is best, or show that every result has been independently reproduced. Their value depends on the detail and consistency of the records contributors make available.

Roborock F25 GT Neo Wet/Dry Vacuum Mop, Cordless Vacuum Cleaner, 180° Flat

Roborock F25 GT Neo Wet/Dry Vacuum Mop, Cordless Vacuum Cleaner, 180° Flat

Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black

Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black

As an affiliate, we earn on qualifying purchases.

How AISI and EvalEval Built the Format

The release follows earlier work between AISI and EvalEval that began at a joint workshop alongside NeurIPS 2025. EvalEval says feedback from AISI helped shape Every Eval Ever (EEE), its shared schema for documenting evaluations. Evaluation Cards use related infrastructure to present results alongside benchmark and model information.

AISI has also worked on evaluation efficiency through OptStop, statistical rigor through HiBayES, and standardisation in transcript analysis and capability elicitation. The current release applies shared reporting infrastructure to selected, publicly reported AISI methods and findings. It addresses a practical difficulty: results published in different formats may lack details needed to interpret them, while repeating costly evaluations may not always be feasible.

“AISI is using EvalEval’s infrastructure to openly share evaluation results, supporting more reproducible and verifiable evaluation science.”

— EvalEval Coalition

Coverage and Reproduction Details

The announcement does not specify how many records or transcripts are available, which setup fields appear for every benchmark, or whether outside researchers have independently reproduced the results. It also does not enumerate the model set used in the two cyber evaluations. Readers should not assume that the six models in the main experiments apply to those tests.

The release does not describe a process for resolving disagreements between results reported under different protocols, or give dates for individual records. It says selected public methods and findings are shared where appropriate, so the cards should not be treated as a complete archive of AISI’s evaluation work.

Wider Use of the EEE Schema

EvalEval says it expects to continue standardising and sharing evaluations with AISI and other evaluation organisations. Under the EEE schema, model developers can submit verified results and evaluation developers can report benchmark and run data. Researchers in evaluation, governance and policy can browse Evaluation Cards by benchmark or model.

No further release date or adoption milestone was specified. Broader use could make comparisons across studies easier, depending on how consistently and fully contributors document their evaluations.

Key Questions

What is AISI releasing?

AISI is sharing selected benchmark results through EvalEval’s Evaluation Cards, with verification, context and configuration information. The release accompanies its paper on inference-time compute and evaluation protocols.

Which benchmarks and models are included in the main experiments?

The five benchmarks are HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0. The named models are Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4.

Are the same six models used in the cyber evaluations?

Not necessarily. Cyber CTFs and The Last Ones use a different model set that overlaps only partly with the main experiments. The announcement does not list the full cyber-evaluation model set.

Why include evaluation context with benchmark scores?

Inference-time compute and protocol choices can affect measured performance. Context and configuration details help readers understand the conditions behind a score and judge whether it can be compared with another result.

Does this release include all AISI evaluations?

No such claim was made. The announcement describes selected publicly reported methods and findings being shared where appropriate; it does not say that every evaluation or underlying transcript is included.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Beyond Green Tests: GPT-6 Astra’s Review Of Claude’s Model Edit

An OpenAI model reviewed a Claude-led edit to a local AI model, finding reproducible test and provenance faults while leaving legal scores unchanged.

Breaking Down Astra: The Most Capable AI Model On The Market

An in-depth analysis of Astra, OpenAI’s latest AI model, highlighting its capabilities, limitations, and what sets it apart from competitors.

How SenseTime’s SenseNova U1.5 Leverages 8B-MoT For Advanced AI Vision

SenseTime announces SenseNova U1.5, an 8B-parameter unified vision-language model built on Mixture-of-Transformers, with open training code for transparency.

The Eye Over the City: How Wide-Area Motion Imagery Works — and Where It Goes Blind

An in-depth look at WAMI technology, its capabilities, limitations, and future integration with radar for city-wide surveillance.