Hugging Face Blog
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
Hugging Face Trending Papers
Oct 22
VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act
VERA is an open‑source, self‑hosted dashboard designed to streamline responsible AI assessments for large language model deployments in regulated environments, such as a European bank. It interprets the EU AI Act into six principle groups and eighteen technical requirements, scoring twelve that current benchmarks can measure while also tracking energy consumption for environmental impact. A within‑subject study showed that using VERA cut assessment time from 56 to 30 seconds and received a high usefulness rating of 4.36 out of 5.
arXiv AI
Sep 3
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
EvalDetectBench is an open pipeline and benchmark designed to measure evaluation awareness in frontier large language models, enabling practitioners to test models against any Inspect-compatible evaluation. It includes a curated transcript suite from current frontier system-card evaluations and diverse deployment sources, and it assesses both how reliably models recognize they are being evaluated and how detectable individual benchmarks are. The benchmark addresses systematic bias by calibrating probes per model and harmonizing generator selection to correct for variance caused by model identity and prompt choice.
arXiv AI
Oct 21
Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims
arXiv:2610.27610v1 Announce Type: cross Abstract: Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluat...
arXiv AI
Aug 17
ASSERT: A Measurement Pipeline for GenAI Audits
arXiv:2608. 13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy.
arXiv AI
Aug 3
Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
arXiv:2607. 28631v1 Announce Type: new Abstract: AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery.
arXiv Computation and Language
6d ago
Beyond accuracy: what AI assurance needs to establish in public services
The paper examines how public bodies use AI-generated evidence—such as classifications and summaries—to inform policy and service delivery, highlighting the need for rigorous testing and governance. Using eight GPT‑4o configurations on 1,200 UK Prime Minister’s Questions, the authors find modest accuracy (68.2–70.3%) but significant label disagreement and variable human‑review flags (0.9–41.0%). Analysis of UK and EU policy documents shows that while testing and oversight are mandated, reporting requirements often miss key issues like changing answers, shared errors, and review routing, underscoring gaps in current AI assurance practices.