The paper introduces the AI Assessment Sandbox Configurator, an open‑source framework designed to support technical assessment in AI Regulatory Sandboxes (AIRS) mandated by the EU Artificial Intelligence Act. It outlines 11 architectural and governance requirements for infrastructure that enables large‑scale, structured technical testing, and presents a catalogue of tests, a shared data model, dashboards, and reporting tools that harmonise heterogeneous outputs. An early‑stage pilot demonstrated the framework’s harmonisation and reporting capabilities within a live AIRS engagement, contributing to an official Exit Report.
By Alessio Buscemi, German Castignani, Daniele Pagani, Maxime Cordy, Jordi Cabot
EviStreams is a live, open‑source, no‑code web platform that enables systematic review teams to control AI‑assisted data extraction at three stages: program design, field specification, and extracted predictions. Reviewers use a form builder to define typed fields, run extraction on PDFs, inspect AI‑generated values with supporting passages, and perform blinded dual review with adjudication to produce an auditable consensus export. An evaluation across four clinical corpora and three model families shows that extraction quality depends more on field specification than on the model choice.
By Sai Karthik Kosuri, Ankita Shashikant Bhosale, Michael Glick, Alonso Carrasco-Labra, Chris Callison-Burch
arXiv:2602. 18446v2 Announce Type: replace-cross Abstract: Users increasingly rely on Large Language Models (LLMs) for Deep Research, using them to synthesize diverse sources into structured reports that support understanding and action.
By Jujia Zhao, Zhaoxin Huan, Zihan Wang, Xiaolu Zhang, Jun Zhou, Suzan Verberne, Zhaochun Ren
arXiv:2606. 14516v1 Announce Type: new Abstract: AI evaluations are widely used for testing and understanding progress.
By Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall, Tommaso Cerruti, Yanan Long, Yifan Mai, Sanchit Ahuja, Asaf Yehudai, Marek \v{S}uppa, John P. Lalor, Oluwagbemike Olowe, Jatin Ganhotra, Brian H. Hu, Eliya Habba, Andrew M. Bean, Chang Liu, Sander Land, Steven Dillmann, Aniketh Garikaparthi, Elron Bandel, Saki Imai, James Edgell, Wm. Matthew Kennedy, Jenny Chim, Patrick Meusling, Asteria Kaeberlein, Venkata Ramachandra Karthik Chundi, Manasi Patwardhan, Martin Ku, Austin Meek, Leon Knauer, Brian Wingenroth, Srishti Yadav, Usman Gohar, Felix Friedrich, Michelle Lin, Jennifer Mickel, Arman Cohan, Stella Biderman, Irene Solaiman, Zeerak Talat, Anka Reuel, Mubashara Akhtar, Gjergji Kasneci, Avijit Ghosh, Leshem Choshen
Benchmark Radar is a living database and search engine that aggregates AI benchmark papers, datasets, code, and score histories. It automatically discovers new benchmark resources from 37 sources, maintains a catalog of 1,283 records with 12,916 numeric observations, and provides tools such as a web dashboard, CLI, and downloadable evidence for researchers. The system also offers visualizations like a Pareto frontier and trend views to help users assess benchmark saturation and adoption.
By Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu
arXiv:2609.21841v1 Announce Type: new
Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically val...
By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid
PaperDoctor is an agent framework that provides evidence‑grounded, actionable feedback for scientific papers before submission. It evaluates writing, layout, references, code, theory, prior work, and experiments through a three‑layer hierarchical system, linking each critique to specific evidence and revision suggestions. The system selectively rebuilds and reruns experiments to uncover reproducibility gaps, and an interactive interface lets authors explore findings tied to their manuscript.
By Kevin Qinghong Lin, Siyuan Hu, Pan Lu, Yu Chen, Yanzhe Chen, Owen Queen, Yupeng Chen, Jialin Yu, Junchi Yu, Zifeng Ding, Yuanfeng Ji, Sheng Liu, Jindong Gu, Linjie Li, Mike Zheng Shou, Philip Torr, James Zou
arXiv:2608. 13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy.
By Riccardo Fogliato, Abhinav Palia, Xiawei Wang, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Nicholas Pangakis, Sharman Tan, Dan Vann, Hannah Washington, P. Alex Dow, Heba Elfardy, Hanna Wallach, Sandeep Atluri
arXiv:2606. 07936v1 Announce Type: cross Abstract: Human evaluation plays a critical role in assessing the quality of generated text.
By Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett, Lucy Lu Wang
EvalDetectBench is an open pipeline and benchmark designed to measure evaluation awareness in frontier large language models, enabling practitioners to test models against any Inspect-compatible evaluation. It includes a curated transcript suite from current frontier system-card evaluations and diverse deployment sources, and it assesses both how reliably models recognize they are being evaluated and how detectable individual benchmarks are. The benchmark addresses systematic bias by calibrating probes per model and harmonizing generator selection to correct for variance caused by model identity and prompt choice.
By Xinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk
arXiv:2609.05947v1 Announce Type: new
Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
By Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan, Lei Chen, Kui Ren
arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.
By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang