The paper outlines a roadmap for advancing pluralistic value alignment in AI, noting that current research lacks evidence of influence on model training or evaluation. It identifies three key barriers: a need for empirical studies on benefits and drawbacks, clearer specifications of when and how pluralistic behavior should occur, and robust evaluation methods suitable for production. The roadmap aims to guide research toward practical deployment in responsible AI systems.
By Elinor Poole-Dayan, Jillian Fisher, Jacob Andreas, Mitchell Gordon, Michiel A. Bakker
The paper proposes a method for assessing AI suitability for workplace tasks by profiling both AI agents and tasks along shared cognitive capability dimensions. It uses benchmark performance to infer agent capabilities and expert input to weight task requirements, allowing independent updates as models and roles evolve. The authors validate the approach on synthetic agents, six AI systems, and 410 employees across six domains, showing that AI systems vary more by cognitive dimension than by model family, while workplace tasks share a common cognitive core.
By Jonathan Prunty, Marko Te\v{s}i\'c, Patrick Quinn, Jos\'e Hern\'andez-Orallo, Lucy Cheke
The paper "Reliability Engineering for AI Systems: Challenges, Methods, and Directions" discusses how AI reliability extends beyond correct outputs to include consistent operation of retrieval, memory, tool use, permissions, human oversight, and inter-system interactions, especially for generative AI. It adapts traditional reliability engineering techniques—such as failure definitions, operational envelopes, FMEA, accelerated testing, field monitoring, and reliability growth—to AI, proposing a four-level diagnostic framework and a TEVV (Test, Evaluation, Verification, Validation) approach. The authors also introduce the SMART framework for statistical guidance and reference NIST’s AI risk management framework, illustrating their concepts with three case studies and emphasizing the urgent need for new measurements and safety guardrails as AI systems evolve rapidly.
By Rong Pan, Yili Hong, Min Xie
AutoBCI is an agentic framework that discovers and selects EEG decoding architectures across diverse tasks such as emotion recognition, motor imagery, and sleep staging. It employs a Designer Agent that performs Pool‑Guided Architecture Discovery (PGAD) and a Forecaster Agent that uses Performance Estimation from Early Knowledge (PEEK) to predict full‑budget validation performance from early learning curves. Evaluated on 14 EEG datasets with six large language models, AutoBCI’s best architecture (Claude Opus 5.5) achieves 64.16% average test balanced accuracy, slightly surpassing the strongest baseline, and PEEK reduces prediction error by 38.1% compared to the best‑observed‑score baseline.
By Muyun Jiang, Yi Ding, Wei Zhang, Jinbo Chen, Chenyu Liu, Zhenjie Yang, Yuxin Li, Jingyuan Chen, Yuhao Lu, Yong Li, Shuailei Zhang, Cuntai Guan
MASBench is a benchmark for evaluating large language model (LLM)-based multi‑agent systems (MAS) under partial observability, a realistic constraint where agents only see limited environmental information. The benchmark is structured into three progressive task categories—Reasoning, Scheduling, and Game—and assesses three collaboration mechanisms: Protocol, Memory, and Routing. It provides deterministic evaluation metrics such as performance score, communication cost, and cost effectiveness, and includes empirical results across various LLM backbones to guide effective MAS design.
By Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu, Chen Qian, Cheng Yang, Chuan Shi, Zhiyuan Liu
AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.
By Shu Zhou, Ziyang He, Yuxin Rao, Yudong Fang, Jiali Qian, Dongyuan Chen, Juntao Zhang, Yong Hu
ScamWorld is an interactive environment that recreates reported scam websites across various scenarios, fraud mechanisms, and languages, allowing researchers to observe fraud indicators before a web agent’s final action. The platform enables controlled, repeatable evaluation of web agents, which are increasingly used to browse, shop, and manage accounts on the open web. A study of six leading LLM-powered agents revealed deception rates between 11% and 35%, with agents more likely to proceed with payments than to provide credentials or wallet information, and some agents continuing after recognizing suspicious conditions.
By Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma
The paper introduces RADT, a decision transformer for offline, reward‑free, goal‑conditioned reinforcement learning that can handle arbitrary avoid‑regions specified at evaluation time. By encoding goals and avoid‑regions as prompt tokens and using an avoid‑region hindsight relabeling strategy, RADT learns reach‑avoid behavior from suboptimal random trajectories without reward or cost supervision. Across 17 tasks and environments, RADT outperforms existing offline goal‑conditioned RL models, achieving significant zero‑shot generalization to novel avoid‑region sizes and counts, and is also applied to a biological cell reprogramming task.
By Kevin Li, Marinka Zitnik
Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been...
We introduce LifeVal, a benchmark evaluating AI model performance on personalized agentic consumer tasks. LifeVal covers the most valuable life tasks that a personalized digital superassistant could a...
The article announces the release of datasette-llm version 0.1a10. It reports a bug fix that corrects the demo site at agent.datasette.io, ensuring that only LLM models specified in the configuration are available. The update addresses a configuration issue affecting model availability.
Unmanned aerial vehicle (UAV)-based collaborative perception enables multiple agents to share complementary perceptual information, overcoming limitations such as occlusion and limited sensing range....
Resource management for distributed edge networks hosting different services has been handled efficiently using Reinforcement learning (RL). Reward generalization in complex workflows is still an open...
Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression,...
AI's inflection point is a fountain of alarming risks. Digital platforms in particular are vulnerable to three major AI risks, as those in control of global platforms cede control to AI, of the algori...
Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's sa...
OpenAI developed an autonomous pipeline that identified, validated, and produced patches for 948 local privilege‑escalation vulnerabilities in the Linux kernel. The system automatically discovered the issues, confirmed their validity, and generated the necessary code fixes. This demonstrates a large-scale, automated approach to kernel security remediation.
IndicDBBench is a new multilingual Text-to-SQL benchmark that evaluates cross‑lingual semantic parsing across 20 databases and 237 tables in seven languages, including English and six Indic languages. The benchmark uses real‑world open‑data schemas from NDAP and IDP, and a three‑agent pipeline to create complex relational structures with high relational density and join depths up to six. Evaluation of state‑of‑the‑art models shows a 9‑point drop in execution accuracy from English to Indic languages, revealing an "Indic Gap" that persists even when gold tables and literal values are supplied.
By Aviral Dawar, Roshan Karanth, Vikram Goyal, Rakhee Kulshrestha, Sumanta Pasari, Dhruv Kumar
ElectroChemLLM is a multimodal large language model pipeline that extracts structured electrochemical records—such as electrolyte compositions, electrode materials, cell configurations, and measured physicochemical properties—from unstructured scientific PDFs. By combining hybrid document parsing with vision-capable LLM orchestration, it captures data across text, tables, and figures, achieving an F‑1 score of 0.97 on a benchmark of 232 battery‑electrolyte papers. The extracted data populate ElectroChemDB, a database of 15,000 research articles, which is then used to train machine learning models for electrolyte property prediction, demonstrating a full workflow from literature to materials discovery.
By Dikshya Mohanty, Manoj Praveen Nandigama, Kuldeepsinh Raj, Sourav Kanti Maiti, Jiannan Yang, Ruiqi Luo, Srinivasa Phani Madhav Marupudi, Mounika Geriki, Prayag Raghunandan, Puneet Lingutla, Tengfei Ma, Niranjan Balasubramanian, Nav Nidhi Rajput
MemRole is a framework that explicitly models the memory-to-decision process for large language model (LLM) role‑playing agents, enabling them to make role‑consistent decisions over long narrative horizons. It introduces Long‑Horizon Memory Thinking (LHMT), a reasoning schema that decomposes hidden reasoning into context understanding, memory activation, and response strategy planning, and is trained on a novel dataset that limits each character’s memory pool to what it could plausibly know at the current narrative point. The framework also includes Self‑Refining On‑Policy Self‑Distillation (SR‑OPSD), where the model critiques its own rollouts and distills token‑level guidance back into the policy, resulting in a 15.2‑point improvement over its backbone and competitive performance with frontier LLMs such as GPT‑5.4.
By Mengting Li, Xilong Cheng, Zhihao Xiao, Zhenhua Xu, Jingyuan Zhou, Linfeng Li, Long Zhang, Borui Cai, Yunxiao Qin