arXiv AI

AgentFairBench: Do LLM Agents Discriminate When They Act?

arXiv:2606. 16723v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers.

arXiv AI

Aug 11

In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores

arXiv:2605. 12530v2 Announce Type: replace-cross Abstract: LLM fairness should be evaluated through in-situ behavioral pattern rather than standardized-test Q&A benchmarks.

By Zeyu Tang, Sang T. Truong, Deonna Owens, Shreyas Sharma, Yibo Jacky Zhang, Brando Miranda, Sanmi Koyejo

llms agents benchmarks safety

arXiv Machine Learning

Jun 5

Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability

arXiv:2605. 03217v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning, yet existing bias evaluations treat model outputs as simply "biased" or "unbiased.

By Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur

llms efficiency safety

arXiv AI

Aug 3

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

arXiv:2607. 28934v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on characteristics like race and gender.

By Martin Lukk (University of Toronto)

llms benchmarks safety

arXiv AI

Jun 17

Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.

By Joel Persson, M{\aa}rten Schultzberg, Sebastian Ankargren

llms safety

arXiv AI

22h ago

Delegation Asymmetry in Agentic Recommender Systems: Measuring Two-Sided Receptivity in Online Dating

arXiv:2608. 18058v1 Announce Type: new Abstract: Autonomous LLM agents that converse on a user's behalf are an emerging design pattern in matching platforms, yet their viability depends on a condition rarely examined: users must accept not only delegating conversation to an agent, but also receiving agent-mediated communication from others.

By Daria Leshchikova, Valentina V. Kuskova, Dmitry Zaytsev, Valerii Klimov

llms agents

arXiv AI

Jun 30

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

arXiv:2411. 10109v3 Announce Type: replace Abstract: Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes.

By Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, Michael S. Bernstein

llms agents benchmarks

arXiv AI

Jul 22

Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling

arXiv:2607. 18310v1 Announce Type: cross Abstract: Synthetic-population tools increasingly run every individual as an independent large language model (LLM) agent.

By Gurkan Ozkan

llms agents

arXiv AI

2d ago

Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

arXiv:2608. 14399v1 Announce Type: cross Abstract: Patients increasingly ask large language model (LLM) assistants which doctor to see, making these systems AI infomediaries: algorithms that intermediate one person's choice among other people and thereby decide, silently and at scale, which physicians become visible.

By Syeda Anshrah Gillani, Mirza Samad Ahmed Baig

llms

arXiv AI

Aug 5

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

arXiv:2608. 03744v1 Announce Type: new Abstract: Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace.

By Sebasti\'an Andr\'es Cajas Ord\'o\~nez, Agastya Munnangi, Aldo Marzullo, Felipe Ocampo Osorio, Quang Bui, Mohammad Shahin, Armaan Grewal, Emmanuel Paul Kwesiga, Anqi Peter Li, Josephine Nanyonjo, Aaditya Panchal, Arshnoor Bhutani, Nikhil Jaiswal, Milit S. Patel, Maximin Lange, Leo Anthony Celi

llms agents benchmarks

arXiv AI

1d ago

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

arXiv:2608. 14606v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level.

By Mantas Lukauskas, Viktorija \v{S}arkauskait\.e

llms benchmarks

arXiv AI

Jul 10

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.

By Kaihua Ding

llms safety

arXiv AI

Aug 10

Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints

arXiv:2608. 06949v1 Announce Type: new Abstract: Prior benchmarking work has shown that a single large language model (LLM), forced to make life-or-death resource-allocation decisions, exhibits measurable demographic bias.

By Paul-Peter Arslan

llms agents benchmarks safety

Related stories