arXiv:2607. 28934v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on characteristics like race and gender.
By Martin Lukk (University of Toronto)
The study examines how the design of audit questions influences perceived bias in large language models (LLMs). Using 40,726 requests across five models and three domains—hiring, lending, and medical triage—the authors find that demographic bias effects are not replicated when the audit is standardized. Instead, the audit’s construction, such as question phrasing and ordering, has a stronger impact on model responses than applicant demographics.
By Siddharth Vohra, Manikandan Ravikiran
The paper proposes a scalable, automated method for auditing candidate‑job matching systems for demographic bias. It employs large‑language‑model agents to generate neutral resumes, injects controlled demographic variations, ranks candidates with a fine‑tuned embedding model, and evaluates nine fairness metrics across counterfactual, group‑fairness, and merit‑aware families, producing a composite risk report. Experiments on a small corpus show that single‑score audits miss nuanced issues, underscoring the need for multi‑metric evaluation and LLM‑generated audits as a low‑cost complement to human reviews.
By Sai Yashwant, Shruti Bansal, Anurag Dubey, Samaroha Chatterjee, Satyam Kumar, Shreyash Gupta, Gantala Thulsiram
arXiv:2606. 30219v1 Announce Type: new Abstract: LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify.
By Bu\u{g}ra Alperen Ulu{\i}rmak, Rifat Kurban
The paper introduces an evidence‑referenced audit for large language models that assesses how much predictive information proxy fields contribute, comparing model reliance to a known outcome rule and a Bayesian linear regression. Applied to a simulated patient‑ranking task, the study finds that models rely on non‑informative fields, grow less reliance as proxies become predictive, are sensitive to display format, and that ranking accuracy does not reliably indicate proxy reliance. The authors argue that audits should report reliance against explicit predictive references alongside accuracy to properly evaluate model behavior.
By Zengqing Wu, Chuan Xiao
arXiv:2610.02492v1 Announce Type: new
Abstract: LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreem...
By Harry Lyu, Neil Thompson