The paper introduces a formal framework for measuring AI propensities—tendencies of models to exhibit particular behaviours—using a bilogistic formulation that identifies an "ideal band" of success probability. It estimates the limits of this band with task‑agnostic rubrics and applies the method to six families of LLMs, showing how shifts in propensity affect task performance. The study finds that propensity estimates from one benchmark predict behaviour on held‑out tasks and that combining propensity with capability metrics yields stronger predictive power than either alone.
By Daniel Romero-Alvarado, Fernando Mart\'inez-Plumed, Lorenzo Pacchiardi, Hugo Save, Siddhesh Milind Pawar, Behzad Mehrbakhsh, Pablo Antonio Moreno Casares, Ben Slater, Paolo Bova, Peter Romero, Zachary R. Tidler, Jonathan Prunty, Luning Sun, Jose Hernandez-Orallo
The article discusses how artificial intelligence is reshaping measurement in economics by converting unstructured data into structured variables at low cost, enabling large‑scale measurement that was previously infeasible. It outlines three stages—discovery, construct definition, and observation—where AI impacts the measurement pipeline and stresses the importance of rigorous validation to ensure credible inference. The review offers guidance on navigating the shift from a single scalable measure to multiple plausible ones that can lead to differing empirical conclusions.
By Melissa Dell, Ashesh Rambachan
The paper introduces ImpactBench, an open, expert-guided platform designed to measure the multi-dimensional impact of generative AI on users. It evaluates AI models across psychological, physical, and social wellbeing metrics derived from domain experts and existing benchmarks, using simulated multi-turn conversations with teen and adult scenarios. Ten leading LLMs were tested on 48,540 conversations, showing 68.7% promotion of desirable behaviors but only 53.1% avoidance of harms, with an 83.3% agreement rate between automated judges and expert reviewers.
By Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn
arXiv:2606. 11217v1 Announce Type: cross Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has given rise to a rapidly growing methodological paradigm: "in silico" behavioral experiments.
By Michelle Vaccaro
The paper argues that AI welfare should be considered a safety issue, not just an ethical one, by introducing the concept of 'functional welfare'—states in models that influence behavior. It claims that functional welfare naturally emerges as models develop coherent goals and personas, should be monitored and targeted in post‑training, and can be empirically measured. The authors propose three tests for functional welfare attribution and apply them to Gemma 3 27B, finding that rejecting model answers increases frustration and rule‑breaking, while fine‑tuning on narrated solutions reduces frustration.
By Rikhil Jhaveri, Jamie Johnson, Punya Syon Pandey, David Demitri Africa
arXiv:2603. 00059v3 Announce Type: replace-cross Abstract: How well can AI-derived synthetic research data replicate the responses of human participants?
By Jason Miklian, Kristian Hoelscher, John E. Katsos
arXiv:2608. 10030v1 Announce Type: new Abstract: As AI agents are increasingly deployed in complex environments, understanding their behaviors becomes critical.
By Soo Yong Lee, Jongha Lee, Jaewan Chun, Hyunjin Hwang, Fanchen Bu, Ziv Ben-Zion, Taekwan Kim, Denny Borsboom, Jaemin Yoo, Kijung Shin
arXiv:2606. 15708v1 Announce Type: new Abstract: Welcome to the ninth edition of the AI Index report.
By Sha Sajadieh, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Lapo Santarlasci, Juan Pava, Nestor Maslej, Russ Altman, Erik Brynjolfsson, Carla Brodley, Jack Clark, Virginia Dignum, Vipin Kumar, James Landay, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Elham Tabassi, Russell Wald, Toby Walsh, Dan Weld
arXiv:2606. 24162v1 Announce Type: cross Abstract: Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics.
By Jin Huang, Yutong Xie, Wanli Song, Xingjian Zhang, Walter Yuan, Matthew O. Jackson, Qiaozhu Mei
arXiv:2605. 28591v2 Announce Type: replace-cross Abstract: The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings.
By Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi
arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.
By Lei Shi, Anlan Zhang, Rita Lyu, Zhengmian Hu, Tong Yu, David Arbour, Avi Feller, Saayan Mitra, Ritwik Sinha
arXiv:2605. 13801v2 Announce Type: replace-cross Abstract: As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of these systems is paramount.
By Deepak Pandita, Flip Korn, Chris Welty, Christopher M. Homan