arXiv AI August 12, 2026 By Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie, Gianluca Demartini

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

arXiv:2608. 10385v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Aug 27

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

The study investigates how prior scores influence large language model (LLM) judgments in the LLM-as-a-Judge paradigm. By testing three prompt conditions—no metadata, revision framing, and anchored metadata containing prior scores—the authors find that prior scores systematically bias evaluations, shifting ratings toward those scores across 192,000 attempts. The bias also affects categorical decisions, blocking 48% of error corrections and flipping 10.18% of correct judgments, and is not mitigated by Chain-of-Thought or a warning, underscoring the need for careful context engineering.

By Ante Kapetanovic, Kemal Altwlkany, Andro Mercep, Tomislav Duricic, Emanuel Lacic

llms safety

arXiv AI

Sep 16

Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening

arXiv:2609.16517v1 Announce Type: cross Abstract: Resume screeners must infer job-relevant competence from resumes whose presentation can vary substantially in wording, structure, stylistic polish, a...

By Qiangju Chen, Yang Xiao

llms

arXiv AI

Jul 10

Persona Cartography: Charting Language Model Personality Traits in Weight Space

arXiv:2607. 07916v1 Announce Type: new Abstract: Large language models exhibit recurring behavioural patterns -- personas -- that shape generalisation and safety, but we lack reliable tools for decomposing, measuring, and controlling them.

By Luke Baines, Anton Gonzalvez Hawthorne, Mariia Koroliuk, Irakli Shalibashvili, Cl\'ement Dumas, Konstantinos Voudouris, David Demitri Africa

llms benchmarks safety

arXiv AI

Jun 3

Decomposing and Measuring Evaluation Awareness

arXiv:2605. 23055v2 Announce Type: replace-cross Abstract: Frontier language models sometimes recognize that they are being evaluated and adjust their behavior, undermining validity of benchmark results.

By Changling Li, Terry Jingchen Zhang, Jie Zhang, Zhijing Jin, Sahar Abdelnabi, Maksym Andriushchenko

llms benchmarks safety

arXiv AI

Jul 10

Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring

arXiv:2604. 07102v2 Announce Type: replace-cross Abstract: Activation-based steering enables inference-time personalization of large language models, but its effects in educational applications are not well understood.

By Yongchao Wu, Aron Henriksson

llms benchmarks

Hugging Face Trending Papers

Sep 15

Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening

Resume screeners must infer job-relevant competence from resumes whose presentation can vary substantially in wording, structure, stylistic polish, and document extraction quality. Ideally, such surfa...

llms

One email a morning, machine-written

Related stories

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening

Persona Cartography: Charting Language Model Personality Traits in Weight Space

Decomposing and Measuring Evaluation Awareness

Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring

Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening