arXiv Machine Learning
Jun 9
Decoy-Calibrated Failure Audits for Language Models
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration
arXiv:2607. 14399v1 Announce Type: new Abstract: Evaluations of language-model honesty read the model's verdicts as evidence about the model.
Related stories
arXiv AI
2d ago
Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
arXiv:2608. 13564v1 Announce Type: new Abstract: Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time.
arXiv Machine Learning
Jul 8
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.
arXiv AI
Jul 21
Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA
arXiv:2607. 08731v2 Announce Type: replace-cross Abstract: National language models are becoming publicly funded epistemic infrastructure.
arXiv AI
Jul 23
Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
arXiv:2607. 20379v1 Announce Type: new Abstract: Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it.
arXiv Machine Learning
Jun 25
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
arXiv:2606. 25449v1 Announce Type: cross Abstract: A language model's memory can be worse than having no memory at all.
arXiv AI
1d ago
What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.
arXiv AI
Aug 7
Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents
arXiv:2608. 06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries.
Hugging Face Trending Papers
Jun 24
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
A language model's memory can be worse than having no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it, and it emits that stale value as a confident answer; give the same model an empty memory and it abstains.
arXiv AI
Aug 6
Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary
arXiv:2608. 04240v1 Announce Type: cross Abstract: Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike.
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv AI
2d ago
Split the Labor: Separating Evidence Interpretation from Decision Aggregation
arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.