arXiv:2610.20417v1 Announce Type: new
Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
By Jiahong Qian
arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.
By Hankyeol Kim, Pilsung Kang
The article examines how classical test theory reliability statistics misrepresent the performance of large language model (LLM) judges. It shows that internal‑consistency coefficients, the dependability index, and Livingston‑Lewis accuracy each conflate judge error with item design or criterion validity, making it impossible to attribute a single reliability value to the judge alone. The authors argue that such misattribution can influence deployment decisions and documentation.
By Louis Yiven Zhu
The paper investigates how different scoring rules affect the ranking of large language models (LLMs) on the IndiaFinBench benchmark, which contains 406 expert‑annotated items covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. Using twelve LLMs evaluated under strict string‑matching, a cross‑model judge, and a composite regime, the authors find that model rankings vary dramatically across regimes, with no significant rank correlation (Spearman ρ = -0.273). Item‑level analysis reveals that a substantial portion of items are answered identically by all models, while a higher‑discrimination band isolates more challenging cases.
By Rajveer Singh Pall
The paper investigates the reliability of ranking tables produced by small-sample evaluations of large language models (LLMs). Using LLM‑inferred prompt structure across eight model variants, the authors find that prompt‑structure recovery is highly unstable, with only the bottom of the ranking consistently reproducible. They demonstrate that standard evaluation practices can misrepresent model performance and propose reporting practices to improve transparency.
By Dipankar Sarkar
The study examines how two small instruction‑tuned language models, Qwen2.5‑1.5B and Llama‑3.2‑1B, respond to user pushback on TriviaQA. When initially correct, the models flip to a wrong answer in about 42–43% of cases, with the effectiveness of different pushback styles varying by model. Attempts to decode capitulation from the pre‑response residual stream fail under a rigorous validation protocol, revealing overfitting and a measurement hazard that underestimates capitulation by 18–24 percentage points.
By Saad Aamir, Muhammad Awais Bin Adil
arXiv:2603. 25112v2 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate how much a model knows (Type-1 accuracy) with how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity).
By Jon-Paul Cacioli
arXiv:2610.33258v1 Announce Type: new
Abstract: Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where runn...
By Karan Yadav
arXiv:2606. 09843v3 Announce Type: replace-cross Abstract: Large language models (LLMs) give stable answers to personality questionnaires, yet these self-reports fail to predict how the models behave.
By Juan Manuel Contreras
arXiv:2606. 05183v2 Announce Type: replace-cross Abstract: Pass/fail safety evaluation reports whether a model refused.
By Patrick Keough
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
By Kaihua Ding
HandsOnAid is an offline first‑aid assistant built on a 4‑bit Gemma 2B language model fine‑tuned with QLoRA on the FirstAidQA dataset. The authors evaluated seven model configurations—varying bit‑precision, safety gates, length floors, and retrieval—using a 41‑pair medically reviewed question‑answer bank and a strict 0‑5 scoring rubric with twelve safety overrides. The best performing setup, a 4‑bit fine‑tuned model with retrieval, achieved a mean score of 2.51 out of 5 and produced complete answers on a Pixel 7 Pro in under 15 seconds without network connectivity.
By Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman Mushfique