The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.
arXiv:2610.15710v1 Announce Type: new Abstract: Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair...
ClaimScope is a 72‑task benchmark that audits mathematical claims by requiring a verdict, scope, counterexample, and repair, covering areas such as optimizer‑state transport and exact $2 imes2$ linear algebra. The study evaluates Qwen3‑8B using a witness checker that only examines the model’s submitted state, showing that verifier feedback improves counterexample acceptance and safe completions compared to generic feedback. The results demonstrate that separating claim classification from executable evidence can yield gains in witness‑structure correction without necessarily improving broader reasoning.
arXiv:2608. 14617v1 Announce Type: cross Abstract: A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline.
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
The study analyzes 373,019 judgments from LLM‑scored benchmarks, decomposing variance into system, item, judge, and interaction components via generalizability theory. It finds that with a single judge, generalizability converges to a ceiling determined by the system‑by‑judge variance, which is substantially lower in pairwise preference settings, allowing one judge to suffice. The research also reveals significant biases in presentation order and highlights that many published win‑rate claims fall below the measured floor of the benchmarks.
arXiv:2607. 02104v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.
The study investigates whether Jev, a typed classifier that outputs probabilities over allowed answers without generating text, can replace large language model (LLM) rubric judges. Across nine panels from seven benchmarks, Jev’s accuracy differed significantly from LLM judges in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons were inconclusive. In terms of cost and speed, Jev was 29 to 325 times cheaper and 30 to 220 times faster than the flash‑tier LLM judges, and a cascade approach that defers uncertain Jev verdicts to an LLM yielded only modest gains. whyItMatters":"The findings suggest that a lightweight classifier like Jev can serve as an efficient first‑stage evaluator, potentially reducing the reliance on expensive and slow LLM judges in automated grading pipelines."
arXiv:2605. 03534v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) grounds answers in retrieved passages, yet relevance does not guarantee sufficiency: a topical passage may still fail to justify the answer.
The paper evaluates nine on‑device named‑entity recognition models ranging from classical taggers to large language models, measuring not only accuracy but also latency and output validity. Using a silver‑gold benchmark derived from an LLM judge panel and a human‑validated corpus, the study shows that encoder‑based models achieve comparable accuracy to a 4 B instruct LLM while being much smaller, faster, and producing no malformed output. Confidence calibration of GLiNER is analyzed, revealing over‑confidence but improved reliability after temperature scaling and thresholding.
arXiv:2606.07834v2 Announce Type: replace-cross Abstract: Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately in...
arXiv:2607. 18960v1 Announce Type: cross Abstract: Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring.
arXiv:2607. 10202v1 Announce Type: new Abstract: Cross-model comparisons read divergence in value dispositions as evidence that language models hold individuated values.