arXiv:2608. 04240v1 Announce Type: cross Abstract: Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike.
arXiv:2609.22245v1 Announce Type: new Abstract: Large language models can produce fluent explanations for chess moves, but plausible language does not necessarily reflect the reasoning behind a decis...
arXiv:2601.08654v3 Announce Type: replace Abstract: Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the...
The paper presents a systematic mapping of recent chess research involving humans, engines, neural and reinforcement‑learning systems, large language models (LLMs), and hybrid approaches. It identifies 84 core study families and classifies them by agent type, strategic‑reasoning stages, and evaluation dimensions, highlighting a strong focus on situation assessment, evaluation, and action selection while noting gaps in planning, explanation, metacognition, and human–AI collaboration. The study also distinguishes hybrid systems by integration timing and cautions that improved human performance in evaluations does not automatically imply human–AI synergy.
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
The paper introduces Bayesian Dialectical Argumentation (BDA), a method for aggregating answers from multiple large language models (LLMs) in a council setting. BDA treats each LLM’s typed moves—proposals, challenges, and concessions—as evidence in a classical annotator model, estimating per-agent reliability even when some agents are persistently unreliable. By weighting evidence according to these inferred reliabilities, BDA produces calibrated posterior probabilities for candidate answers and can invert unreliable agents instead of merely outvoting them, achieving superior calibration and robustness on both binary and multi-class benchmarks without extra LLM calls.
The paper investigates why large language models (LLMs) struggle in strategic decision-making under incomplete information. It identifies two key gaps: an observation‑belief gap where LLMs’ internal representations of game states are accurate but brittle, and a belief‑action gap where converting these internal beliefs into actions is weak, leading to suboptimal payoffs. Experiments with Llama 3.1, Qwen3, and gpt‑oss confirm that acting optimally on decoded beliefs would improve outcomes in most games, highlighting a bottleneck in belief‑to‑action conversion.
arXiv:2605. 12519v2 Announce Type: replace-cross Abstract: Training language models to produce both correct answers and sound reasoning remains an open challenge.
VeriPhy is an auditable physical‑verification system that transforms a text prompt into typed physical obligations and a statically validated execution plan before any video frames are generated. During execution, it gates calls to frozen low‑level experts (segmentation, tracking, counting, depth, OCR, audio‑event detection, etc.) and records provenance‑carrying evidence for each action. The system maps these records to a three‑valued state—supported, contradicted, or unknown—providing traceable verdicts that can be used to refine generation models.
arXiv:2607. 06495v1 Announce Type: cross Abstract: Live sports commentary is grounded generation under a deadline: statements concern real, named athletes, the grounding state changes every few seconds, and no reference text exists at generation time.
Kepler is an open‑source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. In the ARC‑AGI‑3 benchmark, a frozen Claude Opus 5 configuration achieved a perfect 100.00 RHAE on all 25 public games without per‑game model selection or score‑conditioned reruns, and matched or outperformed median‑human action counts on 181 of 183 levels. The study also identified three evaluation failures and highlighted that public‑set score alone has limited discriminative value, advocating for first‑attempt, cost‑conditioned, and verification‑aware reporting. whyItMatters":"The results demonstrate that a purely score‑based evaluation can be misleading, underscoring the need for more rigorous, cost‑aware, and verification‑aware metrics in AI benchmark assessments."
The paper investigates whether large language models (LLMs) correctly gauge their confidence when acting in a hidden‑information chess variant. In experiments where the location of a hidden royal piece is repeatedly relocated, the models’ stated probabilities about the piece’s position were almost never accurate at high confidence levels, with a calibration deficit concentrated in those high‑confidence events. Across multiple model configurations and providers, the same pattern emerged, and conventional evaluation metrics such as legality, cost, latency, and completion rate were found to be uncorrelated with belief quality, yet a model could still win the game despite poor confidence estimates.
VeriPhy is an auditable physical‑verification system that evaluates generated video by compiling prompts into typed physical obligations and a statically validated execution plan before any frames are observed. During execution, it gates calls to frozen low‑level experts (e.g., segmentation, tracking, counting, depth, OCR, audio‑event detection) and returns provenance‑carrying evidence records, which are mapped to a three‑valued state (supported, contradicted, unknown) with full traceability. On a 1,500‑clip corpus of human‑annotated flaw records, VeriPhy accounts for 228 failures out of 304, outperforming a published question‑decomposition evaluator that accounts for 164, while also providing auditable evidence for each verdict.