arXiv AI
Jun 3
Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions
arXiv:2606. 03331v1 Announce Type: cross Abstract: Consumer device repair is an important but underexplored testbed for large language models (LLMs).
Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions
Consumer device repair is an important but underexplored testbed for large language models (LLMs). Repair tasks require reasoning over incomplete problem descriptions, hardware-specific diagnostics, actionable troubleshooting, and safety-critical decisions, where incorrect advice can cause device damage, battery hazards, or permanent data loss.
Related stories
arXiv AI
Jun 16
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
arXiv:2606. 16262v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as UX judges that inspect interfaces, diagnose usability problems, and propose repairs.
arXiv Computation and Language
Sep 23
Faithful Autoformalization via Roundtrip Verification and Repair
The paper introduces a roundtrip verification method for ensuring that large language models (LLMs) produce faithful formalizations of natural language statements. By formalizing a statement, translating it back to natural language, re-formalizing, and checking logical equivalence with a formal tool, the approach detects inconsistencies without needing ground-truth annotations. When inconsistencies are found, a diagnosis localizes the error to a specific translation step, and a scoped repair operator attempts to correct it. The framework is evaluated on the Texas Transportation Code and Texas Parks and Wildlife Code using Claude Opus and GPT-5, showing that diagnosis-guided scoped repair is most effective and that rules failing the equivalence check exhibit significantly more natural language inference drift.
arXiv AI
Aug 19
LLM-Only PDDL Domain Repair with Open-Weight Models
The paper evaluates how well open-weight large language models can repair Planning Domain Definition Language (PDDL) models using only LLMs. Experiments show that while the best LLM achieves an F1 score of 0.87—an improvement of 0.38 over a symbolic baseline—it still fails to reliably satisfy test constraints, with a mean test pass rate of only 0.82 and as low as 0.06 on the Thoughtful domain. The study concludes that current open-weight models cannot guarantee the necessary test constraint satisfaction for dependable automated model repair.
arXiv AI
Jun 30
Diagnosing and Repairing Factual Errors in RAG under Budget Constraints
arXiv:2606. 29377v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) improves the factuality of large language models by grounding responses in external evidence, yet real-world deployments remain fragile.
arXiv AI
Jun 30
Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
arXiv:2601. 05366v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls.
arXiv Computation and Language
Sep 15
RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents
arXiv:2609.15684v1 Announce Type: new Abstract: Language agents increasingly rely on reusable skills, but post-failure repair is often handled by opaque one-shot reflection: a model generates a skill...
arXiv AI
Jul 29
How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair
arXiv:2607. 25873v1 Announce Type: cross Abstract: Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent.
arXiv AI
Aug 24
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge
The paper evaluates lightweight, edge‑deployable large language models—Claude‑Haiku‑4.5, GPT‑5.4‑Mini, and Gemini‑3.1‑Flash‑Lite—on free‑text 5G domain knowledge and fault‑analysis tasks using three benchmarks (TeleQNA ORAN FT, 5G‑Faults FT, TeleInter FT). All models achieve at least 90% accuracy on fault diagnosis, but zero‑shot recall of 3GPP and O‑RAN specifications remains below 60%. Multi‑judge scoring yields a mean inter‑judge agreement of at least 0.90, and Gemini‑3.1‑Flash‑Lite emerges as the most efficient model for production telecom deployments.
arXiv AI
4d ago
One Readout, Many Repairs: Diffusion-Guided Hierarchical Search for Tool-Agent Repair
arXiv:2609.34879v2 Announce Type: replace Abstract: Tool agents use large language models to act through external tools, yet successfully executed calls can still leave user requests unfulfilled. Too...
arXiv AI
Sep 11
Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.
arXiv AI
Jul 21
Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agents
arXiv:2607. 17641v1 Announce Type: new Abstract: Verify-repair loops are a standard means for large language model (LLM) agents to correct faulty plans in code generation, mathematical reasoning, and tool use.