arXiv Computation and Language By Donghwan Kim

When LLMs Sit Above Diagnostic Tools: Unrealized Complementarity in Industrial Fault Diagnosis

Read the original on arXiv Computation and Language →

Large language models (LLMs) are increasingly used as integration layers above specialized diagnostic tools, but stronger components do not always yield stronger combined systems. Across five industrial fault‑diagnosis datasets, LLMs often overturned correct judgments when given conflicting external information, and none of the four datasets with direct integration comparisons showed a consistent advantage for implicit LLM integration over the stronger standalone source. In a Tennessee Eastman confirmation set, the integrated output improved LLM accuracy by 12.8 points but remained 5.9 points below the specialist alone, and a two‑source selector oracle achieved 92.76% accuracy, revealing complementarity that the integrated output did not fully realize.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Oct 1

What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents

The article reviews how replacing components in language‑model agents affects execution trajectories and downstream outcomes. It maps 348 studies, analyzes 90 comparison records, and finds that while many studies report both local decision metrics and task endpoints, they rarely demonstrate matched comparisons or prove that improved local decisions drive task‑level gains. The review identifies three potential mechanisms—recovery and disruption, intervention timing, and downstream use—and proposes eight claim‑specific reporting items to clarify evidence quality.

By Shuyang Zhang (The Hong Kong Polytechnic University), Jianshuo Chang (The Hong Kong Polytechnic University)
arXiv AI
Oct 26

Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.

By Ancil Crayton
arXiv Computation and Language
Sep 14

Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?

The study investigates how deictic ambiguity—specifically the shifting reference of expressions like "previous"—affects Draft‑Verify‑Revise pipelines that use multiple large language models (LLMs). Using a synthetic dataset of 10 base examples and 21 reasoning‑effort configurations, six LLMs were evaluated for their ability to correctly resolve the ambiguous expression across the draft, verify, and revise stages. Results show wide variance in balanced accuracy, with GPT‑5.2 improving from 0.156 to 0.942 with increased reasoning effort, while Gemini 3 Pro consistently achieved high accuracy above 0.94 even at low reasoning effort, and meta‑evaluators often relied on surface cues when making errors.

By Obinna I. Ekekezie