arXiv AI By Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao, Lei Wang

Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

Read the original on arXiv AI →

The paper introduces isloss, a deterministic oracle that tests whether conversations are preserved accurately when translated across different LLM APIs. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to detect silent information loss such as changes to instruction roles, tool interactions, error states, or generation controls. Evaluation shows that isloss can detect a majority of planted faults with few false alarms and reveals that many translations silently alter critical conversation details, affecting model behavior.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
Jul 13

BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution?

arXiv AI
Oct 2

Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning introduces VeriSpec, a method that audits natural‑language model specifications directly using an LLM as a verifier. VeriSpec extracts structured rules, builds a topic‑guided graph to cluster related rules, and applies LLM‑based reasoning to find inconsistencies. In a case study on the OpenAI Model Spec, it identified 405 rules, validated five inconsistencies, and outperformed five baselines with the highest precision and lowest cost per validated inconsistency.

By Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang