arXiv Machine Learning
Aug 3
The Checking Problem: What must be true before AI ships in a regulated firm
arXiv:2607. 28666v1 Announce Type: cross Abstract: Enterprise AI programmes stall at a rate that is widely quoted and poorly explained.
AI Evaluation Should Measure Verification Cost, Not Correctness Alone
arXiv:2608. 08709v1 Announce Type: new Abstract: The reliability of AI generative models is typically measured by output correctness, yet in practice it depends on the effort required to verify those outputs.
Related stories
arXiv AI
Aug 10
Characterizing the Quality Profile of AI-Generated C++ in Production
arXiv:2608. 06640v1 Announce Type: cross Abstract: The widespread integration of AI coding assistants offers undeniable boosts to engineering velocity.
arXiv AI
Jun 12
Cross-Model Disagreement as a Label-Free Correctness Signal
arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.
arXiv AI
Jun 4
Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation
arXiv:2606. 04402v1 Announce Type: new Abstract: Modern reasoning models can allocate different amounts of test-time computation, such as thinking tokens, model calls, or compute budget, to different tasks.
arXiv AI
Jun 2
Benchmarking AI for low-resource contexts: Thinking beyond leaderboards
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
arXiv AI
Jun 30
Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.
arXiv AI
Aug 11
PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
arXiv AI
Jun 30
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.
arXiv AI
Jun 11
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
arXiv AI
2d ago
Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
arXiv:2608. 13867v1 Announce Type: cross Abstract: AI coding agents are commonly evaluated as models but deployed as systems.
arXiv AI
Aug 6
ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation
arXiv:2608. 04439v1 Announce Type: cross Abstract: Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations.
arXiv AI
Jun 30
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.