arXiv AI
1d ago
Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
arXiv:2608. 14711v1 Announce Type: new Abstract: AI coding agent benchmarks rank agents with the Chen et al.
Benchmarks and evaluation
Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv AI
1d ago
Beyond Thresholds: A Quality-Aware Decision Intelligence Framework for Cold Chain IoT Systems
arXiv:2608. 15082v1 Announce Type: new Abstract: Cold chain logistics has advanced technologically, yet most deployed systems remain reactive monitors, not decision-making agents: thresholds trigger alerts, but nothing relates violations to cumulative product degradation or converts degradation signals into logistics decisions.
arXiv AI
1d ago
Large Language Models and their Awareness of Mechanics and Spatial Geometry
arXiv:2608. 14615v1 Announce Type: new Abstract: Large Language Models (LLMs) perform well on established code-generation and mathematical-reasoning benchmarks, but their capabilities in mechanics and spatial geometry, here denoted as mechanical engineering awareness, has not been quantified systematically.
arXiv AI
1d ago
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
arXiv:2608. 14680v1 Announce Type: new Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails.
arXiv AI
1d ago
Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
arXiv:2608. 14903v1 Announce Type: new Abstract: Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief.
arXiv AI
1d ago
Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form
arXiv:2608. 15022v1 Announce Type: new Abstract: Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly.
arXiv AI
1d ago
Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture
arXiv:2608. 14566v1 Announce Type: new Abstract: Recent work on evaluating the moral competence of large language models (LLMs) has focused primarily on what we call the moral value problem, i.
arXiv AI
1d ago
Do LLM Agents Negotiate Rationally? A Mechanism-Design Framework for Verifiable Multi-Agent Interaction over A2A/MCP
arXiv:2608. 14613v1 Announce Type: new Abstract: Modern LLM-agent frameworks increasingly interoperate through standards such as Anthropic's Model Context Protocol (MCP) for agent-to-tool access and Google's Agent2Agent (A2A) protocol for agent delegation and negotiation.
arXiv AI
1d ago
Evaluating Multimodal LLMs across Text and Audio Modalities for Accessible Disaster Assistance
arXiv:2608. 14651v1 Announce Type: new Abstract: Effective disaster risk communication is a foundational humanitarian challenge, yet current emergency infrastructure fails to meet the needs of individuals with access and functional needs, including hard-of-hearing individuals, pregnant women, mothers with toddlers, and elderly individuals with dementia.
arXiv AI
1d ago
Beyond Correctness: Toward Automated Novelty Verification with Lean 4
arXiv:2608. 14669v1 Announce Type: new Abstract: Artificial intelligence systems applied to mathematics verify correctness but not novelty: an automatically generated theorem can compile in Lean without errors and yet be an already known result.
arXiv AI
1d ago
Task-Driven Three-Layer Distributed Scheduling for Emergency Earth Observation in Large Low-Earth-Orbit Constellations
arXiv:2608. 14789v1 Announce Type: new Abstract: Large low-Earth-orbit (LEO) Earth-observation (EO) constellations offer frequent access to geographically dispersed ground targets, but emergency requests may arrive after committed routine-plan execution has begun.
arXiv AI
1d ago
CEDAR-GRPO: Process-Aware Reinforcement Learning for General Abductive Reasoning in LLMs
arXiv:2608. 14791v1 Announce Type: new Abstract: Abductive reasoning, often characterized as inference to the best explanation, is central to explanation under uncertainty, from everyday sense-making and investigation to scientific discovery.
arXiv AI
1d ago
LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks
arXiv:2608. 14927v1 Announce Type: new Abstract: Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost.
arXiv AI
1d ago
Small Models Scout Bottleneck Order for Large-Model Data Control
arXiv:2608. 14936v1 Announce Type: new Abstract: Small proxy models are commonly used to identify data mixtures for larger-scale training.
arXiv AI
1d ago
The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning
arXiv:2608. 14558v1 Announce Type: new Abstract: Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content.
arXiv AI
1d ago
When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL
arXiv:2608. 14559v1 Announce Type: new Abstract: Effective communication in multi-agent reinforcement learning requires agents to decide not only \textit{what} to communicate, but when?
arXiv AI
1d ago
Euclid-Omni : A Unified Neuro-Symbolic Framework for Plane Geometry
arXiv:2608. 14585v1 Announce Type: new Abstract: Euclidean geometry is a compelling testbed for AI reasoning, as it demands the combination of intuitive diagram understanding, axiomatic deduction, and algebraic computation.
arXiv AI
1d ago
When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
arXiv:2608. 14610v1 Announce Type: new Abstract: Legal reasoning tasks such as legal judgment prediction (LJP) require identifying the temporally correct version of the law governing a case -- a capability we term temporal applicable-law determination.
arXiv AI
1d ago
Accuracy and Reliability of Large Language Models in Cosmetic Chemistry and Skin Health: A Benchmarking Study
arXiv:2608. 14631v1 Announce Type: new Abstract: As consumers increasingly turn to AI chatbots for skincare advice, the technical accuracy of Large Language Models (LLMs) in cosmetic chemistry remains largely under-evaluated.
arXiv AI
1d ago
Second-Order Policy Effects as State Transitions: A Source-Linked Benchmark for Policy Simulation
arXiv:2608. 15101v1 Announce Type: new Abstract: Policy evaluation often estimates direct benefits and costs while treating the institutional environment as fixed.