arXiv AI
FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 16
FDE-Gym: The Last Mile of AI Delivery
FDE-Gym is a new benchmark that evaluates AI agents on 95 end‑to‑end delivery tasks drawn from real customer scenarios across 18 sectors. Unlike traditional benchmarks that use well‑specified tasks, FDE-Gym places agents in simulated customer environments where they must clarify ambiguous requirements, interact with stakeholders, and uncover unstated constraints to improve a customer’s success metric. In tests, the best model configuration achieved a mean score of 0.56 and solved only 14 of the 95 tasks, highlighting a significant gap between well‑specified problem solving and real‑world AI delivery.
arXiv Computation and Language
2d ago
Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL
The paper investigates whether large language model (LLM) coding agents can function as forward‑deployed engineer (FDE) agents by discovering customer requirements and executing deployment plans in a real enterprise text‑to‑SQL scenario. In a simulated engagement, agents must uncover acceptance criteria, data permissions, and hardware constraints before planning and delivering a post‑training recipe. Across 15 models, the best agents uncovered over 80% of hidden requirements, yet only 25% of runs met all acceptance criteria, revealing bottlenecks in discovery, execution reliability, compute usage, and benchmark relevance.
arXiv AI
Oct 19
EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
arXiv AI
Jul 24
InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
arXiv:2607. 20468v1 Announce Type: new Abstract: AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces.
arXiv Computation and Language
Oct 26
CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
CRM‑Agent‑Bench is a benchmark that evaluates language agents in customer‑relationship‑management settings by testing 555 stateful, multi‑turn tasks across nine business domains. Each task includes an executable contract with checks on actions, arguments, order, responses, target state, and protected state, and success requires meeting all these checks. The benchmark reports coverage, latency, and cost, showing that evaluated systems achieve 64.32%–83.60% strict success while varying in cost and domain reliability.
arXiv AI
Oct 9
Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking
Agent4RE is a self‑refining multi‑agent framework designed for end‑to‑end software requirements engineering (RE). It coordinates specialized agents and includes two iterative improvement loops, and introduces a new real‑world dataset, RE‑E2E, for evaluating RE workflows from elicitation to generation. The framework, along with two enhanced variants that use autonomous self‑refinement or structured human feedback, consistently outperforms baseline prompting methods across eight large language models, achieving higher text‑based metrics and superior human and LLM‑as‑a‑judge ratings.