arXiv Computation and Language

AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worlds

arXiv AI

Jun 17

Blueprint First, Model Second: A Framework for Deterministic LLM Workflow

arXiv:2508. 02721v2 Announce Type: replace-cross Abstract: While powerful, the inherent non-determinism of large language model (LLM) agents limits their application in structured operational environments where procedural fidelity and predictable execution are strict requirements.

By Libin Qiu, Yuhang Ye, Zhirong Gao, Xide Zou, Junfu Chen, Ziming Gui, Weizhi Huang, Xiaobo Xue, Wenkai Qiu, Kun Zhao

llms agents benchmarks

arXiv AI

Jul 24

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

arXiv:2607. 20468v1 Announce Type: new Abstract: AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces.

By Jehyeok Yeon, Ben Rank, Maksym Andriushchenko

llms agents efficiency benchmarks

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

arXiv AI

Jun 2

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

arXiv:2605. 25246v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines.

By Minwei Kong, Chonghe Jiang, Ao Qu, Wenbin Ouyang, Zhaoming Zeng, Xiaotong Guo, Zhekai Li, Junyi Li, Yi Fan, Xinshou Zheng, Xi Jing, Yikai Zhang, Zhiwei Liang, Seonghoo Kim, Runqing Yang, Zijian Zhou, Sirui Li, Han Zheng, Wangyang Ying, Ou Zheng, Chonghuan Wang, Jinglong Zhao, Hanzhang Qin, Cathy Wu, Paul Pu Liang, Jinhua Zhao, Hai Wang

llms agents benchmarks

arXiv AI

Aug 11

Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon Tasks

arXiv:2608. 05144v2 Announce Type: replace Abstract: Long-horizon reasoning requires an agentic runtime that can persist when evidence supports its current approach and pivot when measurements reveal failure, hidden constraints, or a misspecified objective.

By Boxiu Li, Zimo Wen, Yijia Fan, Chuan Wen, Fan Yang, Hangxi Guo, Jiaao Wu, Jiachen Zhang, Junxiang Lei, Mukai Li, Ruize Tang, Runjing Gu, Shibo Hu, Sihan Chen, Sufeng Guo, Wanbo Zhang, Xian Zhang, Xiaoyu Chen, Xuanhe Zhou, Xuyao Huang, Yifei Gao, Yifei Shen, Yilin Chen, Yuheng Wu, Yuzhe Zhang, Zelong Zhao, Zhijie Deng

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 4

Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problems

arXiv:2606. 04816v1 Announce Type: new Abstract: Large language models (LLMs) increasingly translate natural-language optimization problems into executable solver code.

By Xizi Luo, Changhong He, Dongdong Geng, Chenggong Shi, Yu Mei

llms benchmarks

arXiv AI

Aug 6

Argus: A General-Purpose Agentic Runtime for Long-Horizon Reasoning

arXiv:2608. 05144v1 Announce Type: new Abstract: Long-horizon reasoning requires an agentic runtime that can persist when evidence supports its current approach and pivot when measurements reveal failure, hidden constraints, or a misspecified objective.

By Boxiu Li, Zimo Wen, Yijia Fan, Junxiang Lei, Sufeng Guo, Jiaao Wu, Ruize Tang, Mukai Li, Yifei Shen, Xiaoyu Chen, Wanbo Zhang, Runjing Gu, Yifei Gao, Yuheng Wu, Xuyao Huang, Zelong Zhao, Jiachen Zhang, Shibo Hu, Hangxi Guo, Yilin Chen, Yuzhe Zhang, Fan Yang, Chuan Wen, Xian Zhang, Xuanhe Zhou, Zhijie Deng

llms agents reinforcement-learning benchmarks

arXiv AI

Aug 6

Capability-Gated Planning: Cost-to-Goal Discovery and the Limits of Myopic Experiment Selection

arXiv:2608. 05085v1 Announce Type: cross Abstract: Systems that automate scientific discovery must repeatedly decide which experiment to run, which hypothesis to test, which tool to build, and when to stop.

By Ahmed Hassoon, Mark Dredze

arXiv AI

Jul 29

A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain

arXiv:2607. 25415v1 Announce Type: new Abstract: Production LLM agents are increasingly assembled from a frozen model wrapped in a harness: a prompt template, a tool set, a memory/retrieval layer, a planning strategy, and a verification policy.

By Debjyoti Paul

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 19

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

arXiv:2606. 19787v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear.

By Jiajun Li, Mingshu Cai, Yixuan Li, Yu Ding, Ran Hou, Guanyu Nie, Xiongwei Han, Wanyuan Wang

llms agents benchmarks

arXiv Machine Learning

Jun 2

Evaluating Real-World Generalizability of Algorithm Selection Models

arXiv:2606. 02016v1 Announce Type: new Abstract: Algorithm Selection (AS) aims to automatically identify the most suitable optimization algorithm for a given problem instance by leveraging measurable problem characteristics and historical performance data.

By Gjorgjina Cenikj, Jakub Kudela, Eva Tuba, Tome Eftimov

robotics benchmarks

arXiv AI

Jul 15

Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution

arXiv:2607. 13034v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires.

By Junjie Yin, Xinyu Feng

llms agents benchmarks

Related stories