arXiv Machine Learning

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

arXiv Computation and Language
Oct 1

QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

The paper introduces the QuantCode Model, a specialized language model for generating executable algorithmic trading code. It combines continued pretraining on trading framework code with supervised fine‑tuning on validated request‑to‑code pairs, achieving significant gains on the QuantCode‑Bench benchmark and improving both single‑turn and multi‑turn agentic success rates. The study also highlights the importance of retaining parsing and tool‑calling capabilities during domain specialization.

By Alexey Chernysh, Orkhan Ekhtibarov, Dmitry Zmitrovich
arXiv AI
Aug 26

Evaluating Language Models on Cross-Language Code Functional Equivalence

The paper introduces PolyHuman, a dataset of human-written programs in C++, Java, and Python, to test whether large language models can judge functional equivalence across languages. Using this dataset, the authors evaluate several open-weight and proprietary LLMs, finding that models struggle more with harder problems, show language-specific biases, and rely partly on superficial similarity cues. They also observe run‑to‑run instability in GPT‑o4‑mini, concluding that current LLMs do not reliably capture functional equivalence within or across programming languages.

By Hui Sun, Anderson Uch\^oa, Rohit Gheyi, Wesley K. G. Assun\c{c}\~ao
arXiv Machine Learning
Oct 3

How Much Can Language Models Gain from Test-Time Computation?

The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.

By Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu