OpenAI Blog

Terminal X matches senior investor judgment 88.6% with GPT-6.1

Terminal X’s Eval Agent evaluates GPT‑6.1’s outputs using a 54‑criterion rubric, achieving an 88.6% match with senior investor judgment. The system detects 25% more deviations from a fund’s mandate compared to previous methods.

arXiv AI
Oct 1

Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking

The paper evaluates an agentic harness that combines a 27B language model, financial calculations, narrative templates, and validation checks to produce corporate and investment banking presentation decks. Using a panel of five judges, the system consistently scores higher than a baseline model that generates directly from a short prompt, with scores ranging from 20.4 to 33.6 out of 95. However, judge variability and changes in grading criteria make it challenging to discern small improvements in deck quality.

By Ludovic Gibert, Matis Despujols, Andre-Louis Rochet
arXiv AI
Oct 28

FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents

FIDES-Live extends the FIDES framework to audit sequential trading agents by separating decision validity, numerical commitment adherence, execution and ledger reconciliation, and receipt‑grounded reporting. In a retrospective audit of 7,299 cryptocurrency account‑rounds and a 310‑round equity pilot, decision validity reached 97.35% and structured reporting accuracy 90.81%, while commitment adherence overall was 94.36% but dropped to 62.63% for non‑empty order proposals. The study shows that all cash and position transitions reconcile with recorded fills, and highlights the need to report concordance, evidence coverage, activity, and financial outcomes separately.

By Arther Tian, Alex Ding, Simon Wu, Aaron Chan
arXiv Machine Learning
Oct 3

VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence

VERITYGATE is a four‑gate framework that checks whether LLM‑generated narrations adhere to a fixed schema of declared evidence IDs, entities, numbers, and claim types, rather than verifying every fact in the prose. In experiments with GPT‑4o‑mini, Llama‑3.3‑70B, and Claude Sonnet 4.6 on 900 grounded‑ungrounded pairs, the framework identified high failure rates (up to 80.3% for mini claims) and demonstrated that a single repair pass can improve claim survival rates. The authors also provide code, data, and preliminary human studies to validate the rules and highlight gaps between schema compliance and correct prose. "whyItMatters":"The framework offers a systematic way to evaluate and improve the faithfulness of LLM explanations to structured evidence, revealing significant failure rates and the impact of repair strategies."

By Sachin Gupta
arXiv Machine Learning
Oct 3

Sentence Specificity Scores for Collaborative Technical Documentation: A Domain-Transfer Study

The paper investigates how sentence‑specificity scores can guide the selection of revisions in collaborative technical documentation. It compares two predictors—SpeciTeller and a target‑adapted model by Ko et al.—across Wikipedia and three technical corpora, finding that the predictors rank sentences differently and that SpeciTeller can improve direction‑valid selection rates in certain datasets. The study also shows that filtering and token‑length adjustments alter but do not reconcile these ranking differences.

By Rocker D'Antonio, Thomas Benton Townsend, Dimitrios Michael Manias
arXiv AI
Jul 31

APEX-Accounting

arXiv:2607. 27189v2 Announce Type: cross Abstract: We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants.

By Julien Benchek, Austin Bennett, Jasmin Kern, Ryan Stevens, Rene Sultan, Charis Ching, Hayley Popiel, Vaibhav Mittal, Felix Mercier, Brendan Foody, Bertie Vidgen
arXiv AI
Sep 18

When Hiring Becomes Agent-Mediated: Evaluating Access and Recurrence in Two-Agent R\'esum\'e Screening

The study examines a two‑agent résumé screening process where both employer‑side and candidate‑side agents exchange evidence before deciding who advances, contrasting it with the traditional one‑call automated screening. Using GPT‑5.5 and Claude Opus 4.7 on 600 constructed résumé‑job pairs, the two‑agent method increased the proportion of applications advanced (up to 39.3% for GPT‑5.5) and raised pass rates for borderline cases from 4.5% to 26.2% (GPT‑5.5) and 6.5% to 16.1% (Opus 4.7). The results show that the screening procedure itself, rather than just the underlying model, determines which candidates reach human review and how consistently that access recurs.

By Jian Gao, Hang Jiang