The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2607. 05031v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct.
arXiv:2607. 19442v1 Announce Type: cross Abstract: Machine unlearning is commonly evaluated by matching a retrained oracle on trained probes.
arXiv:2607. 20537v1 Announce Type: cross Abstract: We introduce ReliableTableQA, a framework for training an LLM to annotate the statistical reliability of tabular QA results, not whether the query is answerable, but whether the computed answer is statistically meaningful.
The article examines how classical test theory reliability statistics misrepresent the performance of large language model (LLM) judges. It shows that internal‑consistency coefficients, the dependability index, and Livingston‑Lewis accuracy each conflate judge error with item design or criterion validity, making it impossible to attribute a single reliability value to the judge alone. The authors argue that such misattribution can influence deployment decisions and documentation.
arXiv:2608. 07946v1 Announce Type: cross Abstract: Text-to-SQL benchmarks ship schemas whose column names already say what the columns mean.
arXiv:2608. 09254v1 Announce Type: new Abstract: LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number.