Checking an A/B test until it crosses p < 0. 05 can turn a nominal 5 percent false-positive rate into almost 28 percent.
The article discusses a small adversarial test set designed to detect retrieval failures in Retrieval-Augmented Generation (RAG) pipelines that typical evaluation sets might miss. It emphasizes the importance of proactively testing your own RAG system to uncover hidden weaknesses before users encounter them. By using this targeted test set, developers can improve the reliability and robustness of their RAG models.
How a single evaluation choice inflated my results by 25 points, and what rebuilding honestly taught me about ML systems people might depend on The post My Fall-Detection Model Scored 94%, and It Was Lying to Me appeared first on Towards Data Science .
An online simulation and a novel method for increasing power The post How to Get More Statistical Power from Fewer Research Participants appeared first on Towards Data Science .
A preprocessing pipeline let my car price model peek at the test set before the exam, and the twelve points of R squared it cheated its way to The post My Model Was Cheating on Its Own Test appeared first on Towards Data Science .
A hands-on guide to tracking experiments, logging models, and reproducing results with ML Flow. The post Are Your ML Experiments a Mess?
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
The article discusses Anthropic’s Claude Fable 5.1 release, highlighting its claimed improvements in coding, knowledge work, and problem‑solving, particularly a 52.6% score on the new Terminal‑Bench‑Science 0.1 benchmark. The author examines the model’s performance on the pelican benchmark, noting that Fable 5.1’s five reasoning levels (low, medium, high, xhigh, max) sometimes skip reasoning entirely for certain prompts, as evidenced by token counts and cost metrics. The piece provides detailed transcript data for each reasoning level when generating an SVG of a pelican riding a bicycle.
The article describes how the author constructed a prompt dependency graph to identify which prompts are affected when a single prompt changes. By separating all reachable components from the smaller subset that truly requires evaluation, the graph helps focus retesting efforts. This approach streamlines testing by pinpointing only the prompts that need targeted evaluation.
How to decide when an AI agent should act on its own by using cost asymmetry instead of a fixed confidence cutoff The post The Threshold Is a Price, Not a Percentage appeared first on Towards Data Science .
One near miss, four months of running agents, and the question almost nobody is asking: what are you supposed to do while the AI writes the code? The post AI Made Me 5x Faster. It Also Made Me 5x Wors...
The article describes a real‑world case of scaling an enterprise integration pipeline from 500 to 8,000 events per second. It emphasizes that during this throughput increase, two correctness guarantees were strictly maintained and never compromised. The post illustrates how to achieve high performance while preserving essential data integrity constraints.
arXiv:2608. 14425v1 Announce Type: new Abstract: LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise.