arXiv:2608.00152v3 Announce Type: replace-cross
Abstract: AI evaluation can support the wrong inference when an in-domain benchmark success does not survive distribution shift, or when the benchmark...
By Mehrdad Shoeibi, Niloofar Yousefi
arXiv:2608. 00152v1 Announce Type: new Abstract: Predicting the magnitude of a CRISPRi perturbation's transcriptomic effect on held-out target genes is an important open problem in single-cell biology.
By Mehrdad Shoeibi, Niloofar Yousefi
SCALE is a conditional transport model that treats cells as unordered sets to predict treated cell populations without requiring cell-level matching. It uses a shared set-aware encoder and a conditional DiT backbone to learn latent transport, enabling endpoint supervision that is directly delta-aligned. Across diverse perturbation types—including genetic, chemical, developmental, and immune—SCALE accurately recovers gene‑expression changes, response directions, and population structure, outperforming competing methods on CRISPR data and successfully prioritizing cytokines that elicit distinct immune responses.
By Shuizhou Chen, Lang Yu, Xueqin Lin, Xinjie Mao, Songming Zhang, Xinyu Gu, Hao Wu, Sheng Xu, Kedu Jin, Lei Bai, Quan Qian, Qin Chen, Qiang Gao, Siqi Sun, Zhangyang Gao
The paper introduces CELLAUDIT, a method for auditing whether inputs claimed to influence predictive models actually do so. By testing if an input can enter the computation, whether predictions depend on it, and if that dependence improves observed responses, the authors evaluate agent-generated predictors on a morphology‑transcriptomics benchmark (BBBC047). Their findings show that many models claim compound contributions that are not supported by the data, and that falsification‑guided revisions can recover genuine input effects while improving performance.
By Mengran Li, Bo Li, Chengyang Zhang, Yang Yan, Jinfeng Xu, Zhenchao Tang
The paper introduces CP‑BG‑Bench, a paired‑view evaluation framework for Cell Painting vision encoders that fixes a central cell across four matched views (raw crop, segmented, and density‑augmented variants). Using this framework on three datasets and three encoders, the authors show that standard single‑metric rankings (e.g., replicate mAP) vary systematically across protocols, revealing disagreements along axes of cell versus background, morphology versus context, and within‑study versus across‑batch performance. The study demonstrates that segmented views can outperform crops in certain tasks and that background‑driven gains are largely determined by experimental design rather than encoder choice.
By Tim Treis, Nikita Moshkov, Johan Fredin Haslum, Shantanu Singh, Fabian J. Theis
The paper introduces AssayBench-Loop, a large benchmark of 1,389 CRISPR screens across five phenotype categories, and builds on it to develop AssayLoop, a sequential experimental design framework that combines a transformer-based acquisition policy (AssayFormer) trained on historical data with LLM-derived biological priors. AssayLoop achieves a 5.67‑fold enrichment over random selection, recovering 27.7% of hits after testing only about 5% of the library, and outperforms existing adaptive-design methods and standalone LLMs. The authors also present AssayLLM, extending the approach directly to an LLM via task‑specific post‑training, and show that performance improves with more historical training data and transfers to unseen phenotype categories.
By Carl Edwards, Edward De Brouwer, Xiner Li, Namkyeong Lee, Ehsan Hajiramezanali, Anne Biton, Sara Mostafavi, Gabriele Scalia