The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.
By Micah Adler, John W. Byers, Mark Crovella
Omni-Embed-Mini is a 0.9B‑parameter model that embeds text, speech, audio, images, video, and visually‑rich documents into a single shared cosine space without updating any text‑side parameters. It uses a dense cascaded caption as a teacher signal, allowing the teacher and student to share identical backbone weights and requiring only lightweight projectors and phased LoRA adapters for alignment. The model achieves strong text retrieval performance (49.57 nDCG@10 on MTEB‑v2 BEIR‑8) while extending to five additional modalities and is significantly smaller than other open omni‑modal embedders.
By Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Sahal Shaji Mullappilly, Ivan Laptev, Hisham Cholakkal
The paper introduces AMRA, a weight‑editing technique that mitigates abliteration—an attack that removes refusal capabilities from large language models by projecting weight matrices orthogonal to a refusal direction. AMRA obscures the refusal signal through rank‑$k$ updates to residual stream writer matrices, replaces refusal‑inducing activations with random aliases, and adjusts downstream reader matrices to maintain original behavior. Experiments on Llama‑3‑8B and Gemma‑2‑9B show significant improvements in post‑abliteration refusal scores with minimal impact on overall model performance.
By Nathan Truong
arXiv:2608. 05238v1 Announce Type: new Abstract: Training multimodal models to align time series with language runs into a self-supervision trap.
By Xinran Feng, Yi Xie, Chao Zhang, Ruikun Li, Wanyun Ling, Ziyue Li, Chenxi Liu
arXiv:2605.28190v2 Announce Type: replace
Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...
By Manuel Frank, Haithem Afli
arXiv:2608. 04021v1 Announce Type: cross Abstract: Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits.
By Han-yu Wang