← Back to all news
arXiv Machine Learning By Hao Fu, Jichao Sun, Baiting Zhu, Qiaoling Liu, Yan Shi, Cheng Lu, Liu Liu, Yubo Wang, Xin Yao, Xiangyu Niu, Xu Dong, Wenhan Lyu, Chiyao Shen, Yinjie Huang, Minglei Chen, Shuai Ding, Li Fan, Xiao Kong

Hybrid GPU-CPU Retrieval for Personalized Search at Ultra-Large Scale

Read the original on arXiv Machine Learning →

The Flow has not summarised this story yet — read it at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 23

Efficient Iterative Retrieval with Heterogeneous Batching

Orthrus is a serving system that performs heterogeneous batching of embedding and generative models within a single inference loop. It uses chunked embedding with incremental pooling and workload‑aware batch composition to unify conflicting computational patterns. Experiments on four A100 GPUs show that Orthrus improves throughput by 1.28×–4.52× and reduces p99 latency by up to 55.8% compared to baseline deployments.

By Dohyun Park, Hubertus Franke, Daniel G. Waddington, Swaminathan Sundararaman, Yongjoo Park
ragefficiencybenchmarks
More like this →
arXiv AI
Sep 1

Efficient GPU Retrieval for Semantic Search

The paper introduces a GPU‑optimized retrieval framework for LinkedIn’s semantic search, partitioning embeddings into eight category‑supervised segments and applying a min/median aggregation rule aligned with the existing relevance policy. A lightweight Stage‑1 scorer generates high‑recall candidates, while a two‑stage GPU architecture—FP8 coarse ranking followed by FP16 re‑ranking—boosts throughput and recall, achieving 99.6‑99.8% of full‑FP16 recall at over 500 QPS per shard. In A/B testing, the system raises exploratory‑query Precision@10 from 63.7% to 79.0% and navigational Precision@1 from 65.5% to 74.7%, with human evaluation confirming the improvement.

By Dhritiman Das, Chujie Zheng, Ronak Kaoshik, Pratik Dixit, Vishal Shah, Yanbo Li, Jiahao Xu, Manika Agarwal, Chinmay Naik, Lingyu Zhang, Chetan Bhole, Chirag Bhanuprasad Mehta, Meng Zheng, Puneet Singh Ahluwalia, Shirisha Singh, Ping Jin, Manas Apte, Gokulraj Mohanasundaram, Tugrul Bingol, Raghavan Muthuregunathan, Fedor Borisyuk
llmsragefficiency
More like this →
arXiv Computation and Language
3d ago

KuaFu: Compressing Long User Behavior into Understanding at Billion Scale

KuaFu is a unified behavior‑compression layer that reduces each user behavior item to 2–4 tokens, dramatically shrinking per‑item cache size while preserving fidelity through a four‑stage training process. In production across four profiling tasks, it matches or outperforms uncompressed single‑task models, boosts GPU throughput by 37–350%, and saves 190 GPUs. On public benchmarks it consistently beats prior compressors at the same compression ratio, and on RecBench a 4B KuaFu model outperforms its 8B counterpart by 1.90 points, contributing to a 1.37% lift in overall GMV on Tencent’s advertising and recommendation platform.

By Jiahao Hui, Lin Zhu, Yishen Hu, Jingdong Shu, Zetai Jiang, Xining Ran, Ben Tan, Yeshou Cai, Gong Chen, Haijie Gu, Jie Jiang
agentsnlpefficiencybenchmarkssafety
More like this →
arXiv Machine Learning
Jul 22

Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

arXiv:2602. 22647v2 Announce Type: replace-cross Abstract: Generative retrieval has emerged as a powerful paradigm for LLM-based recommendation.

By Zhengyang Su, Isay Katsman, Yueqi Wang, Ruining He, Lukasz Heldt, Raghunandan Keshavan, Shao-Chuan Wang, Xinyang Yi, Mingyan Gao, Onkar Dalal, Lichan Hong, Ed Chi, Ningren Han
llmsefficiencybenchmarks
More like this →