arXiv AI
Aug 5
UL-UNAS: Ultra-Lightweight U-Nets for Real-Time Speech Enhancement via Network Architecture Search
arXiv:2503. 00340v2 Announce Type: cross Abstract: Lightweight models are essential for real-time speech enhancement applications.
Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search
The paper presents a reinforcement learning-driven neural architecture search (RL‑NAS) framework that discovers compact audio‑visual speech enhancement models. By preserving temporal alignment and using a reward function that balances SI‑SNR improvement with model size, the search evaluates only 35 candidates and finds a 2.9‑million‑parameter model that achieves a 14.5 dB SI‑SNR improvement—just 0.7 dB below the 22‑million‑parameter baseline. On commercial mobile CPUs, the new architecture reduces the real‑time factor by 4.3‑fold, enabling practical on‑device deployment while maintaining high enhancement performance.
Related stories
arXiv AI
Jul 16
LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement
arXiv:2603. 13952v3 Announce Type: replace-cross Abstract: In existing Audio-Visual Speech Enhancement (AVSE) methods, objectives such as Scale-Invariant Signal-to-Noise Ratio (SI-SNR) and Mean Squared Error (MSE) are widely used; however, their correlation with perceived speech quality is often suboptimal and provides limited interpretability for optimization.
arXiv Machine Learning
Sep 22
NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware
NAVIR is an end‑to‑end audio‑visual speech recognition system designed for the BrainChip Akida neuromorphic processor, which only supports sequential 2‑D convolutions. The architecture separates spatial and temporal encoding into three AkidaNet modules—per‑frame visual, temporal video, and spectrogram audio encoders—fused by a lightweight predictor and decoded with constrained beam search. Trained with CTC on noise‑augmented audio and fine‑tuned via quantization‑aware training, the quantized model achieves 14.0% WER on GRID’s unseen‑speaker split and 3.3% on overlapped‑speaker split, outperforming audio‑only baselines, and delivers 98.6% command accuracy at 1.5% WER on an industrial‑command corpus, while offering a 13‑fold energy advantage over conventional ANNs and roughly 5‑fold lower energy per inference than a Raspberry Pi CPU.
arXiv AI
Sep 4
StrixAE: An Intelligent Agent for Audio Enhancement under Complex Distortion Coupling in Real-World Scenarios
StrixAE is an intelligent audio enhancement agent that uses a multimodal large language model to coordinate multiple enhancement and personalization models. It is trained in two stages: first, chain‑of‑thought supervised fine‑tuning on AcoustBench, and second, Audio Perception Reinforcement Learning that optimizes format validity, structural coherence, and perceptual quality. The approach yields state‑of‑the‑art performance and strong generalization on real‑world test datasets, outperforming many existing open‑source and proprietary solutions.
arXiv Machine Learning
Sep 25
Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement
The paper presents a redesign of the LiSenNet speech‑enhancement model for deployment on the STM32N6570‑DK Neural‑ART microcontroller accelerator. By replacing the recurrent bottleneck with convolutional mixers, converting unsupported operations to static int8 primitives, and using bounded decoder activations, the authors achieve an NPU‑compatible model that matches or surpasses the original LiSenNet in quality (PESQ 3.08 vs 3.01 FP32) while running each 16 ms input hop in 4.83 ms (real‑time factor 0.30). The study demonstrates that co‑designing parameter count, operator compatibility, quantization range, and streaming state is essential for efficient real‑time speech enhancement on constrained NPUs.
arXiv AI
Sep 17
Performance and Complexity Trade-off Optimization of Speech Models During Training
The paper introduces a reparameterization technique that injects feature noise to jointly optimize speech model performance and computational complexity during training. Unlike traditional pruning, this method dynamically adjusts model size for a desired performance‑complexity trade‑off without heuristic weight removal. The authors validate their approach with a synthetic example and two real‑world applications—voice activity detection and audio anti‑spoofing—providing publicly available code for further research.
arXiv AI
Jul 3
UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
arXiv:2510. 20441v2 Announce Type: replace-cross Abstract: Neural audio codecs have largely promoted the application of language models (LMs) for speech applications.
arXiv Computation and Language
Sep 21
SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement
arXiv:2609.18009v2 Announce Type: replace-cross Abstract: Lightweight audio-visual speech enhancement (AVSE) models face a critical trade-off between computational efficiency and cross-modal alignmen...
arXiv AI
Aug 11
DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation
arXiv:2608. 09288v1 Announce Type: cross Abstract: Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions.
arXiv AI
Oct 16
CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge
The paper introduces CF-AVSS, a two‑stage audio‑visual speech separation framework that first generates coarse estimates using a synchronized visual‑stream SPAV‑TFGridNet, then refines these estimates with a TFGridNet‑based target‑speaker refinement network. The final output is a fusion of coarse and refined estimates calibrated on a development set. In the ISCSLP 2026 Real‑World AVSE Challenge, the system tied for first place on Track 1 and ranked second on Track 2, achieving the lowest character error rates of 12.31 % and 14.79 % respectively.
Hugging Face Trending Papers
Oct 15
CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge
Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage A...
arXiv Machine Learning
Sep 25
Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching
The paper adapts Reinforce Adjoint Matching (RAM) to generative speech enhancement, allowing a pretrained model to be post‑trained on real recordings using weak supervision such as text transcripts. RAM shifts the model’s conditional distribution toward higher‑reward outputs by generating enhanced speech on‑policy, evaluating each output with a potentially non‑differentiable reward, and analytically re‑noising the endpoint to create inputs for a reward‑guided regression objective. Experiments on real CHiME‑4 recordings show a 5.08‑percentage‑point reduction in word error rate compared to the pretrained FlowSE model, while maintaining all reported non‑intrusive speech quality metrics and receiving no significant preference in a subjective listening test.