arXiv Machine Learning By Daehwa Ko, Jaehyeon Kim, Seunghyun Ham, Jay Hoon Jung

TESLA: Taylor Expansion of Sinusoidal Learnable Activations

Read the original on arXiv Machine Learning →

arXiv:2608. 11970v1 Announce Type: new Abstract: The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
5d ago

Stable initialization without the CLT

The paper introduces a new method called uniform‑phase initialization for deep neural networks with sine activations, eliminating the need for the Central Limit Theorem and fully decoupling layers. This approach avoids distributional approximation errors and coupling between layers, leading to stable weight initialization. Experiments show that models using this initialization outperform state‑of‑the‑art methods on image and audio fitting tasks and remain competitive without tuning, while also supporting μP width scaling.

By Simon Kuang, Kyle Chickering, Xinfan Lin