Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

arXiv:2608. 10288v1 Announce Type: new Abstract: The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

• llms