r/MachineLearning • u/Justgototheeffinmoon • 1d ago
Research Sliding-window attention beats linear on long-context reasoning [R]
Sliding Window Attention with sinks, one of the simplest existing fixes for the quadratic-cost problem in LLMs, holds up as well or better than the linear-attention variants labs have been spending post-training compute to produce. That is the claim of a [new arXiv preprint](https://arxiv.org/abs/2608.28444) by Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron and Emy Gervais.
On the long-context reasoning benchmarks the paper singles out, the gap is not close. "SWA achieves massively higher performance (2 to 10 times higher than linear attention)," the abstract reports, naming Needle-in-a-Haystack and BABILong as the two tasks.
The pitch is that the whole post-training-to-linear pipeline has been benchmarked against the wrong thing. "This line of research has not been properly compared to simpler baselines," the authors write. Their alternative needs no post-training, runs fast, and holds memory low.
The recommendation is blunt: "we strongly recommend switching to SWA instead of post-training linear models." Linear attention, the abstract concedes, "may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA."
---
1
42
u/sqweeeeeeeeeeeeeeeps 1d ago
IMO this title is extremelydisingenuous. SWA does not beat linear in long context reasoning when trained from scratch. The setting they used is transformer-to-SSM distillation (which has a known failure mode of long context performance). This says more about the training recipe than the model architectures.