Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu and 5 more
Who it is from
Georgia Institute of Technologyaffiliation declared by the authors, according to OpenAlex- Sichuan Universityaffiliation declared by the authors, according to OpenAlex
Other papers from the same organizations
- Evaluating large language models for accuracy incentivizes hallucinationsGeorgia Institute of Technology · 801 citations
- Foundation Models Defining a New Era in Vision: A Survey and OutlookGeorgia Institute of Technology · 733 citations
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM ReasoningAlibaba Group · 722 citations
- Small Language Models are the Future of Agentic AIGeorgia Institute of Technology · 628 citations
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeGeorgia Institute of Technology · 537 citations
- VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic NavigationGeorgia Institute of Technology · 502 citations