GQAGrouped

Demystifying GQA — Grouped Query Attention for Efficient LLM Pre-trainingThe variant of multi-head attention powering LLMs like LLaMA-2, Mistral7B, etc.A “Group” of Llamas (Source — Image created by the author using Dalle-3)In the previous article on training large-scale models, we looked at LoRA. In this article, we will examine another strategy adopted by different large language models for efficient training — Grouped Query Attention (GQA). In short, Grouped Query Attention (GQA) is a generalization of multi-head…