The transformer's self-attention mechanism is what makes LLMs so powerful — every token can attend to every other token in the context. But there is a price: the attention matrix grows with the square of the sequence length. Doubling the context window does not double the cost — it quadruples it. This lesson covers the engineering breakthroughs that made 128K and 1M-token context windows practical.