Serving LLMs: vLLM, TGI, KV-cache, batching

Training teaches a model to predict the next token. Serving is everything that happens when users actually call it: scheduling thousands of concurrent requests, growing KV-cache tensors as sequences lengthen, batching incompatible shapes without wasting GPU RAM, and saturating tensor cores under real latency SLOs. Frameworks like vLLM (PagedAttention) and Text Generation Inference (TGI) exist because naive PyTorch loops cannot handle production load.