Tokenisation: BPE, SentencePiece and why it matters
An LLM does not read letters or words — it reads tokens. Before any text reaches the model, a tokeniser splits it into a sequence of integers from a fixed vocabulary. This invisible step shapes everything: how well the model handles arithmetic, how expensive a non-English prompt is, why GPT-4 cannot reliably count letters in a word.