Anyone who dives into AI will sooner or later come across the term tokens. In documentation, in conversations with vendors, in discussions about what a model can and cannot do. Yet few people outside of developers know exactly what a token is.<\/p>\n\n\n\n
That is a shame, because tokens explain a large part of the behavior that users actually notice: why an AI agent cannot oversee a long document all at once, why an answer trickles in word by word, and why Dutch \u2018weighs heavier\u2019 than English.<\/p>\n\n\n\n
A language model reads neither letters nor words. It reads tokens: pieces of text into which the text has been pre-divided. A token is usually a short word, a part of a word, or a punctuation mark.<\/p>\n\n\n\n
Short, common words like \u201cde\u201d, \u201cen\u201d, or \u201cniet\u201d are a single token. A longer or rarer word is cut into pieces. \u201cArbeidsongeschiktheidsverzekering\u201d is one concept for a human, but for a model, it is a series of around ten separate pieces. Spaces and punctuation marks also count.<\/p>\n\n\n\n
As a rule of thumb: one token is about four characters. In English, that comes down to roughly three-quarters of a word per token. In Dutch, that ratio is less favorable, and we will get back to that in a moment.<\/p>\n\n\n\n
The processing of an AI conversation takes place in three steps.<\/p>\n\n\n\n
First, everything the model is given to read is broken down into tokens. This includes not only the user's question, but also the instructions that determine the agent's behavior, the previously asked questions in the same conversation, and the documentation retrieved from the knowledge base. Together, this is called the input.<\/p>\n\n\n\n
Next, the model repeatedly predicts the most likely next token, given everything it has seen so far. After that, it looks again, including the token it just created itself, and predicts the next one. This is how the answer builds up token by token. Exactly for this reason, you see an answer appear on your screen as if someone is typing: it is truly being generated piece by piece at that very moment.<\/p>\n\n\n\n
Finally, those tokens are merged back into readable text, or in the case of a voice agent, passed on to the speech synthesis.<\/p>\n\n\n\n
One important limitation applies: the context window. This is the maximum number of tokens that a model can process at once, and you can think of it as the agent's working memory. Everything that fits inside it can be taken into account. Everything that does not fit inside it does not exist for the model at that moment. An agent that seems to forget the initial agreements halfway through a long conversation does not have a bad memory: its working memory is full.<\/p>\n\n\n\n
The way text is split into tokens is learned from large amounts of text, most of which is in English. As a result, English words are stored efficiently, while Dutch words are more likely to be split into pieces.<\/p>\n\n\n\n
Then there is our language itself. Dutch has long compound words that consist of separate words in English. \u201cVerzuimbegeleiding\u201d, \u201cziekmeldingsprocedure\u201d and \u201carbeidsdeskundig onderzoek\u201d are split into multiple fragments, while their English counterparts often require fewer tokens. Therefore, the same message takes about one and a half times as many tokens in Dutch as it does in English.<\/p>\n\n\n\n
That has three practical consequences. The context window fills up faster, meaning less knowledge can be provided at the same time. Processing takes slightly longer, which is barely noticeable in a chat conversation but definitely audible in a phone call. And the required computing power is higher.<\/p>\n\n\n\n
An organization provides its entire forty-page absence protocol as a source for the AI agent. That document does not fit into the model's working memory all at once. By splitting the protocol into logical pieces and retrieving only the relevant passage per question, the agent gets exactly the context it needs and nothing more.<\/p>\n\n\n\n
A voice agent reads a response consisting of six sentences instead of two. As a result, the caller not only waits longer for the end of the response, but also for the beginning of it, because the model has to generate more tokens. In speech, therefore, brevity is literally faster.<\/p>\n\n\n\n
An agent is equipped with a comprehensive instruction detailing every conceivable scenario. That instruction is resent with every single question and fills the working memory even before the user has said anything. Precise phrasing yields more results here than being exhaustive.<\/p>\n\n\n\n