What is Token (LLM)?
Turkish: Token (Belirteç)
A token is a word, subword, character, or symbol unit that language models process, shaping context size, cost, and speed.
What is Token (LLM)?
A token lets a model process text as smaller units rather than as plain words. One word can become multiple tokens, and punctuation, numbers, and spaces can be handled differently depending on the tokenizer.
In LLM services, pricing, context limits, and speed are often calculated through tokens. The user input, system instruction, retrieved documents, and model answer all consume the total token budget.
Why It Matters
A context window is limited by token capacity. Sending a very long history or unnecessary documents can leave no room for the evidence that matters. Short, well-structured prompts can reduce both cost and latency.
Token accounting becomes important in RAG, bulk document summarization, customer support assistants, and API-based content generation. The same task may use a different number of tokens with another model or another language.
Business Use
Production systems should monitor token usage. Unexpected cost increases can come from overly long system prompts, repeated conversation history, or uncontrolled document insertion.
Good design evaluates summarization, chunk selection, answer length limits, and model choice together.
Related Terms
A context window is the total token capacity a language model can read and consider while generating one response in a single request.
Fine-tuningFine-tuning retrains a pre-trained model on selected examples so it behaves more consistently for a task, tone, or domain.
LLM (Large Language Model)An LLM is a model trained on large text datasets that can understand and generate natural language, forming the basis of tools like ChatGPT.