AI Tokens and Context Windows Explained

You paste a long contract into an AI assistant, ask a question and get a great answer. An hour later, deep into the same chat, it seems to have forgotten a detail you mentioned at the start. Or your company’s AI bill arrives, and it is 3 times what you expected.

Both surprises come down to 2 concepts most people never get told about: tokens and the context window. Understanding them takes about 10 minutes and pays off every time you use AI, whether you type into a chat app or pay for API access. This guide explains both in plain English, with practical ways to save money and get better answers.

What a token is

AI language models do not read words or letters the way people do. They read tokens: small chunks of text that the model’s tokenizer splits your input into. A token might be a whole common word (“the”), part of a longer word (“context” and “ual”), a number, a space plus a word, or a punctuation mark.

OpenAI’s rule of thumb for English is that 1 token is about 4 characters, or roughly three-quarters of a word. That gives you quick estimates:

• 100 tokens is about 75 words, or a short paragraph.
• 1,000 words is about 1,300 tokens.
• A 300-page book of about 90,000 words is roughly 120,000 tokens.

These are averages, not guarantees. Each model family uses its own tokenizer, so the same text can produce different counts. Code, numbers, unusual names and many non-English languages often use more tokens per word than everyday English.

Images, audio and documents also become tokens. Multimodal AI models convert pictures into patches and audio into time slices, each counted as tokens. A large image can cost as many tokens as several pages of text.

What a context window is

The context window is the maximum amount of information, measured in tokens, a model can consider in a single request. Anthropic’s documentation describes it as the model’s “working memory,” separate from the huge body of data it was trained on.

Everything in the request counts toward the limit:

Context windows have grown dramatically. Early chat models handled a few thousand tokens. Today, several leading models advertise windows of 1 million tokens, which Anthropic lists for many of its current Claude models. That is enough for several long books at once.

Why AI chatbots “forget” things

Language models have no memory between requests by default. When you chat, the app resends the conversation history with every new message. That is how the model “remembers” what you said earlier.

When a conversation outgrows the context window, something has to give. Depending on the app, older messages may be dropped, summarized or compressed. Anthropic notes that chat interfaces can manage the window on a rolling “first in, first out” basis. Either way, early details can disappear, which is why a long chat can lose track of instructions you gave at the start.

Some assistants now offer separate “memory” features that save facts about you across chats. Those work by storing notes and inserting the relevant ones into the context window, so they still use tokens.

Bigger is not always better: context rot

A huge context window does not mean the model pays equal attention to everything in it. Research published in 2023, “Lost in the Middle” by Liu and colleagues, found that models answered questions best when the relevant information sat at the beginning or end of a long input, and worse when it was buried in the middle. Performance also declined as inputs grew longer.

Model makers have improved long-context handling since then, but the effect has not vanished. Anthropic’s documentation warns that as token count grows, accuracy and recall degrade, which it calls “context rot,” and says curating what goes into context matters as much as how much space is available.

Key takeaway: Give the model the right information, not all the information. A focused 5-page excerpt usually beats a 500-page dump, both in answer quality and in cost.

This is also why techniques like retrieval-augmented generation (RAG) remain popular even with giant windows. RAG searches your documents first and sends only the most relevant passages to the model.

How tokens affect your AI costs

If you use a subscription chat app, tokens mostly show up as usage limits: heavy use with long files may hit a cap sooner. If you use AI through an API, tokens are the bill.

API providers typically price per million tokens, with separate rates for input (what you send) and output (what the model writes). Output usually costs more. On Anthropic’s published price list, for example, output tokens cost 5 times as much as input tokens for many models. Prices change often and vary widely between models, so check the provider’s current pricing page.

The main cost levers:

Remember the hidden multiplier: in a chat, every new message resends the history. A 20-turn conversation with a long document attached can bill that document’s tokens 20 times over unless caching applies. For a fuller view of budgeting, see how much AI implementation costs.

How tokens affect output quality

Tokens shape answers in a few less obvious ways:

• Output limits. Models cap how many tokens they can write in one reply. Very long reports may need to be produced in sections.
• Reasoning tokens. Many newer models “think” before answering, producing internal reasoning that counts as output tokens. Harder questions can use far more tokens than the visible answer suggests.
• Spelling and counting quirks. Because models see tokens, not letters, tasks like counting letters in a word can trip them up.
• Noise drowns signal. Irrelevant pasted material can distract the model and increase the risk of hallucinations.

Where to start: 6 habits for smarter token use

1. Start fresh chats for new topics. Do not carry a long, unrelated history into a new task.
2. Put key instructions and facts at the start or end of long prompts, not buried in the middle.
3. Paste excerpts, not entire archives. Pull the relevant pages or sections first.
4. Ask for the format and length you need, such as “5 bullet points” or “under 200 words.”
5. Restate critical details in long conversations, or summarize the chat so far and start a new one with that summary.
6. Check token counts before big API jobs. Most providers offer a token-counting tool or return usage figures with each response, so you can estimate costs on a sample first.

Better prompting habits compound these savings. Our guide to prompt engineering covers more techniques, and the glossary defines related terms.

The bottom line

Tokens are the units AI models read, write and charge for, roughly three-quarters of an English word each. The context window is how many tokens a model can consider at once, covering your instructions, files, chat history and its reply. Bigger windows help, but focused inputs still produce better answers at lower cost. Manage what goes into the window, watch your output lengths and use caching or batching for repeat work, and you will get more from every AI tool you use.

Sources

• OpenAI Help Center: What are tokens and how to count them? — token-to-character and token-to-word rules of thumb for English
• Claude Platform Docs: Context windows — what counts toward the context window, 1M-token windows and context rot
• Claude Platform Docs: Pricing — per-million-token pricing, prompt caching and batch discounts
• arXiv: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — accuracy drops when relevant information sits in the middle of long inputs