What Is the Context Window in AI?

The context window is the single most practical concept in AI that most people learn about by hitting its limit instead of reading about it first.

I did the same. I pasted a long document into Claude, got a truncation warning halfway through, and spent twenty minutes wondering why the model kept forgetting what I'd written at the top. That confusion prompted me to dig into how context windows actually work - and what I found changed how I use every AI tool I test.

This guide is the explanation I wish I'd had before that moment.

What Is the Context Window?

The context window is the maximum amount of text - measured in tokens - that an AI model can process in a single request, including both the input you provide and the output the model generates.

Think of it as the model's working memory. Everything the model can "see" and reason about in one session lives inside that window. Anything outside it is invisible.

When you send a message to an AI, you're not talking to a system with persistent memory by default. You're handing it a document. That document is your entire conversation history, any files you've attached, any system instructions from the application, and your current message - all stacked together. The context window is the maximum size that document can be.

This matters immediately for practical work. If your window is 128,000 tokens and your conversation history plus your new request uses 130,000, the model either refuses to respond or silently drops some of the earlier content - depending on how the app is built.

The word "token" is worth pausing on. Tokens are not exactly words. They're chunks of text that can be a whole word, part of a word, or a punctuation mark. As a rough rule, 1,000 tokens is approximately 750 words in English. The tokenization system varies slightly by model, which is why context limits are always stated in tokens rather than words.

Context Window Sizes Across Models in 2026

Context window sizes have expanded faster than almost any other AI benchmark over the past three years - from GPT-3's 4,096 tokens in 2020 to Gemini 1.5 Pro's 1 million tokens today.

Here's where the major models sit as of mid-2026:

The table looks clean, but the practical differences are sharper than the numbers suggest.

A 200K window means you can paste an entire novel, a full codebase, or 40 research papers into a single session and ask questions across all of it. A 128K window is generous for most tasks but hits limits when you're doing document-heavy legal or financial analysis.

I'll say plainly: the 1M token window from Gemini is impressive on paper, but in testing for our 2026 AI tools reality check, I found that throwing 800K tokens at a model doesn't automatically produce better answers. More on that in the section below on why bigger isn't always better.

For most business users trying to choose an AI model for their work, the 128K-200K range covers the overwhelming majority of real tasks.

What Actually Fits in a Context Window?

The context window is not an abstract concept - it maps directly to concrete volumes of text, code, and documents you work with every day.

Here's a practical breakdown based on the rough conversion of 1,000 tokens = approximately 750 words:

Text documents

• 128K tokens holds about 96,000 words - a full-length novel like The Great Gatsby with room to spare.
• 200K tokens holds roughly two novels back to back, or a full PhD thesis.
• 1M tokens holds approximately 10 full novels.

Code files

• A single Python file is typically 200-2,000 tokens depending on length.
• A complete medium-sized codebase (10,000 lines of code) runs roughly 30,000-50,000 tokens.
• A large production repo with 100,000+ lines can exceed even a 200K window.

PDFs and reports

• A 20-page research report is typically 10,000-15,000 tokens.
• A 100-page financial filing is roughly 60,000-80,000 tokens.
• A full 300-page annual report will likely approach or exceed 128K on its own.

The tricky part is that you're not just fitting your document - you're fitting the document plus all prior conversation turns plus the model's output.

One pattern I see consistently when testing AI code assistants: developers underestimate how fast context fills up during a long coding session. Each file you paste, each error message you include, each explanation the model generates - it all accumulates. A 128K window can fill up in under an hour of intensive coding work.

Understanding how tokenization works helps you estimate your usage more precisely. Some languages are more token-efficient than others - English prose runs around 1.3 tokens per word, while code with lots of indentation and special characters can run higher.

Why Bigger Isn't Always Better

A larger context window sounds like an unambiguous improvement. After extensive testing, I can tell you it's more complicated than that.

The core problem is what researchers call the "lost in the middle" effect. Studies have shown that models often perform worse at retrieving information that appears in the middle of a very long context compared to information at the beginning or end.

This is counterintuitive. If I hand you a 1,000-page document and ask you about page 500, you don't get an answer from a model with perfect 1M-token recall - you get an answer from a model that has mild attention drift in the middle of its context.

There are a few other practical downsides worth knowing:

Cost scales with tokens. Most AI APIs price per token - both input and output. If you're stuffing 900K tokens into every request, the cost of each call can be 7-10x higher than using a 128K model for the same task. Running cost estimates before committing to a workflow is worth doing - the AI tools cost calculator can help with this.

Latency increases. Processing 1 million tokens takes more time than processing 128K tokens. For interactive use where you want a fast response, a massive context can slow things down noticeably.

Attention quality isn't uniform. The transformer architecture uses attention mechanisms that have to distribute their capacity across all tokens in the window. More tokens means each individual token gets slightly less attention. For most tasks this doesn't matter - but for tasks requiring precise recall of specific details buried in a huge document, it can.

My honest take after a year of testing: for most knowledge work tasks - summarizing a report, drafting based on a brief, reviewing a codebase - 128K to 200K is enough. The 1M window matters when you need everything in one request - like analyzing an entire legal case file or indexing a large codebase without RAG infrastructure.

If you want to go deeper on the actual research: the original "Lost in the Middle" paper from Stanford is available on arXiv and is readable even without a machine learning background.

How I Hit the Context Limit and What Broke

I want to share a specific incident because it illustrates something the documentation doesn't tell you.

I was using Claude to refactor a medium-sized Python project - about 3,500 lines across 12 files. I pasted all the files at the start of the session with detailed instructions about architecture and naming conventions. For the first 45 minutes, the outputs were excellent.

Then I noticed something odd. The model started ignoring constraints I'd set at the beginning.

It began using snake_case for variables in places where I'd explicitly specified camelCase. It started adding imports I'd said to avoid. When I pointed this out, it acknowledged the instruction - then repeated the mistake on the next file.

I assumed the model was hallucinating or drifting. It took me another 30 minutes to realize what was actually happening: I'd passed the 128K context limit. The application was using a sliding window approach, which meant the earliest messages - including my initial architecture instructions - were being silently dropped to make room for the growing conversation.

The model wasn't ignoring me. It literally couldn't see my original instructions anymore.

This experience is what I call "the context cliff." The degradation isn't abrupt - it's gradual and subtle, which makes it much harder to catch than a hard error. If you're doing any long-form work with AI agents, this is a failure mode worth planning for explicitly.

After that incident, I started tracking context usage explicitly during long sessions. Most frontends show a token counter - start watching it around 70% fill. That's when I now pause to either summarize earlier context or start a fresh session.

For users who've hit similar walls with tools like Cursor or other AI code assistants, the solution is usually the same: shorter sessions with clearer handoffs, not bigger windows.

Context Window vs Memory vs RAG

This is the section that took me the longest to get right, because I conflated all three terms for months.

The context window, memory, and retrieval-augmented generation solve different versions of the same underlying problem: how does an AI model know things?

Let me separate them clearly.

The context window is what the model can see right now, in this request. It's transient. When the session ends, it's gone. It's like the model's desk - things on the desk are accessible instantly, but nothing gets filed away automatically.

Memory refers to mechanisms that persist information across sessions. Some AI products (ChatGPT with memory enabled, Claude with Projects, etc.) save facts from past conversations and inject them back into future sessions. This is application-level behavior, not a property of the underlying model. The saved memories eventually get added to the context window of future requests - so memory is really "saved context that gets injected automatically."

RAG - retrieval-augmented generation - is a different architectural approach entirely. Instead of fitting all your documents into the context window, RAG indexes your documents externally, then retrieves only the most relevant chunks at query time and places those chunks in the context. It's like having a filing cabinet next to the model's desk - instead of reading all 500 files, you fetch only the three files most relevant to the current question.

The important insight here: RAG doesn't replace the context window. It manages what goes into it. If your codebase has 10 million tokens of content and your model has a 200K window, RAG decides which 200K tokens of that codebase are most relevant to the current question and places those in the window.

This is why RAG doesn't simply become obsolete because Gemini has a 1M token window. Fitting 1M tokens is expensive and slow. RAG is often cheaper and faster for large knowledge bases, even when the alternative would technically fit.

I've covered RAG in depth at what is retrieval-augmented generation if you want to go deeper on how the retrieval side works.

For context on how embeddings power the retrieval side of RAG, what is embedding in AI covers that clearly.

For business users deciding between approaches, how to build an AI tool stack has a section on when to architect RAG versus just extending context. And if you're thinking about data privacy implications of sending documents to large-context models, the AI privacy checklist is worth reading alongside this.

How to Work Efficiently Within Context Limits

Understanding the context window is only useful if it changes how you work. Here are the practices that have made the biggest difference in my day-to-day testing of AI tools.

Start sessions with anchored instructions. Put your most important constraints and context at the very top of the session, before any files or documents. If the context window fills up, apps typically drop content from the middle of the conversation history - not the very beginning. Your initial system-level instructions are the last thing to get cut.

Be selective about what you paste. Before pasting a 50-page document, ask whether you need all 50 pages or just the relevant sections. Trimming a document from 30,000 tokens to 8,000 tokens doesn't just save cost - it gives the model a cleaner signal with less noise to reason through. This connects directly to good prompt engineering practice: less is often more.

Use explicit handoff prompts when resetting. When you need to start a new session to clear the context, write a concise summary of everything important before you go. I keep a running list during long sessions: key decisions made, constraints established, outstanding questions. A good handoff prompt might run 500-800 tokens and can replace 20,000 tokens of conversation history.

Watch the token counter. Most AI frontends show you current context usage. I set a mental checkpoint at 70% - at that point I evaluate whether to continue or summarize and reset. Waiting until 95% means you're reacting to drift rather than preventing it.

Structure your prompts to put critical info near the top and bottom. Given the lost-in-the-middle effect, if you have to include a long document, put your specific question before it and your most critical constraints after it. Sandwiching the document between your instructions helps the model attend to both.

For recurring workflows, use system prompts properly. If you use an AI tool daily for the same type of task, invest time in writing a clean system prompt that stays at the top of every session. Good prompt engineering here multiplies across every session you run.

One more thing worth adding: context management is one of the biggest differentiators between AI tools that feel productive and ones that feel like fighting a system. When I'm comparing tools for our best AI agents roundup, how gracefully a tool handles context limits - whether it warns you, summarizes automatically, or lets you set up persistent system prompts - is a serious evaluation criterion.

The best tools I've reviewed make context management nearly invisible. The worst ones let you discover the limit by producing subtly broken outputs. If you're evaluating tools for your team and context management matters to your workflow, the methodology page explains how we test for this specifically.

If you're choosing between models for a business use case that involves long documents, how to evaluate AI output quality covers the specific tests I run to check for context degradation before recommending a tool.

Frequently Asked Questions

What is the context window in simple terms?

The context window is the maximum amount of text an AI model can read and reason about in a single session. Think of it as the model's short-term memory - everything it can "see" at once. Anything outside the window is invisible to the model.

How many tokens are in a typical context window?

As of mid-2026, most frontier models offer between 128,000 and 1,000,000 tokens. GPT-4o and most Llama variants sit at 128K, Claude 3.5 Sonnet offers 200K, and Gemini 1.5 Pro and Flash both offer 1 million tokens. Smaller and older models may have 4K-32K windows.

Does the context window reset between conversations?

Yes. By default, every new conversation starts with an empty context window. Anything from a previous session is gone unless the application has an explicit memory system that injects saved information into new sessions.

What happens when you hit the context limit?

It depends on the application. Some will refuse to process your request and show an error. Others use a sliding window - dropping the oldest messages silently to make room. A few will summarize earlier content automatically. The most dangerous behavior is the silent sliding window, because the model keeps responding but has lost earlier context without telling you.

Does a bigger context window cost more?

Yes, in most cases. API pricing is typically based on input and output tokens, so a request using 500K tokens costs significantly more than one using 50K tokens. For applications that use large context windows in every request, the costs can add up quickly. The cost calculator tool can help you model this for your specific use case.

Is the context window the same as the model's memory?

Not exactly. The context window is temporary and resets each session. Memory, in the product sense, refers to information that persists between sessions and gets injected into future context windows. Memory is application-level behavior built on top of the context window, not a replacement for it.

What is the "lost in the middle" problem?

Research has shown that AI models recall information from the beginning and end of a long context more accurately than from the middle. When you fill a 200K context with a very long document, the model may give less accurate answers about content that falls in the middle portion of the document. This is one reason why a massive context window doesn't automatically produce better results.

What's the difference between context window and RAG?

The context window is how much text the model can see at once. RAG - retrieval-augmented generation - is a technique for managing what gets placed in that window. Instead of stuffing an entire knowledge base into the context, RAG retrieves only the most relevant chunks at query time. They complement each other rather than competing.

How does the context window affect AI agents?

Context management is one of the core challenges in building AI agents that run long tasks autonomously. As an agent executes multiple steps, its conversation history grows. Without careful context management, long-running agents hit limits and lose track of their initial instructions. The model context protocol is one architectural approach to addressing this for agentic systems.

Can I increase the context window for a model I'm using?

No - the context window is a fixed property of the model architecture and is set at training time. You can choose a model with a larger context window, but you can't expand the window of an existing model. What you can do is manage your use of the available window more efficiently, or implement RAG to work around the limit for large document sets.

How do I check how much context I've used?

Most AI frontends show a token counter or context usage indicator somewhere in the interface. If you're using the API directly, the response object includes token usage data. For long sessions, I recommend checking your usage at regular intervals - most applications start having issues at 80-90% of the context limit, not just at 100%.

What context window size do I actually need?

For most knowledge work - drafting, summarizing, answering questions from a report - 128K is sufficient. If you regularly work with very long documents (100+ pages), 200K becomes valuable. Only if you need to analyze entire codebases or large document sets in a single request without RAG infrastructure do you need 1M tokens. For help thinking through your specific use case, the AI tools quiz asks the right questions to match you with an appropriate tool.