What Happens When You Exceed the Context Limit

How Each Provider Handles Overflow

OpenAI (GPT-4o)

OpenAI returns an HTTP 400 error with the message "This model's maximum context length is 128000 tokens. However, your messages resulted in X tokens." The API does not truncate or modify your input. Your application must handle this error and reduce the context before retrying. If you are using the Chat Completions API directly, this error crashes your request. If you are using a framework like LangChain, the framework may handle truncation for you, typically by removing the oldest messages.

Anthropic (Claude)

Anthropic also returns an error when the input exceeds the context limit. The error specifies the token count and the maximum. No silent truncation occurs at the API level. However, applications built with the Anthropic SDK can implement their own truncation logic, and some do so by default. If you are seeing unexpected behavior with long conversations on Claude, check whether your application framework is truncating messages before sending them to the API.

Framework-Level Truncation

Many LLM application frameworks (LangChain, LlamaIndex, Vercel AI SDK) implement automatic truncation to prevent API errors. The typical behavior is to remove the oldest messages from the conversation until the total fits within the context limit. This prevents errors but creates a different problem: the model loses context from earlier in the conversation, potentially including the system prompt, important decisions, or established preferences.

The worst outcome is when framework truncation removes the system prompt. Without the system prompt, the model reverts to its default behavior, ignoring all the instructions, guardrails, and formatting requirements you defined. The user sees a response that is technically correct but ignores the application's personality, constraints, and output format. This failure mode is common in chatbots with long conversations and is one of the most reported "the AI forgot how to behave" complaints.

The Failure Modes

Silent Truncation

The most dangerous behavior because there is no error, warning, or indication that information was lost. The model generates a response as if everything is fine, but it is working with incomplete context. Users do not know that earlier conversation context was removed. Developers do not know that the system prompt was truncated. The failure manifests as degraded quality that is difficult to reproduce and diagnose.

API Error

Less dangerous but still disruptive. The user sees an error message or a failed request. If your application does not have retry logic with context reduction, the conversation is stuck. The user cannot continue without starting a new conversation or manually shortening their messages.

Degraded Quality Without Error

Even before hitting the hard limit, context pressure can degrade quality. As the context fills, the model has less capacity for its response, and the quality of attention on each piece of context decreases. A conversation at 95% context utilization produces measurably worse responses than the same conversation at 50% utilization, even though no error occurs.

Prevention Strategies

The right approach is to make overflow structurally impossible. Count tokens before every API call. Set a budget that is 80% of the model's limit. When the count approaches the budget, apply reduction strategies: summarize old conversation history, reduce retrieved context to fewer results, or remove few-shot examples. The goal is that every API call is guaranteed to fit within the window.

For applications that need long-lived conversations or large knowledge bases, external memory eliminates the overflow risk entirely. Persistent knowledge lives outside the context window and is retrieved on demand. Conversations can run indefinitely because old messages are stored in memory rather than accumulated in the prompt. The context window holds only the current interaction, which is always within budget.