Context Rot

What Context Rot Means

Context rot is an informal name for the decline in a language model's ability to use relevant information as its supplied context becomes longer, noisier, or more complex. The information may still fit inside the context window and remain visible to the model, yet answers become less reliable.

This is not literal decay inside the prompt. It describes an observed performance pattern, not one universal failure curve or a standardized model property.

Core Idea

  1. A short prompt contains a clear signal with few competing details.
  2. More documents, messages, and instructions add potentially useful information.
  3. They also add distractors, conflicts, and relationships the model must resolve.
  4. The target fact becomes harder to retrieve and apply consistently.

Why It Happens

Position Effects

Models do not always use information equally across every position. Research has found that relevant content placed near the middle of a long input can be used less reliably than content near the beginning or end. This behavior is commonly described as lost in the middle.

Distractor Interference

Irrelevant or similar-looking passages compete with the target evidence. The model may retrieve the wrong detail, merge separate details, or answer from a strong learned prior instead of the supplied source.

Conflicting Context

Long conversations and document collections can contain outdated statements, duplicated facts, or incompatible instructions. The model must infer which source is authoritative and current.

Multi-Step Composition

A task may require finding several facts and combining them correctly. Even when each fact can be retrieved alone, reliability can fall when the model must track dependencies across a long sequence.

Context Rot vs. Overflow

Failure What happens
Context overflow The request exceeds the supported token limit and must be rejected or shortened
Truncation Part of the context is deliberately removed to fit the limit
Context rot The content fits, but the model uses it less reliably
Hallucination The model produces unsupported or false content; context rot can contribute to it

Reducing the Risk

  • Retrieve a small set of high-relevance passages instead of including everything available.
  • Remove duplicate, stale, and contradictory material before prompting.
  • Put critical instructions and evidence in clear, labeled sections.
  • Break complex work into stages and carry forward structured intermediate results.
  • Use summaries for navigation, while preserving source text for details that require exactness.
  • Evaluate retrieval and reasoning at several lengths, positions, and distractor levels.

The goal is not always to maximize the amount of context. It is to supply the smallest context that contains enough trustworthy evidence for the task.

Evidence

Related