CAG vs Long Context: How AI Models Use and Remember Information

Summary

The article compares two strategies for injecting external knowledge into large language models (LLMs). One approach relies on long-context prompting, which involves fitting all documents into the context window. The other is Cache Augmented Generation (CAG), a method that pre-computes and reuses the transformer's key-value (KV) cache. This allows repeated queries to be answered without re-reading the same documents. The article positions CAG as a practical, provider-supported optimization that operates via prompt caching. It can reduce inference costs by up to 90% for static knowledge bases and improve response speeds by 10 to 40 times.

Why it matters

As context windows expand toward millions of tokens, the 'paste everything' approach is becoming unsustainable. In high-volume workloads, such as HR chatbots or product documentation Q&A, this method is both prohibitively expensive and slow. Understanding when to recompute versus reuse KV states is now a fundamental architectural decision that directly impacts latency budgets and API spending. It has become a practical priority for every team developing LLM-based features.

Highlights

• Context windows have grown from approximately 1K tokens in GPT-3 (2020) to 2M tokens in Gemini 1.5 Pro (2024). This expansion enabled the 'skip retrieval entirely' strategy. However, the token cost per query increases linearly with document size.
• CAG's three-stage pipeline (prepare → pre-compute KV cache → load cache and respond) delivers 10x speedups on small datasets and up to 40x on large ones. These figures are achieved compared to reprocessing the entire context for every request.
• Major LLM providers now offer prompt caching as a managed service. This effectively means 'CAG as a Service,' providing approximately 90% discounts on tokens read from the cache and eliminating the need to build custom caching infrastructure.

Critical take

The article leans too heavily on the binary framework of long context versus CAG, overlooking scenarios where RAG remains superior. For instance, RAG is more effective for very large or rapidly changing corpora that do not fit into a single window. Additionally, the claim of a 40x speedup cites an unsourced 'research' figure. The 'lost-in-the-middle' problem is also treated merely as a footnote, despite being a primary accuracy risk. If critical policy clauses are buried in the middle of a 200K-token context, this issue becomes vital.