Jason Eisner - ACL Anthology

Summary

Prompt Choreography introduces a dynamic global KV cache, enabling LLM calls to attend to reordered subsets of previously encoded messages; this enables parallel execution and significant latency reduction.

Why it matters

As LLMs become central to multi-agent workflows, unnecessary computations hinder scalability. By caching message encodings and allowing flexible attention patterns, the framework dramatically speeds up token generation, making large-scale deployments more feasible.

Highlights

• The dynamic KV cache enables any LLM call to attend to a reordered subset of previous messages.
• Supports parallel calls, reducing per-message latency by 2.0–6.2×.
• Fine-tuning the LLM to operate with caching helps preserve original outputs despite context changes.

Critical take

While caching boosts speed, shifting contexts can lead to divergent outputs; relying on fine-tuning to mimic original results introduces additional complexity and potential accuracy trade-offs.