Dual-Flow Transformer activates additional computation only from the end of the text during response generation

Summary

Dual-Flow Transformers separate the prompt-filling computation from the auxiliary decoding task, sharing core weights and a single KV cache while only activating the auxiliary flow starting from the final prompt token.

Why it matters

This design tackles the growing cumulative inference cost of large language models by allocating extra computation to continuation prediction without increasing prompt processing costs. It enables fine-grained expert routing, improves verification loss across architectures, and provides independent control over prompt versus decoding resource usage—critical for scaling services that handle many requests.

Highlights

• The primary flow processes the entire prompt and writes a persistent KV cache; the auxiliary flow is skipped during filling and only activates from the final prompt token onward.
• The main attention mechanism, MLPs, and output matrices are shared across flows; this reduces weight loading costs and allows the keys/values in the cache to be reused during batched execution.
• In MoE models, this separation allows independent fan-out control for prompt and continuation experts; thus balancing filling cost against decoding quality.

Critical take

The added complexity of managing dual token embeddings and context mechanisms can increase the total parameter count and complicate training pipelines; this may offset some of the claimed efficiency gains.