DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
Modern RL post-training methods such as GRPO and DAPO train on response sequences of tokens sampled from a shared prompt of tokens, but standard FlashAttention replicates all prompt tokens times across both forward and backward passes -- duplicating compute and memory on identical hidden states. In large-rollout, long-context RL training (, ), this redundancy dominates the policy update cost. We observe that in decoder-only models, causal masking makes prompt representations invariant across sequences at every layer, so all per-token operations (norms, projections, MLP) and attention can process the prompt once -- a property not yet exploited at the kernel level for training. We propose \textbf{DualKV}, the first FlashAttention kernel variant that eliminates shared-prompt replication during RL training, via (1)~fused CUDA forward and backward kernels that iterate over two disjoint KV regions -- shared context and per-sequence response -- in a single kernel launch, and (2)~a data-pipeline redesign in veRL that repacks tokens into tokens per micro-batch, extending the token reduction from attention to the entire model by a factor . DualKV is mathematically equivalent to standard attention and introduces no approximation. On Qwen3-8B GRPO training with 8H100 GPUs (, 8K-context), DualKV achieves -- policy-update speedup, enables larger micro-batches, and raises MFU from to . Similar gains hold for DAPO ( speedup, MFU). At 30B MoE scale on 16H100, DualKV achieves policy-update and end-to-end step speedup over FlashAttention (which requires 4-way Ulysses sequence parallelism to avoid OOM).