MetroRLHF: Enabling Memory-Effective Training for On-Policy RLHF via Adaptive Sequence Streaming

Reinforcement learning from human feedback (RLHF) has become the standard post-training technique for endowing large language models (LLMs) with helpful, harmless, and intent-consistent behavior. In practice, however, its adoption is hampered by prohibitive memory consumption during the phase of the policy-model update, especially when training on long-form generation tasks. In this paper, we propose MetroRLHF, a memory-efficient, on-policy RLHF ap proach that exploits the inference-time computations to reduce the training-time memory budget and to skip unnecessary work. By re-using the inference-phase materialized K,V context, the inter-token dependencies are freely removed that normally force the entire sequence to train in parallel. Building upon fine-grained subsequence streaming, RLHF can train the productive tokens in an effective manner. This yields a training pipeline that matches the exact behavior of conven tional full-sequence RLHF while using less memory and incurring no arithmetic recomputation. Experiments on the Qwen-3 models demonstrate that MetroRLHF rescheduled algorithm reduces peak training memory usage to 1/3.8 ∼ 1/5.9, enabling not only memory-effective but also semantic-reliable fine-tuning for LLM.