F2T: Fast Failover in LLM Training with Near-Zero Overhead State Management

Recent advances in large language models (LLMs) have imposed new demands on the efficiency and resilience of large-scale training systems. As LLM clusters continue to scale, node failures, lengthy recoveries, and bulky checkpoints erode efficiency. Infrequent asynchronous checkpoints trigger costly rollbacks, yet higher frequencies add prohibitive overhead. To address these challenges, we propose F2T, a system for robust LLM training. F2T leverages surplus network capacity to quickly save and load states, thereby preventing rollbacks and accelerating recovery. Compared with prior checkpointing approaches, F2T reduces recovery time by up to 98% and decreases GPU utilization loss by up to 68%, all without affecting normal training throughput.

An earlier version of this work appeared as a preprint: FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management.

Recommended citation: Bohan Zhao, Yuanhong Wang, Chenglin Liu, Jiaqi Pan, Guang Yang, Ruitao Liu, Tingrui Zhang, Kai Luo, and Wei Xu. (2026). "F2T: Fast Failover in LLM Training with Near-Zero Overhead State Management." In Proceedings of the World Artificial Intelligence Conference Academic (WAICA 2026).