Full-Stack FP4: Stable LLM Pretraining

Pretraining large language models (LLMs) has reached an inflection point where computational efficiency and numerical precision must coexist in a delicate balance. Traditionally, 4-bit quantization has been successfully applied to linear layers in transformers, but it left out critical components such as optimizer states, optimizer arithmetic, and attention. This gap prevented complete and stable 4-bit pretraining, as each module presents very specific numerical failure patterns: linear layers suffer from error amplification due to dimensional propagation; the second moments of AdamW are non-negative, heavy-tailed values that are extremely fragile with low-precision denominators; and attention contains error-prone computation paths that demand strict consistency between forward and backward passes. In response, a comprehensive proposal called Full-Stack FP4 emerges, solving all three bottlenecks through modular precision strategies. For linear projections, a lightweight LoRA-SVD decomposition is used to reduce quantization noise and break the error ceiling of direct quantization, reducing the linear loss gap from 1.40% to 0.61%. For optimizers, a transformation of AdamW's second moments is designed to enable their robust storage in NVFP4, and native NVFP4 Newton-Schulz iterations are implemented for the Root (Muon) optimizer. In attention, a mixed-precision scheme quantizes Q/K/V and the backward dS while protecting vulnerable paths in BF16, accompanied by unified tensor reuse to maintain forward-backward alignment. The analysis also reveals rapid error accumulation in low-precision matrix multiplication and extreme sensitivity of the PV/dOV^T branches. All modules are pluggable and offer cumulative improvements in stability and efficiency. Validations with 3B parameter pretraining and 64B tokens demonstrate nearly identical performance to BF16, with only a 1.47% loss gap, verifying the feasibility of complete and stable NVFP4 pretraining for LLMs. This innovation has direct implications for enterprise-scale artificial intelligence development. At Q2BSTUDIO, we understand that optimizing language models requires not only cutting-edge algorithms but also robust, customized infrastructure. That is why we offer AI for businesses that integrates quantization techniques and efficient deployment, tailored to each organization's specific needs. Additionally, our custom software solutions allow implementing training and fine-tuning pipelines with full control over precision and performance. If your company needs to maximize cloud resources, our AWS and Azure cloud services facilitate horizontal scaling of GPU clusters and storage of quantized models. Cybersecurity also plays a key role: by reducing memory and bandwidth footprint, FP4 models minimize attack surfaces in multi-tenant environments. Likewise, we deploy AI agents that use optimized attention techniques, and offer business intelligence services with Power BI to visualize model performance metrics in production. Ultimately, the evolution toward complete FP4 pretraining not only accelerates academic research but also paves the way for companies to adopt LLMs with reduced costs and latencies, without sacrificing quality. At Q2BSTUDIO, as a software development and technology company, we accompany our clients at every stage: from conceptualizing custom applications to cloud deployment, ensuring that innovation in artificial intelligence translates into real competitive advantages.