.jpg)
In the world of modern machine learning, optimizing large language models (LLMs) is both a computational and theoretical challenge. Techniques like GaLore have popularized the idea of projecting gradients onto a low-rank (rank-r) subspace that is periodically recomputed, under the premise that this subspace evolves slowly and can be tracked. However, recent research reveals a more complex reality: beyond a very small reproducible core, no such stable subspace exists. Comparing top-r subspace estimates obtained at the same step from disjoint minibatches shows a discordance nearly identical to that observed between estimates separated by T steps, indicating that the apparent rotational movement at each update is dominated by estimation noise, not by an underlying drift. This phenomenon replicates across multiple model families, from 70M to 6.9B parameters, and intensifies with size, also affecting vision architectures. Only a fraction of the subspace directions is reproducible between minibatches, and averaging fails to recover the rest: the spectral tail of the gradient decays as N^{-1/4} instead of the expected N^{-1/2} for pure noise, making it impossible to define a well-conditioned subspace no matter how much averaging is invested.
The solution is not to try to improve subspace estimation, but to treat each update as a coordinate change for the optimizer state. In particular, inertial transport of the second-order moment is suboptimal, while the first moment can be exactly transported via rotation. This is the principle used by LDAdam, an optimizer that achieves superior results (18.7 perplexity on a 1B model with 40k steps, compared to 19.3 for GaLore with its best fixed configuration). The fundamental lesson is that the non-identifiability of the subspace (measurable through the reproducible rank k*) explains why GaLore works despite seemingly erroneous assumptions and, above all, which patches are effective and what to verify before trusting a low-rank approximation.
This understanding has practical implications for any company developing or deploying large-scale language models. At Q2BSTUDIO, as a company specialized in software development and technology, we understand that efficient optimization is key for artificial intelligence projects. Our team integrates these advances into the creation of custom applications and AI solutions for businesses, ensuring that models are not only accurate but also computationally sustainable. For example, when designing AI agent systems or integrating AWS and Azure cloud services, we consider the most appropriate optimization profile based on the architecture and data, avoiding fragile assumptions about subspace stability.
Furthermore, in environments where cybersecurity is critical, such as in sensitive data processing, optimization errors can propagate into incorrect decisions. Therefore, we offer pentesting services and assurance for AI pipelines. Also, for projects requiring business analysis, our business intelligence solutions with Power BI integrate predictive models that must be trained with robust optimizers. Cloud infrastructure management, whether with AWS or Azure, allows us to scale these trainings efficiently. Ultimately, research on subspace non-identifiability reminds us that in AI, it is not enough to apply trendy techniques: one must understand their fundamentals and adapt them to the specific context of each client. At Q2BSTUDIO, we offer custom software and artificial intelligence consulting so that each project benefits from the most robust innovations, without compromising reliability or performance.
