.jpg)
Fine-tuning large language models has become an indispensable practice for adapting them to specific tasks, but it carries a silent risk: catastrophic forgetting of already acquired capabilities. Traditionally, preservation strategies have relied on proxies such as distance between parameters, importance penalties, or projection onto dominant singular directions of the weights. However, a more promising approach emerges when asking not about the geometry of the weight matrix, but about the activation directions in which the preserved capability manifests itself. This is where FORA (Function-space Orthogonal Residual Adaptation) proposes a paradigm shift: protecting the functional space of the network, that is, the subspace of activations that defines the behavior of a specific skill. Instead of merely restricting weights, FORA estimates, from unlabeled calibration inputs, the principal directions of the covariance of input activations per layer, and builds a right projector that blocks the reading of those functional directions during the update. This design allows a high-capacity channel that structurally cannot touch the relevant functions, along with a narrow spectral channel that provides controlled plasticity. Experiments on models such as Qwen3-1.7B demonstrate that this protection in function space clearly outperforms traditional projections onto weight space, with minimal loss in new task performance. The key is not the projection itself, but that the projection directions come from the activations of the capability to be preserved, not from the decomposition of the weights.
From a business perspective, this distinction is crucial. When an organization decides to fine-tune a base model for a vertical application —for example, a customer service assistant that understands both technical and colloquial questions—, it cannot afford to lose the general linguistic fluency the model already possesses. The solution is not simply freezing layers or applying generic regularization; an architecture is needed that understands what each capability does and how to protect it. FORA represents a significant advance in the development of what we could call more robust and reliable artificial intelligence for production environments. In this context, companies like Q2BSTUDIO, specialized in software development and technology, offer services ranging from custom applications to cloud and cybersecurity solutions, integrating these cutting-edge techniques into real projects. The ability to preserve prior knowledge while incorporating new skills is particularly relevant in custom software development for sectors such as banking, healthcare, or logistics, where models must adapt without forgetting regulations or historical contexts. Furthermore, FORA's functional approach opens the door to a new generation of AI agents that can learn additional tasks without losing their original identity, something that fits perfectly with Q2BSTUDIO's vision of offering business intelligence services and personalized AI agents, supported by infrastructures such as AWS and Azure cloud services and analysis tools like Power BI.
FORA's methodology also invites us to reflect on model engineering in business environments. It is not enough to fine-tune; one must understand what is being protected. The concept of activation subspaces offers a useful metaphor: each model capability occupies a region in the representation space. If we want to add a new one, we must do so in a direction orthogonal to that region, not simply reduce the magnitude of changes in the weights. This idea has direct applications in process automation, where a model trained to classify legal documents could be extended to also handle invoices, as long as the new learning does not interfere with already assimilated laws and regulations. The practical implementation of these approaches, however, requires qualified technical support. Therefore, alliances with companies like Q2BSTUDIO make it possible to integrate these innovations into custom software projects, ensuring that each layer of protection —whether in function space or cloud infrastructure— is deployed with the best cybersecurity and data governance practices. Ultimately, FORA is not just an academic advance; it is a conceptual tool that, when well applied, can transform the way companies develop and maintain their artificial intelligence systems, turning continuous learning from a threat into a strategic opportunity.
