In the fast-paced world of artificial intelligence development, one of the most persistent challenges is how to teach a large language model (LLM) multiple skills without it losing the ones it already learned. The traditional approach, known as sequential fine-tuning, often causes catastrophic forgetting: the model specializes in the latest task and degrades performance on previous ones. Facing this limitation, an elegant and efficient proposal emerges: multi-task distillation with soft prompts, a technique that allows a single 'student' model to learn from several 'teachers' represented by learnable soft prompts, all without modifying the base model's weights.
The central idea is simple yet powerful. Instead of fine-tuning the model's weights for each task (which is costly and causes drift), a small learnable prompt is trained to act as a specialized teacher. This prompt, when added to the model's input, conditions its behavior for a specific task, while the base model remains frozen. Thus, a single student can handle multiple tasks by simply selecting the appropriate prompt for each example. In practice, this means a company can have one LLM that knows about science, tool use, biology, and math, without needing to maintain four separate models or perform recurrent fine-tuning.
From a technical perspective, this approach falls within on-policy self-distillation (OPSD), but with a key innovation: the teacher is not a fine-tuned version of the model, but a soft prompt trained on (x, y_gold) pairs while keeping the backbone frozen. This preserves the exact representational geometry of the student, avoiding the post-hoc rationalizations that occur when the teacher injects privileged context into the input. Furthermore, the multi-task extension is natural: each example from a merged corpus is routed to its corresponding soft-prompt teacher, allowing the student to absorb knowledge from K teachers in parallel. During inference, all prompts are discarded, and the model returns to its base state.
What does this mean for companies developing AI-based applications? It means it is possible to build versatile and robust systems without incurring the costs of training and maintaining multiple models. At Q2BSTUDIO, as a software and technology development company, we see in this technique an opportunity to offer custom software solutions that integrate AI capabilities in a modular way. For example, a corporate virtual assistant could handle customer service queries, technical support, and business data analysis, all from a single base model, but with specialized prompts activated according to context.
Training efficiency is another strong point. While full fine-tuning can require millions of parameters and GPU hours, training a soft prompt only modifies a few thousand parameters, drastically reducing time and cost. This is especially relevant for companies that need to update their models frequently or add new skills without interrupting service. Moreover, by keeping the backbone frozen, the model's general capabilities are preserved—something sequential fine-tuning fails to achieve. In recent benchmarks, the multi-task variant of this technique achieved a higher average score (56.2 on Qwen3-1.7B-Base) while maintaining performance on general tasks, unlike sequential fine-tuning which degrades both.
From a business perspective, this technique fits perfectly with cloud-based AI integration strategies. At Q2BSTUDIO we offer cloud services on AWS and Azure, where we can deploy base models along with their soft prompts as lightweight microservices. Each task can have its own prompt, and the orchestrator routes requests to the correct prompt. This enables horizontal scalability and independent maintenance of each skill. Additionally, cybersecurity is a critical aspect: by not modifying the base model's weights, the attack surface is reduced and behavior auditing is easier. Our cybersecurity team can help ensure that prompts do not introduce vulnerabilities.
Another application area is business intelligence (BI). Language models can enhance Power BI or Tableau dashboards by providing natural language answers to data questions. With multi-task distillation, a single model can be expert in different business domains (sales, finance, logistics) simply by swapping the soft prompt. At Q2BSTUDIO we develop custom BI solutions that integrate these capabilities, allowing users to interact with their data conversationally and accurately.
We cannot forget the role of AI agents. An autonomous agent needs to execute multiple skills (planning, tool use, reasoning) in response to a goal. Soft prompt distillation allows the agent to have a single base model and activate different prompts depending on the task phase. This greatly simplifies architecture and training. At Q2BSTUDIO we are exploring how to apply this technique to create more robust and adaptable AI agents, combining it with process automation.
In summary, multi-task distillation with soft prompts represents a significant advance in how language models are trained and deployed. It allows a single student to learn from many teachers without sacrificing performance, with reduced costs and maintaining flexibility. For companies looking to incorporate AI efficiently and modularly, this technique opens new possibilities. At Q2BSTUDIO we are ready to guide our clients in implementing these strategies, whether through process automation or custom AI solutions. The future of artificial intelligence is not to have one model per task, but a single model that can be everything it needs to be, simply by adding a virtual teacher.
