RoME: Robustness with Low-Rank Experts against Adversarial Perturbations

In the current landscape of artificial intelligence, model security against adversarial attacks has become a critical challenge. These attacks consist of small, imperceptible perturbations for humans that can deceive machine learning systems, causing catastrophic errors in applications such as autonomous vehicles, medical diagnosis, or fraud detection systems. Traditionally, adversarial training techniques focused on a single type of perturbation (e.g., L8 or L2), but in real-world environments, attackers can employ multiple strategies. This gives rise to multi-perturbation adversarial training (MAT), which seeks to make the model robust against several types of threats simultaneously. However, this approach often generates a trade-off dilemma: improving defense against one perturbation can weaken protection against another. Recent research proposes solutions based on mixture of experts (MoE), where different model pathways specialize in distinct threats. A notable example is RoME (Robust Mixture of Low-Rank Experts), which introduces low-rank experts as additive updates to a shared backbone, allowing it to capture features common to all perturbations while each expert focuses on threat-specific information. Additionally, it incorporates a dual-scale mechanism in the selection gate that exploits discriminative signals at local and global levels, and a forced diversification of expert usage to avoid generic pathways. Results show a significant improvement in joint robustness and natural accuracy, even against threats unseen during training. This advancement is relevant for companies integrating artificial intelligence for businesses and seeking to protect their models with cutting-edge techniques. At Q2BSTUDIO, we understand that cybersecurity is not limited to firewalls or network protocols, but encompasses the integrity of algorithms that make critical decisions. Therefore, we offer services of robust and adaptable AI agents, as well as customized AI for businesses that incorporates adversarial training techniques. Our portfolio includes custom applications and custom software with integrated security modules, and we leverage aws and azure cloud services to scale high-performance solutions. Likewise, we complement these capabilities with business intelligence services using power bi to monitor model health in production. The combination of adversarial robustness and intelligent analysis allows organizations to deploy reliable and transparent systems, minimizing operational risks. At Q2BSTUDIO, each project is approached as an ecosystem where security, artificial intelligence, and the cloud converge to deliver real and sustainable value.