In the field of emotion recognition, integrating multiple data sources—such as speech, facial expressions, and text—has proven to be an effective strategy for improving the accuracy of systems. However, how these modalities are combined remains a technical and conceptual challenge. The traditional early fusion approach, which concatenates all the characteristics before sorting, can achieve high performance but is monolithic and difficult to interpret. On the other hand, late fusion, which combines the predictions of independent unimodal models, offers modularity but misses the cross-interactions between modalities. This is where an innovative proposal emerges: SHAP-guided adaptive fusion (XGAF), which uses Shapley-based explanations to dynamically weight the contribution of each unimodal and cross-modal expert. This article discusses the findings of a recent study evaluating the impact of different SHAP attribution reduction strategies, and explores the practical implications for the development of more robust and explainable AI applications.
The SHAP (Shapley Additive Explanations) technique has established itself as one of the most powerful tools in explainable artificial intelligence (XAI), by breaking down predictions into individual contributions of each characteristic. In the multimodal context, its application allows assigning weights to each expert according to their real relevance for the prediction in each sample. The aforementioned study compares three ways to reduce SHAP attributions in a mix-of-experts scheme: the sum of absolute values, the absolute mean, and the absolute median. The choice is not trivial, especially when the experts have very different dimensionalities. For example, an expert based on visual features can have hundreds of dimensions, while a textual one has only a few dozen. The reduction by sum of absolutes preserves the total mass of attribution, giving more weight to high-dimensional experts if they actually contribute more, while the mean or median tend to standardize the weights, diluting the influence of complex cross-modal experts.
The experimental results on the MELD (emotion recognition in 7 classes) and CMU-MOSEI (sentiment in 3 classes) datasets are revealing. With sum-abs reduction, XGAF fusion achieves a performance comparable to early fusion (0.5983 vs 0.6018 in MELD for the Transformer variant), far outperforming late fusion by average odds (0.4598). McNemar's tests confirm that there is no significant difference with early fusion (p=1,000), while the improvement over late fusion is statistically solid (p
Beyond the numbers, this work provides a fundamental reflection for the engineering of multimodal systems: modularity and explainability are not at odds with performance if they are designed carefully. The ability to train independent experts and then dynamically combine them with interpretable weights opens the door to applications where transparency is critical, such as in AI-assisted clinical diagnostics or in customer care systems where decision auditing is required. In addition, the strategy can be extended to other domains, such as multimodal fraud detection or cybersecurity security, where combining network data, logs, and user behavior can benefit from an explainable merger.
At Q2BSTUDIO, as a software and technology development company, we see these advancements as fertile ground for creating AI solutions for businesses that are not only accurate, but also understandable and maintainable. Our expertise in custom applications allows us to design modular architectures that integrate these adaptive fusion principles, whether for sentiment analysis in social networks, emotionally capable virtual assistants or contextual recommendation systems. In addition, we combine these capabilities with AWS and Azure cloud services to ensure scalability, and with business intelligence tools such as Power BI to visualize and monitor the behavior of models in production.
The research also underscores an important limit: the design of cross-modal experts and the choice of reduction metric are decisions that must be made with knowledge of the facts. There is no one-size-fits-all solution; Each application will require an analysis of the dimensionality and relevance of each modality. In this sense, having a team that understands both the underlying theory and practical implementation is invaluable. That's why at Q2BSTUDIO we offer services ranging from artificial intelligence consulting to the development of custom AI agents, including cybersecurity audits to protect data pipelines. Our approach is holistic: we don't just implement algorithms, but we ensure that every component – from data capture to explanation of decisions – is aligned with business objectives.
In conclusion, the multimodal SHAP fusion represents a significant advance towards more modular, explainable and competitive emotion recognition systems. Empirical results show that it is possible to match or even exceed the performance of monolithic mergers, provided that attention is paid to details such as the reduction of attributions and the composition of experts. For companies looking to implement AI solutions for companies with high standards of quality and transparency, this approach offers a clear path. And along the way, collaboration with experts in artificial intelligence and custom software development can make the difference between an academic prototype and a productive, robust, and scalable solution.
.jpg)
.jpg)