We present a new benchmark and an innovative model capable of simultaneously generating sung voices and full body movement directly from textual prompts such as rap lyrics. By aligning both modalities during training, the system manages to outperform current benchmarks in vocal quality, movement realism, and synchronization, evaluated with metrics such as BC, FID, and LVD.
The proposed architecture avoids the traditional cascade of independent models and demonstrates better results than chained approaches such as DiffSinger plus Talkshow, while reducing computational overhead and inference times. Joint training allows intonation, rhythm, and choreography to influence each other, generating coherent and naturally synchronized performances.
Ablation studies show that modality-specific VQ-VAEs are key components for preserving acoustic fidelity and movement expressiveness; conversely, generic large language models present limitations in coordinating fine multimodal signals without specialized adaptations. These findings mark a relevant advance in text-driven performance synthesis and open new avenues for music creation, entertainment, and automated content production.
At Q2BSTUDIO we combine experience in software development and custom applications with applied research in artificial intelligence for businesses. We offer custom software services and solution design that integrate AI agents, AI for businesses, and data pipelines optimized for audiovisual and creative projects requiring multimodal generation.
Our services include cybersecurity, AWS and Azure cloud services, as well as business intelligence services and Power BI consulting to turn data into actionable decisions. We are specialists in implementing solutions that range from data capture and model training to cloud orchestration and comprehensive infrastructure security.
If your goal is to explore custom applications that incorporate synchronized voice and movement generation, or to deploy AI agents that automate creative workflows, at Q2BSTUDIO we design and implement the custom solution you need. We combine technical research with best practices in cybersecurity and cloud management for scalable and secure projects.
Keywords for positioning: custom applications custom software artificial intelligence cybersecurity AWS and Azure cloud services business intelligence services AI for businesses AI agents Power BI.
Contact Q2BSTUDIO to explore how the convergence of voice, movement, and text can transform your digital products and user experiences, leveraging the latest innovations in multimodal artificial intelligence.
