Building a real-time voice chatbot for African languages like Igbo, Yoruba, and Hausa goes far beyond swapping language codes in a cloud API. These languages, spoken by millions, are severely underrepresented in commercial models: Whisper hallucinates on silence, NLLB mistranslates common Yoruba phrases, and off-the-shelf TTS has never heard Igbo spoken. In this article, we explore how to design and deploy a production-grade architecture that serves all four languages (English, Hausa, Yoruba, Igbo) with fine-tuned components behind an OpenAI-compatible API.
The architecture consists of five independent microservices, each owning a specific concern, exposing OpenAI-compatible HTTP endpoints. The chatbot calls them using the same OpenAI Python client it would use for GPT-4, just with a different base_url. This allows swapping or scaling any component without application code changes. From custom software development to complex AI systems, modularity is key for production flexibility.
The first component is speech-to-text (STT). We start with Whisper, a multilingual model that, without fine-tuning, achieved a word error rate (WER) of 30–40% on Hausa. After fine-tuning on approximately 50 hours of native speaker audio per language, WER drops below 12%. The model is loaded as a HuggingFace pipeline with 30-second chunking, handling audio of any length. However, the most critical problem is silence. In production, 15–25% of incoming audio is background noise, accidental activations, or dead air. Without filtering, Whisper hallucinates plausible text on all of it. The fix is running Silero VAD (voice activity detection) before the model sees anything: if no speech is detected, the request returns an empty transcription immediately, eliminating inference and hallucinations. This single change eliminated the most common class of user-visible errors.
For language identity, we force the correct language token in Whisper's decoder for English, Hausa, and Yoruba using forced_decoder_ids. Without this, Whisper may silently switch to a language it recognizes better mid-output. Igbo is excluded because it is not in Whisper's vocabulary; forcing it would raise an error, so the fine-tuned model handles Igbo freely. Transcription is streamed as Server-Sent Events (SSE), emitting each 30-second chunk as it completes. The client starts receiving text while the model is still processing the tail of a long audio clip.
The second component is translation. We use Meta's NLLB-200 rather than an LLM. LLMs translate well for high-resource languages but are unreliable and expensive for Hausa ↔ English or Igbo ↔ English at production throughput. NLLB was purpose-built for these pairs. The challenge is latency: the unquantized PyTorch model runs at ~6 tokens/second on CPU—unacceptable. We use CTranslate2, a C++ inference engine optimized for encoder-decoder models. Converting NLLB to INT8 format is a one-time step: the converted model runs at ~150ms per sentence on two CPU cores, viable without GPU. We deploy separate instances per language (Hausa, Igbo, Yoruba) as independent Docker containers. One non-obvious requirement is that translation direction must be explicit: the API enforces a required field like 'hausa_to_english' with no default. The caller always knows which direction it wants.
The LLM agent forms the core conversational logic. We serve a fine-tuned model via vLLM, which offers two critical features: PagedAttention manages KV cache as virtual memory pages, preventing OOM crashes under concurrent load; and continuous batching allows requests arriving mid-generation to join the current batch, roughly doubling effective throughput compared to static batching. In front of vLLM, a Redis semantic cache compares incoming query embeddings against cached ones at a 0.95 cosine similarity threshold. In production, a large fraction of queries are semantically identical rephrasings of the same underlying question, cutting LLM calls by 30–40%. The agent exposes two endpoints: /api/agent/chat for complete JSON responses and /api/agent/stream for token-by-token streaming via SSE. After streaming completes, conversation logging and RAGAS quality evaluation are queued as background tasks without blocking the response. A practical note: the X-Accel-Buffering: no header is required when running behind Nginx to prevent buffering the SSE stream.
The final component is text-to-speech (TTS). No commercial service handles tonal Yoruba or Hausa with acceptable naturalness. We built a custom pipeline on ChatterboxTTS—a voice cloning architecture with a T3 acoustic transformer and S3Gen neural vocoder—fine-tuned on native speaker recordings for all four languages. Fourteen named voices are available, each cloned from a real speaker. A key technical insight: the T3 transformer runs in bfloat16 on CUDA, halving memory and increasing throughput with negligible quality loss, because transformer attention is numerically stable. However, the S3Gen vocoder must remain in float32; vocoders use deep convolutional stacks with accumulated residuals, and at bfloat16 numerical error produces audible artifacts like buzzing or complete corruption. Each submodel must be tested independently before assuming bfloat16 is safe across a composite architecture.
Voice identity in Chatterbox is determined by a speaker embedding extracted from a reference audio clip. This extraction is expensive (200–400ms per request). To avoid it, all named voice conditionals are computed at startup and cached in memory; switching speakers is a simple dictionary lookup. For low perceived latency, long responses are split at sentence boundaries and synthesized chunk-by-chunk. Audio starts streaming to the client as soon as the first chunk is ready. The HTTP response begins with a WAV header advertising open-ended RIFF and data sizes (0xFFFFFFFF), a standard trick for streaming PCM audio without knowing total length. The difference is significant: a 500-word response synthesized as a single block takes ~8 seconds before audio plays; chunked, the user hears the first sentence in ~1.2 seconds.
From a business perspective, this architecture shows it is possible to serve underrepresented languages with production quality without relying on tech giants. At Q2BSTUDIO, a software and technology development company, we apply these principles across multiple domains: from custom AI solutions to cybersecurity systems that protect voice data, and cloud infrastructures on AWS or Azure that auto-scale on demand. Cost optimization is essential: NLLB services on CPU with INT8 quantization drastically reduce GPU spending, which is reserved for STT and TTS where latency is critical. Additionally, microservice modularity allows integrating conversational AI agents with BI systems like Power BI for usage pattern analysis and experience improvement.
One of the most valuable lessons is that none of these five services is exotic on its own—fine-tuned Whisper, CTranslate2, vLLM, and a voice-cloning TTS model are all individually well-documented. What makes this stack production-grade is the accumulation of small, hard-won decisions: gating STT with VAD, forcing translation direction explicitly, splitting precision at the submodel boundary rather than the pipeline boundary, and separating evaluation from storage. Each of these was a real incident before it became a design rule. For teams building voice interfaces for underrepresented languages, the lesson generalizes beyond this specific tech stack: budget time for the boring infrastructure problems, not just the model quality problems—they are usually the ones that actually break in production.
In summary, the presented architecture offers a practical roadmap for bringing voice processing to languages that the market has ignored. With a smart combination of fine-tuned models, cloud infrastructure, and good engineering practices, it is possible to build robust systems that work in the real world. From voice recognition with VAD to chunked synthesis, each component delivers tangible value, and together they demonstrate that inclusive technology is not only possible but also scalable and cost-effective.
