Synthesizing natural speech for Indian-language voice agents is not primarily a fidelity problem. Modern neural systems already produce clean, human-sounding audio on well-formed sentences. The failures that matter in production are narrower and harsher: a ten-digit account number read as a compound quantity, an abbreviation like NEFT pronounced as a word, a brand name mangled, or a model that loses the speaker's identity the moment an English clause appears inside a Hindi sentence.
These behaviours are invisible to a benchmark built from clean, natural sentences, because every competent system scores near zero error on such text. Ranking therefore requires deliberately adversarial input alongside the natural case. The evaluation below is constructed on that principle.
Before the tables, the audio. Four clips in the Sherin voice — banking and property-services copy in English, Hindi, and Hindi–English code-mix.
Select a sample to listen to generated voice output:
“Your home loan application has been approved, and the sanction letter will reach your registered email address by this evening.”
“आपकी जमा राशि अगले सोमवार को परिपक्व हो रही है, ब्याज सहित पूरी रकम आपके बचत खाते में अपने आप आ जाएगी।”
“आपका home loan pre-approved है, property documents verify होने के बाद disbursement उसी हफ़्ते शुरू हो जाएगा।”
“Sure, I can arrange a site visit for that two bedroom apartment on Saturday morning, and the builder will share the floor plan before you arrive.”
Systems compared
Five systems, forty utterances per language, eighty in total. The text set is held identical across every system, so differences are attributable to the synthesizer rather than to what it was asked to say. Half of it is deliberately adversarial — numerals, abbreviations, entities, code-mix — which is where production voice agents actually fail.
The benchmark data is taken from Sarvam AI's tts-general-benchmark, with sampled data from AI4Bharat's IndicVoices-R.
| Model | Engine | Voice | Languages |
|---|---|---|---|
| Ragini Pro | — | Sherin | 2 |
| ElevenLabs | eleven_v3 | Matched clone | 2 |
| Cartesia | sonic-3 | Native per language | 2 |
| Google Chirp | Chirp 3: HD | Single name, all locales | 2 |
| Sarvam | bulbul:v3 | Native, male | 2 |
Intelligibility and speech round-trip accuracy
Each system's output is transcribed by a single fixed ASR model with the language forced per utterance, and the transcript compared against the reference text. WER and CER are corpus-level within each language — total edit distance divided by total reference length, rather than a mean of per-utterance rates, so one short bad utterance cannot dominate. PER measures phone-level distance after phonemisation, which partially credits near-miss pronunciations instead of scoring a whole word wrong. The headline figures below are the unweighted mean of the English and Hindi rates; the per-language breakdowns follow.
| Model | WER ↓ | CER ↓ | PER ↓ |
|---|
How to read this table
The two halves of every table below were not produced by the same procedure. The four commercial systems' figures are carried over from the Ragini report: they are scored on the human-recorded half of the text set only, and carry a 0.8 factor applied in that report. Ragini Pro's figures are raw corpus-level scores over the full text set, adversarial half included and no factor applied. Scored on matched terms — same subset, same factor — Ragini Pro reads 0.066 WER on English and 0.142 on Hindi, alongside Sarvam and ElevenLabs. Read the columns as two different measurements, not as a ranking.
Per-language results
Word error rate for both target languages. Hindi is roughly twice as hard as English for every system, and the ordering is not the same in the two columns — ElevenLabs leads on English, Sarvam on Hindi — so a single averaged score would hide that.
| Model | English | Hindi |
|---|
On English the four commercial systems land between 0.042 and 0.090, with ElevenLabs ahead; on Hindi they compress into 0.112–0.144, close enough to be a four-way tie. Ragini Pro reads 0.180 and 0.296. Those two halves of the table were not produced the same way — see the note above the previous table — so the distance between them is a difference in method as much as in quality.
The same breakdown at character and phone level. These are the informative views for Devanagari: a single wrong vowel sign fails a whole word under WER while barely moving CER, so a WER gap that does not reproduce here is an inflection error rather than a system mishearing the sentence.
| Model | English | Hindi |
|---|
| Model | English | Hindi |
|---|
The character and phone views repeat the word-level pattern rather than adding to it, which is what should happen when the difference between two columns is largely one of scoring recipe: the ratios between the commercial systems and Ragini Pro stay roughly constant across all three metrics instead of narrowing at character level, as a genuine pronunciation gap would.
What these numbers cannot tell you
The ASR is a measuring instrument, not ground truth
Every error rate here is mediated by a single ASR model. Where it mishears correct audio, that scores against the synthesizer; where it silently repairs a mispronunciation into the intended word, the error vanishes. Transcripts are released alongside the scores so individual failures can be inspected rather than assumed.
Naive text normalisation destroys Indic scripts
The standard Whisper BasicTextNormalizer replaces every Unicode Mark character with a space. In Devanagari the vowel signs are marks, so it silently rewrites कॉल as क ल and inflates error rates for every system simultaneously. This evaluation strips only punctuation and symbols and preserves marks.
Speaker identity is not held constant
Ragini Pro speaks in the Sherin voice. The four commercial systems were run earlier against a different reference speaker — ElevenLabs as a matched clone of that one, Cartesia and Google with their own native voices, Sarvam with a male voice. This does not affect WER, CER or PER, which measure intelligibility — but no speaker-similarity comparison can be drawn from this run, and none is offered.
Sample size
40 utterances per language per system is adequate for error rates, which aggregate over hundreds of words, but the gaps among the four commercial systems are small enough that 40 utterances cannot separate them confidently. The Hindi column in particular is a four-way tie within the noise.
Reproducing
The 80-row English/Hindi benchmark text set, per-utterance scores and ASR transcripts are released as CSV. Because every system speaks identical text, adding a new system requires only generating those 80 utterances and rerunning the scoring step.