Malayalam Speech to Text on a Free Tier: It Was Script Selection, Not Acoustics

Malayalam speech to text works on a free tier. We got near-verbatim transcripts from Groq's hosted whisper-large-v3 by adding one short Manglish example to the API's prompt parameter. No paid Indic model, no fine-tuning, no language router. Whisper had been hearing Malayalam correctly the whole time. It was writing it in the wrong script.

This is the build log from 31 July 2026, the day we moved the four agents in our internal AI operating system (JARVIS, Kiara, Taurus and Neo) off a weak local model and onto something that could understand our founder's voice notes. Most of them are in Manglish: Malayalam words, typed or spoken, mixed freely with English. We got the diagnosis wrong once before we got it right, and the wrong version nearly cost us a paid integration we did not need.

Why did Whisper seem unable to transcribe Malayalam?

Because its language detection was confidently and stably wrong. On three real Malayalam voice notes, Groq's whisper-large-v3-turbo with auto-detect returned Tamil script twice and French once. Repeated runs gave identical wrong output, so it looked like a hard capability wall rather than noise.

The starting point was worse. None of the agents had an STT section in its config, so all four fell through to Hermes Agent's free default: local faster-whisper base, 74 million parameters, the second-smallest Whisper. At that size Malayalam is effectively noise. We moved to Groq, whose free tier allowed 28,800 audio seconds a day, 20 requests a minute and 2,000 a day when we set it up, and whose largest Whisper is 1.5 billion parameters. English immediately became good. Malayalam did not.

The results on the three notes, before any fix:

• Note 1 (5.6 seconds): Tamil script, 3 of 3 runs
• Note 2 (7.2 seconds): Tamil script, 2 of 2 runs
• Note 3 (3.2 seconds): French, "Je ne suis pas encore plus de la main."

Forcing language=ml helped a little and broke other things. Turbo then produced Malayalam script that was phonetically close and orthographically wrong, and on the 3-second note it emitted Tibetan. whisper-large-v3 with the same hint produced Gurmukhi, the script used for Punjabi, on two notes.

What did we conclude, and why was it wrong?

We concluded that Whisper via Groq could not do Malayalam, and that its broken language detector ruled out routing Malayalam to a separate Indic model. The routing logic was sound. The premise under it was not: we had never compared the output against what was actually said.

The reasoning looked tight at the time. A router needs a detector, and the detector was the component failing. Retries or voting could not rescue it either, because the errors were stable rather than random. Manglish makes routing worse still: any per-utterance language decision is wrong for half of a code-mixed sentence. So the log recorded the verdict "Whisper cannot do Malayalam" and queued two options. One was a paid, code-mix-native Indic STT API. The other was AI4Bharat's IndicConformer, free and local but Malayalam-only, which would need a NeMo wrapper plus the very router we had just ruled out.

The one open item on that entry was ground truth: what did the three notes actually say? That turned out to be the only question that mattered.

How did ground truth overturn the verdict?

Our founder wrote down what each note said, in Latin script, the way he types Manglish every day. Read against those sentences, the garbage outputs were not garbage. Whisper had transcribed the right sounds and rendered them in whichever script it had guessed.

Take note 2. large-v3's Gurmukhi output was ਆਰਕੇਂਕੀਲੁਂ ਕੇਲਕਾਨ ਪੈਟੁਮੋ. Read aloud, that is "aarkenkilum kelkaan paitumo". What was said: "aarkenkilum kelkkaan pattumo" (can anyone else hear this?). Note 3's ਮਨ੍ਸੀ ਲਾਗਾ reads "manassilaga" against a spoken "manassilakathath". Tamil, French, Gurmukhi and Tibetan were one bug wearing four hats. The acoustic model was fine. Script selection was failing.

Three transliterated sentences did what another day of model testing would not have. Without them we were scoring outputs by eye in scripts none of us read.

How do you prime Whisper for Manglish with the prompt parameter?

Pass a short example of the output you want in the transcription request's prompt field, and leave the language unset. Whisper treats the prompt as the preceding transcript and continues in its style, including its script. Our primer is a single Manglish sentence in Latin letters.

This is documented behaviour, just rarely applied to script. OpenAI's Whisper prompting guide puts it plainly: "Fictitious prompts can be submitted to steer the model to use particular spellings or styles." The same guide warns that "when prompts are short, Whisper may be less reliable at following their style", and that the model follows the style of the prompt rather than any instruction inside it. Writing "transcribe in Latin script" does nothing on its own. An example sentence in Latin script does.

The production primer reads: "Manglish transcript in Latin script. Example: ippo njan ninnod samsarikkunnath ningalkk kelkkaan pattumo? entha ninakk manassilakathath?" With that on whisper-large-v3 and no language hint:

• Said: "ippo njan ippo ninnod samsarikkunnath, veare aarkenkilum kelkkaan pattumo?" Transcribed: "ippo njan ninnod samsarikkunnath vera aarkankilum kelkkaan pattumo?"
• Said: "entha ninakk chelappo manassilakathath pole?" Transcribed: "entha ninakkichalappo manassilakathath?"
• Said: "ippo njan samsarikkunnath ellaam baakki ullavarkk kelkkaan pattumo?" Transcribed: "ippo njan samsarikkunath allam bakku ulla aarkankilum kelkkaan pattu?"

The second is near-verbatim. The third is the weakest, and still a usable transcript for an agent that has to act on it. The cost is zero, on the key we already had.

Before shipping, we checked that English does not regress. Clean English audio came back byte-identical with and without the primer across repeated runs, and an accented English note became more stable with it. One configuration handles English, Malayalam and the code-mixed middle, so the routing layer we had ruled out was no longer needed at all.

Why not force language=ml or use the smaller turbo model?

Forcing language=ml produces Malayalam script, and nobody on our side writes Malayalam script in chat. Turbo leaked the primer into its transcripts. Both findings reversed conclusions we had logged earlier the same day, which is why each one was tested rather than assumed.

• Turbo versus large-v3. Hours earlier, with no prompt, turbo had been the stable one on a code-mixed sentence (3 of 3 correct against large-v3's 1 of 4). Add a prompt and it flips: large-v3 is clearly better, and turbo starts emitting "Example, njan..." as if the primer were speech.
• Canonical script versus the user's script. The correct target was the one the reader uses, not the language's official script. Matching how our founder writes beat matching how Malayalam is written.
• The translate endpoint. It returned fluent English ("Can anyone else hear me?") and dropped the detail. Fine for a gist, wrong for a transcript an agent will act on.
• Short audio. Nothing rescued the 3.2-second note before the primer, and very short clips remain the weakest case. Anything under about 4 seconds gets less context to decode from.

If you are building agents that take voice input from Kerala, this is the sort of detail our AI automation services team tests against real customer audio before a single workflow goes live.

What did it take to ship the fix in production?

A small script wired in as a Hermes command STT provider, not a patch to Hermes itself. Hermes's built-in Groq branch ignored both the model and the language set in config, so the working settings were unreachable any other way. A command provider survives image upgrades; an in-container source edit would be silently reverted by the next pull.

That config bug had already bitten us once. We set the Groq model to whisper-large-v3 and the log said so, but the Groq branch reads a module default instead of the config section, so production ran turbo for hours. Setting the model through the agent's env file did not help either, because that default is read from the raw process environment at import time and never sees it. We only found it because a later check read the code path instead of trusting the config.

Three details in the shipped script were load-bearing:

• It runs under the Hermes virtualenv's Python, where the OpenAI SDK lives. An earlier hand-rolled multipart request returned 403; the SDK encodes the upload correctly, and Groq's endpoint is OpenAI-compatible.
• It reads the Groq key from the agent's own env file itself, because Hermes parses that file internally rather than exporting it, so a subprocess inherits nothing. Each agent still calls Groq with its own key, which keeps per-agent credential isolation intact.
• It pins temperature=0, because unpinned runs flipped between output scripts.

One caveat, stated plainly: temperature 0 did not make Groq deterministic. Note 2 differed by a single letter between two runs. The variance is now cosmetic, where before it was a different writing system, but calling it deterministic would be an overclaim. We run the same setup across agents in our Hermes Agent production write-up, which covers the rest of the stack.

Can a speech-to-text pipeline test text-to-speech?

Yes, and it was the fastest diagnosis we ran that week. Synthesise the reply, transcribe it with the same Whisper setup, and compare strings. It turns "does this sound right?" into a check a machine can do with nobody listening.

Later the same day JARVIS's voice replies turned to gibberish. We had set him to a Malayalam Edge voice for everything, including English. Round-tripped through our STT, two real replies came back as "Beth ydy'r ddew? Ie, ie..." and "Kurve. Helasve jeleni witte...". No judgement call needed. The matrix that followed showed a hard boundary: the Malayalam voice reads only Malayalam script and turns Latin letters into phoneme soup, while the Indian-English voice given Malayalam script returns no audio at all. The fix was to route by script, and later to transliterate Manglish into Malayalam script with a small LLM call before synthesis. It was the same lesson as the STT bug, arriving from the other direction.

What is the reusable lesson?

Get ground truth before you blame the model. We had a confident verdict, a clean argument against routing, and two paid or heavy alternatives queued, all built on outputs nobody had checked against what was actually said. Three sentences typed by the speaker overturned all of it. For speech in particular: before concluding that a model cannot hear a language, read its wrong output aloud. If the sounds are right and only the letters are wrong, you have a script-selection problem, and a one-line prompt may be the entire fix.

For background on why Whisper generalises this way, the original Whisper paper describes training on 680,000 hours of weakly supervised multilingual audio. That breadth is why the acoustics held up for Malayalam even when script choice did not.

Frequently asked questions

Does the prompt trick work for Tamil, Hindi or other Indian languages?

We have only tested Malayalam and Manglish, so we will not claim results for the others. The mechanism is not Malayalam-specific: the prompt steers output style and script, not recognition. For Hinglish or Tanglish written in Latin letters, a short example in that register is the first thing we would try, and ground truth from real speakers is the only way to know whether it worked.

What does Malayalam transcription cost on Groq's free tier?

Nothing, within the limits. When we set it up the free tier allowed 28,800 audio seconds a day (8 hours), 20 requests a minute and 2,000 requests a day for Whisper. Four agents handling voice notes from a small leadership team stayed well inside that. Check Groq's current limits before planning for customer-facing call volume.

Should we use a dedicated Indic speech model instead?

Try the primed Whisper path first and measure it against transcripts from real users. We deferred both a paid code-mix-native API and a local IndicConformer build because the free path was good enough in daily use. If your audio is long-form, noisy telephony, or needs Malayalam script output, a dedicated Indic model may still win, and the only honest way to decide is to score both on the same files.

Will the primer show up in the transcript?

Sometimes, on near-silent audio or with the turbo model. That is why we keep the primer to one sentence and why our script strips a leading "Example:" or the primer's opening phrase if it appears. On whisper-large-v3 with normal voice notes we have not seen it leak.

If your team or your customers speak in Manglish and your automations cannot understand them yet, talk to us about your voice pipeline.