How Arrowhead Quill hears short, noisy BFSI calls

Share

We scored 7,552 caller utterances from custom Indian banking and insurance sales and support calls dataset, in five code-mixed languages, across five speech-to-text providers. An LLM judge graded each transcript on whether it kept the speaker's meaning, from 1 to 5.

0.5%
of Arrowhead Quill's sub-second clips came back empty. Other providers: 0.7% to 15.3%.
4 of 5
languages where Arrowhead Quill scores highest on clips under 1 second.
43%
of all caller turns in the dataset are shorter than 1.5 seconds.

Here is the short version. Arrowhead Quill is the top-scoring model overall in Hindi (4.06) and Tamil (4.15), by a hair over Sarvam Saaras v4 in both. Sarvam leads overall in Telugu, Kannada and Malayalam. Where Quill pulls clearly ahead is the part of the call a voice agent hears most: very short turns. On clips under one second Quill leads in Hindi, Tamil, Telugu and Kannada, and it returns the fewest empty transcripts at every clip length.

The results

Mean semantic score per language. Higher is better, and 5 is a perfect meaning match.

Table of mean semantic scores by language for Arrowhead Quill, Sarvam, Gnani, Deepgram and ElevenLabs Caption: Scale 1–5, LLM judge (Gemini 3.5 Flash Lite, temperature 0). Deepgram nova-3 does not support Malayalam.

Two things stand out. First, the top tier is tight: Quill and Sarvam sit within 0.01 of each other in Hindi and Tamil and within 0.09 in Telugu. Second, the gap to the rest is wide in the Dravidian languages. Deepgram nova-3 drops below 3.0 in Tamil, Telugu and Kannada, and ElevenLabs scribe v2 realtime ranges from 2.86 to 3.49.

Averages hide the shape of the call, though. So we split every language by clip length.

Short turns are most of the call

On an outbound BFSI call the bot does most of the talking. The customer mostly answers: haan, okay, illa madam, nahi chahiye, kal call karo, a date, a pincode, an amount. In this dataset the median caller turn is 1.4 to 2.0 seconds long, and 19% to 43% of clips are one or two words.

Those short answers carry the decisions. A missed "no" can turn into a mis-sold policy. An empty transcript leaves the bot waiting in silence or asking the question again. We trained Arrowhead Quill with short utterances deliberately in the mix, and the sub-second numbers are where that shows.

The pattern repeats across languages. Most providers lose ground as clips get shorter. Outside Malayalam, Quill stays flat or even improves. In Tamil, Quill scores 4.25 under one second against Sarvam's 3.92 and Gnani's 3.68. In Kannada, Gnani falls from 3.80 at 1–1.5 s to 2.95 under 1 s, while Quill holds at 3.83.

Malayalam is the exception. Sarvam is ahead there at every length, including 4.09 against Quill's 3.70 under one second. Malayalam also has the most one- and two-word clips in the set (43%).

The empty transcript problem

The worst thing an ASR can do in a live call is return nothing. The agent has no text to act on, so it stalls or repeats itself while the customer is still on the line. Empty output scores an automatic 1 in this benchmark, and it is concentrated in short clips.

Under one second, Gnani returns nothing for 15.3% of clips and Deepgram for 14.1%. Sarvam is at 4.1%. Quill is at 0.5%, the lowest of the five at every length, with ElevenLabs close behind. ElevenLabs avoids empty output but often returns a confident wrong phrase instead, which is why its scores stay low.

How often each provider fails outright

A mean score can hide a long tail of misses. This breakdown shows how each provider's transcripts spread across the five score levels. The red end is where a voice agent goes wrong.

In Telugu, only 1.8% of Quill's transcripts score 1, against 3.8% for Sarvam and 9.3% for Gnani. Sarvam has more perfect scores in every language, while Quill has fewer complete misses in four of five. For a bot that has to act on every turn, the misses are the costlier side.

Arrowhead Quill against Sarvam, clip by clip

Sarvam Saaras v4 is the strongest peer in this benchmark, so we compared the two directly. For each language we counted the clips where one model got the meaning right (score 4 or 5) and the other got it wrong (score 1 or 2).

Quill wins more of these disagreements in Hindi, Tamil and Telugu. Sarvam wins more in Kannada and Malayalam, by a wide margin in Malayalam. We are publishing that because it is what the data says.

What this looks like on a real call

A few Tamil clips from the benchmark. Each is under a second and a half, recorded on a live call, and each one decides what the bot should do next.

வேணாங்க வாங்கணும்reference · 0.9 s
"No sir, I need to buy it." A sales intent.
Arrowhead Quill5வேணாங்க வாங்கணும் Sarvam2வேணாங்க வேணாங்க. "No, no." Gnani2வேணாங்க வேணா "No, don't want." Deepgram3வேணாங்க வாங்குறோம் "we are buying" ElevenLabs2வேணாம், வேணாம். "Don't want, don't want."
Four of five providers turned a buying signal into a refusal. WER on Sarvam's output is only 50%.
அது வரலங்கreference · 0.9 s
"It didn't come." A support caller reporting a missing credit or OTP.
Arrowhead Quill5அது வரலைங்கள Sarvam2அது வந்து. "It came." Gnani2அதெல்லாம் "All those." Deepgram1(empty output) ElevenLabs2அது வேண்டாம். "Don't want it."
Sarvam's output says the opposite of what the caller said. Only Arrowhead Quill keeps the complaint.
ஒன்னும் இல்ல mamreference · 1.0 s
"Nothing, ma'am." The caller is declining.
Arrowhead Quill5ஒன்னும் இல்ல மேம் Sarvam2இன்னும் இல்லை மேம். "Not yet, ma'am." Gnani2உண்மை இல்லை மேம் "Not true, ma'am." Deepgram2உண்மை இல்லாமல்! "Without truth!" ElevenLabs4இல்ல, இல்ல ma'am. "No, no, ma'am."
"Not yet" invites the bot to follow up on a lead that has already said no.
இல்ல madamreference · 0.5 s
"No, madam."
Arrowhead Quill5இல்ல மேடம் Sarvam1(empty output) Gnani1(empty output) Deepgram1(empty output) ElevenLabs2சொல்லுங்க, madam. "Tell me, madam."
A half-second "no". Three providers return nothing; one hears an invitation to continue.

Why we score meaning instead of WER

Word error rate counts how many words in a transcript differ from the reference. For Indian code-mixed speech that number misleads in both directions.

It punishes differences that do not matter. The same English word can be written in Latin or native script, numbers can be digits or words, and spoken Tamil spells words differently from written Tamil. All of these count as errors under WER.

It also forgives differences that do matter. Change one word in a two-word reply and WER reports 50%, even if that word turned "I want to buy it" into "I don't want it".

A voice agent does not read the transcript the way WER does. It passes the text to a language model, which decides the reply. That model copes easily with a transliterated word or a number written as digits. It cannot recover a reversed answer or a wrong amount. So we grade transcripts the way the agent's LLM will use them: with an LLM that asks whether the meaning survived.

The rubric

Each language has its own rubric. All five share the same scale:

ScoreMeaning
5Meaning fully preserved. Differences in script, numeral format, spacing, punctuation, fillers or code-mixing style don't lower the score.
4Minor deviation. One or two small word errors that don't change the message.
3Moderate. Intent recognisable, but notable errors in details, entities, numbers or key phrases.
2Major deviation. Key content wrong or missing; intent partly obscured.
1Completely wrong. No semantic resemblance, gibberish, empty output, or output in the wrong language.

The judge is lenient on form and strict on facts: names, numbers, dates and amounts must be right. Output in the wrong language scores low, and the Telugu and Kannada rubrics say explicitly that the other's script is wrong even though the two look alike. Empty transcripts are scored 1 automatically, without calling the judge. References are written code-mixed, with native words in native script and English words in Latin script.

Judge setup: google gemini-3.5-flash-lite , temperature 0, 15 utterances per request, structured JSON output with a score and a one-sentence reason for each utterance.

We're not alone in this

Sarvam makes the same argument in its Evaluating Indian Language ASR post (April 2026), which pairs WER with LLM-judged metrics such as LLM-WER and an intent score, because "the transcript is right, the score is wrong" in so many Indian-language cases. Researchers at IIT Bombay reach a similar conclusion in LASER, an LLM-based ASR scoring rubric whose Hindi scores reached about 94% correlation with human annotators. We agree with that direction, and for a voice agent we think meaning is the score that matters most.

The dataset

Every clip is a caller turn from a real BFSI sales or support call. The audio is telephony-grade and noisy, and we kept it that way on purpose, because that is what production sounds like.

Providers and caveats

  • Models tested: Arrowhead Quill, Sarvam Saaras v4, Gnani Prisma v2.5 (realtime), Deepgram nova-3 and ElevenLabs scribe v2 realtime.
  • Deepgram nova-3 does not support Malayalam, so it has no Malayalam score and is left out of the pooled Malayalam figures.
  • One judge model. Every transcript is scored by Gemini 3.5 Flash Lite at temperature 0. A different judge could shift absolute scores.
  • Small gaps are small. Quill's overall lead in Hindi and Tamil is 0.01. Treat those two as a tie at the top, and look at the clip-length and failure numbers for the real differences.
  • Clip length buckets are under 1 s, 1–1.5 s, 1.5–2.5 s, 2.5–4 s and over 4 s, on clip duration.
What it means for a BFSI voice agent. If your calls are full of short answers, pick the ASR by how it handles those answers, not by its average on long sentences. On that measure Arrowhead Quill is the most reliable model we tested in Hindi, Tamil, Telugu and Kannada.

Sources

  1. Sarvam AI, "Evaluating Indian Language ASR", April 2026. sarvam.ai/blogs/evaluating-indian-language-asr
  2. Parulekar & Jyothi, "LASER: An LLM-based ASR Scoring and Evaluation Rubric", IIT Bombay. arxiv.org/abs/2510.07437