NeurIPS 2026 · Africa in AI Workshop

Beyond Exact Match: Unmasking WER Inflation in Akan ASR through Variant‑Aware Evaluation

1Google Research
Google
2Ashesi University
Ashesi University
Paper arXiv soon
15–17%
relative WER inflation from valid phonological & code-switching variation alone
22–25%
inflation once human spelling inconsistencies are included
7,434
canonical lexicon entries mined from WAXAL
Overview

Abstract

Standard Automatic Speech Recognition (ASR) evaluation relies on Word Error Rate (WER), which treats every token-level mismatch as an error regardless of linguistic validity. For dialectally diverse languages like Akan, phonological variations, spelling discrepancies, and code-switching markers are penalized identically to genuine misrecognitions, inflating reported error rates. We introduce a variant-aware WER (va-WER) that normalizes predictions and references through a canonical lexicon compiled exclusively from training and validation data before scoring. Our framework classifies near-miss word pairs into a five-tier linguistic taxonomy using an LLM-as-a-Judge approach validated against native-speaker human annotators, then compiles frequency-based canonical lexicons.

Across three Akan ASR models (Whisper-large-v3, Gemma-3n-2B, MMS-1b-all), we demonstrate that: (1) standard WER is inflated by 15.0%–17.0% (relative) due strictly to legitimate phonological and code-switching variations, rising to 22.4%–25.4% when human spelling inconsistencies are included; (2) the LLM judge achieves 81.4% agreement with native speakers when orthographic categories are collapsed, and sensitivity simulations (N = 50 Monte Carlo trials) confirm that metric reduction remains statistically robust (p < 10−4) even under substantial boundary noise; and (3) lexicons from spontaneous speech corpora transfer to independent Akan datasets, confirming that captured variants represent systematic linguistic phenomena rather than corpus-specific noise. We advocate reporting va-WER alongside standard WER in orthographically fluid languages to distinguish genuine model failures from benign dialectal variations.

Background

A five-tier taxonomy of Akan variation

Akan spans Asante Twi, Akuapem Twi and Fante, and three literary traditions (Basel Mission 1842, Christaller 1875, Akan Orthography Committee 1978) still coexist. Every near-miss pair is assigned one of five categories.

va-WER

Intra-Dialect Phonological

Spoken pronunciation shifts within a dialect, e.g. intervocalic rhotacism (d → r).

akwadaa ↔ akwaraa
va-WER

Inter-Dialect Phonological

Systematic differences between dialect groups where both forms are valid.

adwuma ↔ edwuma
va-WER

Code-Switching

English loanwords phonetically adapted into Akan orthography.

car ↔ kar
Ablation only

Spelling Error

Transcriber violations of orthographic conventions with no phonological basis.

bebree ↔ beberee
Discarded

Different Words

Genuine semantic or grammatical errors that change meaning.

tokua ↔ ntokua
Approach

Method

From raw transcripts to a variant-aware score in five stages.

End-to-end va-WER pipeline
Raw audio corpus Unique tokens & frequencies train + val only Candidate pairs (edit distance = 1) akwadaa ↔ akwaraa school ↔ sukuu bebree ↔ beberee tokua ↔ ntokua Gemini 3.5 Flash Chain-of-thought prompt + 5-tier taxonomy Phon. variant(inter-dialect) Phon. variant(intra-dialect) Code-switching Spelling error Different words Valid variants legitimate variant pairs Ignored pairs V₁ V₂ Vₙ CANONICAL FORM std va WER Pairs generation LLM judge Filtering Lexicon Evaluation
01

Near-miss candidates

Extract unique tokens from train + val only and pair every word within Levenshtein distance d = 1. The test split is never seen.

02

LLM judge

Gemini 3.5 Flash classifies each pair with a 4-step chain-of-thought prompt, grounded in two real corpus sentences per pair.

03

Canonical lexicon

The more frequent form becomes canonical. A directed graph resolves diverging mappings, cycles and transitive chains into O(1) lookups.

04

Normalize & score

Map both reference and hypothesis through the lexicon, then compute ordinary WER: WER(norm(ref), norm(pred)).

Why edit distance 1? The low-CER / high-WER paradox

Character error rates cluster at 10–30% while WER sits at 40–50%: models produce grapheme sequences that are nearly identical to references but get penalized at the word level. Over 10,000 substitutions occur at exactly one character of difference, and frequency drops off sharply beyond that.

WER vs CER distributions
(a) WER and CER score distributions.
Levenshtein edit distances
(b) Edit distance of substituted word pairs.
Main results

Results

Held-out WAXAL test split, three Akan fine-tuned foundation models, lexicon compiled from WAXAL train + val.

WER by model on the held-out WAXAL test set (%, lower is better)
Standard WER Strict va-WER Fully normalized (+ spelling)
Table 2Standard WER vs. strict va-WER and fully-normalized WER (WAXAL test, train + val lexicon)
StandardStrict va-WERFully normalized (+ spelling)Absolute points recovered, by category
ModelWERWERΔ RelWERΔ RelPhonologicalCode-switchSpelling
Gemma-3n-2B40.0034.0214.95%31.0222.45%5.870.112.54
Whisper-large-v338.9732.3516.99%29.0725.40%6.470.152.73
MMS-1b-all39.5433.1416.19%30.0124.10%6.260.142.60
Phonological variation is the dominant driver, accounting for 5.87–6.47 absolute WER points across models; code-switching adds a steady 0.11–0.15. For Whisper, the total 9.90-point recovery splits into 65.4% phonological, 27.6% spelling, 1.5% code-switching, and 5.6% from alignment synergies when normalizations are applied jointly.
Where the near-misses come from

Most raw near-miss pairs are genuinely different words and are discarded. Valid variants concentrate in spontaneous speech: over 30% of WAXAL candidate pairs are legitimate variants (7,434 lexicon entries), while scripted read speech in FID yields only 25.

Distribution of candidate word pairs by taxonomy category across corpora
Share of candidate word pairs in each taxonomy category, per corpus.
Generalization

Lexicons transfer across corpora

Relative WER reduction on WAXAL test predictions when the lexicon is compiled from each source corpus.

Cross-corpus lexicon transfer (relative WER reduction on WAXAL test)
WAXAL (native, spontaneous) GhanaNLP (community) FID (scripted)
0%5%10%15%20%
Gemma-3n-2B
Whisper-large-v3
MMS-1b-all
Relative WER reduction (%)

The GhanaNLP lexicon, built from community spontaneous speech, still removes 3.1–3.5% of relative WER on WAXAL, while scripted FID removes under 0.8%. Variant coverage scales with how spontaneous and diverse the source register is, which suggests the lexicon captures recurring linguistic phenomena rather than corpus artifacts. (Lexicon size differs substantially across corpora and partly confounds the comparison.)

Broader impact

Why it matters

Voice interfaces are a gateway to civic and financial services across Africa, where oral communication often outpaces written literacy. A Fante-speaking farmer in the Central Region says edwuma (work); an Ashanti clinician expects adwuma. Reporting an unadjusted 40% WER instead of an operational 32% can get a viable model abandoned. va-WER aligns benchmarks with how Akan is actually spoken, and the compiled lexicons can be reused to clean noisy corpora, aid cross-dialect translation, and inform language planning. The recipe (near-miss mining, LLM-guided taxonomy, graph resolution) is a low-cost blueprint for other multidialectal languages such as Yoruba, Igbo, Wolof and Luganda.

Responsible use. The lexicons are intended for acoustic benchmarking and corpus normalization, not for cultural or orthographic standardization. Collapsing variants erases regional distinctions, so va-WER is unsuitable for accent or dialect identification.

Citation

BibTeX

@inproceedings{mireku2026beyond,
  title     = {Beyond Exact Match: Unmasking {WER} Inflation in {Akan} {ASR}
               through Variant-Aware Evaluation},
  author    = {Mireku, Prince and MohamedKhair, MohamedElfatih and
               Brempong, Emmanuel Asiedu and Helmy, Amr and Nelson, Perry and
               Siyavora, Tavonga and Diack, Abdoulaye and Walcott-Bryant, Aisha},
  booktitle = {NeurIPS 2026 Workshop on Africa in AI},
  year      = {2026}
}