Study 02 · A25 · October 2, 2026
Four models. The same development recordings.
Large v3 PLD normalized had the lowest word and character error rates in every tested condition. Its median processing time was about four times Whisper Small’s on GB10.
Quality and processing cost
All 2,400 transcription attempts succeeded: four existing models × 200 recordings × three conditions. This is an inference comparison, not four new training runs.
| Model | Condition | WER % | CER % | Median ms | CUDA GiB |
|---|---|---|---|---|---|
| Whisper Small | Clean | 32.39 | 9.15 | 381 | 1.03 |
| Whisper Small | 10 dB noise | 42.20 | 15.14 | 390 | 1.07 |
| Whisper Small | 200 ms dropout | 33.83 | 9.93 | 380 | 1.03 |
| Small FSC | Clean | 36.67 | 10.66 | 385 | 1.03 |
| Small FSC | 10 dB noise | 44.37 | 15.79 | 390 | 1.07 |
| Small FSC | 200 ms dropout | 38.39 | 11.58 | 384 | 1.03 |
| Small PLD Filipino | Clean | 30.63 | 13.67 | 392 | 1.07 |
| Small PLD Filipino | 10 dB noise | 43.30 | 20.78 | 389 | 1.07 |
| Small PLD Filipino | 200 ms dropout | 32.22 | 18.05 | 394 | 1.07 |
| Large v3 PLD normalized | Clean | 23.96 | 7.05 | 1550 | 6.40 |
| Large v3 PLD normalized | 10 dB noise | 27.93 | 8.62 | 1569 | 6.40 |
| Large v3 PLD normalized | 200 ms dropout | 25.09 | 7.67 | 1568 | 6.40 |
The public Small FSC model is an existing SapinSapin checkpoint. It is not our unpublished A17 adapter.
Matched protocol
The same pinned FLEURS Filipino development audio was scored under clean audio, seeded white noise at 10 dB SNR, and a seeded 200 ms dropout. Greedy Tagalog decoding used 30-second chunks and a 448-token cap.
Primary normalization lowercased and collapsed whitespace after NFKC. Punctuation and accents remained in the score. All 12 case scores passed local independent recomputation.
Recommendation within this benchmark
Use Large as the quality reference and Small as the compact baseline. Small PLD improved clean and dropout WER, but had worse CER in every condition and worse noisy WER than unadapted Small.
Reuse the results
Published model-card scores use other protocols. This table ranks only the four models under this shared development benchmark.