Whisper Normalizer (version 0.1.12) is a Python package that implements the text standardization and normalization approach used in OpenAI's Whisper ASR model. It provides robust text normalization functionalities, crucial for evaluating Automatic Speech Recognition (ASR) systems by reducing unintentional penalties in metrics like WER and CER. The library extends beyond basic English normalization to include specialized normalizers for Indic languages, addressing challenges such as diacritic preservation. The project appears to have an active development and release cadence.
pip install whisper-normalizerVerified import paths — ran on the pinned version, not inferred.
Demonstrates the use of EnglishTextNormalizer, BasicTextNormalizer, and a specific Indic normalizer (MalayalamNormalizer) to process and normalize text. This highlights the different normalization strategies available for various linguistic contexts.
Always import and use the appropriate language-specific normalizer from `whisper_normalizer.indic_normalizer` (e.g., `MalayalamNormalizer`, `PunjabiNormalizer`) for Indic languages.
Users upgrading from versions prior to 0.1.4 should carefully review normalization outputs in their applications, especially for number-heavy text, to ensure compatibility with the new implementations.
For standalone text normalization tasks, ensure you are importing from `whisper_normalizer`. If integrating with the full OpenAI Whisper ASR model, use its internal `whisper.normalizers`.
pip install whisper-normalizer
from whisper_normalizer.basic import BasicTextNormalizer
Ensure the input passed to the `normalize()` method is always a string. For example: `normalizer.normalize(str(123))`
Use one of the supported language codes (e.g., 'hi', 'ta', 'bn') listed in the error message for `IndicTextNormalizer`.