sacremoses

library0.2.0pypypi✓ verified 29d ago

SacreMoses is a Python port of the widely-used Moses tokenizer, truecaser, and punctuation normalizer tools, essential for many Natural Language Processing (NLP) tasks, particularly in machine translation workflows. The current version is 0.1.1. Releases are made periodically, addressing bug fixes, performance improvements, and alignment with the original Perl implementation.

ai-ml data serialization

pip install sacremoses

github ↗package ↗

INSTALL

IMPORT

SIG · SACREMOSES

sacremoses

ai-mlpythonv0.2.0

Install

2.7s avg

Import

1733ms

Disk

26MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.2.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 1.810s · 27.3MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.7s · import 1.656s · 28MB

26MB installed

● package 26MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

MosesTokenizer

✓ from sacremoses import MosesTokenizer

✗ from nltk.tokenize.moses import MosesTokenizer

MosesTokenizer was previously part of NLTK but was moved to sacremoses due to licensing issues.

MosesDetokenizer

✓ from sacremoses import MosesDetokenizer

MosesPunctNormalizer

✓ from sacremoses import MosesPunctNormalizer

MosesTruecaser

✓ from sacremoses import MosesTruecaser

This quickstart demonstrates the core functionalities: punctuation normalization and the tokenization/detokenization of a sample English sentence.

from sacremoses import MosesTokenizer, MosesDetokenizer, MosesPunctNormalizer # Punctuation Normalization mpn = MosesPunctNormalizer(lang='en') text_to_normalize = 'THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."' normalized_text = mpn.normalize(text_to_normalize) print(f"Normalized: {normalized_text}") # Tokenization and Detokenization mt = MosesTokenizer(lang='en') md = MosesDetokenizer(lang='en') sample_text = "Hello, world! This is a test sentence with numbers 123 and some special characters like @#$%." tokenized_list = mt.tokenize(sample_text) detokenized_text = md.detokenize(tokenized_list) print(f"Original: {sample_text}") print(f"Tokenized: {tokenized_list}") print(f"Detokenized: {detokenized_text}")

sacremoses --version

Debug

Known issues

breakingSacreMoses dropped official support for Python 2. If you are using Python 2, you must use `sacremoses==0.0.40` or an earlier version. Later versions (`sacremoses>=0.0.41`) require Python 3.

fix

Upgrade to Python 3.8+ or pin sacremoses to version 0.0.40.

affects: <0.0.41 (Python 2)

breakingVersion 0.1.0 introduced changes that can affect output, including how `use_known` works in `MosesTruecaser.truecase()` and how the order of `protected_patterns` is handled in `MosesTokenizer.tokenize()`.

fix

Review your code for any reliance on previous behavior, especially regarding truecasing and custom protected patterns. Retest your NLP pipeline after updating.

affects: >=0.1.0

gotchaThe `MosesPunctNormalizer` gained a `perl_parity:bool` argument in version 0.1.0 to align behavior with the latest Perl Moses implementation. This argument might become the default or only behavior in future releases.

fix

Consider explicitly setting `perl_parity=True` in `MosesPunctNormalizer` to ensure future compatibility and consistent behavior with the latest Perl Moses. Be aware that this might subtly change normalization output.

affects: >=0.1.0

gotchaWhen implementing custom span tokenization by subclassing `MosesTokenizer`, be cautious with the `escape`, `unescape`, and `detokenize` interactions, as they can sometimes lead to flaky results.

fix

Thoroughly test any custom span tokenization logic, especially when dealing with escaped characters or complex detokenization scenarios. Consult existing implementations if available.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'sacremoses'

The `sacremoses` library is not installed in the current Python environment.

fix

pip install sacremoses

AttributeError: module 'sacremoses' has no attribute 'Tokenizer'

The main tokenizer class in `sacremoses` is `MosesTokenizer`, not a generic `Tokenizer` directly under the `sacremoses` module.

fix

from sacremoses import MosesTokenizer
tokenizer = MosesTokenizer()

FileNotFoundError: [Errno 2] No such file or directory: '.../sacremoses/truecaser/truecase-model.en'

The necessary truecasing model for the specified language could not be downloaded, or the download was interrupted/corrupted, leading to the file not being present when `MosesTruecaser` attempts to load it.

fix

Ensure an active internet connection, clear any proxy or firewall issues, and try re-initializing `MosesTruecaser` to trigger a model download. You might need to delete incomplete model files manually.

TypeError: tokenize() missing 1 required positional argument: 'text'

The `tokenize` method of `MosesTokenizer` requires the text to be tokenized as its first argument.

fix

tokenizer = MosesTokenizer()
tokenized_text = tokenizer.tokenize('This is a test sentence.')

Upgrade

Version history

0.2.0latest on PyPI · released Aug 14, 2026

Audit

Dependencies

No dependency data recorded yet.

Agent activity

12 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/hplt-project/sacremoses ↗

packagehttps://pypi.org/project/sacremoses/ ↗