flashtext

library2.7pypypi✓ verified 30d ago

Flashtext is a Python library designed for efficient keyword extraction and replacement in sentences. It employs a custom algorithm based on Aho-Corasick and Trie data structures, providing significant performance gains over regular expressions, especially for large dictionaries of keywords. The current stable version is 2.7, released in 2018, and it is largely in a maintenance state, though still widely used.

data ai-ml

pip install flashtext

github ↗package ↗

INSTALL

IMPORT

SIG · FLASHTEXT

flashtext

datapythonv2.7

Install

2.7s avg

Import

—

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.7 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 19.2MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.7s · import 0.000s · 20MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

KeywordProcessor

✓ from flashtext import KeywordProcessor

This example demonstrates how to initialize the `KeywordProcessor`, add keywords with optional clean names, and then use it to extract or replace keywords in a given text. It also shows how to get span information for extracted keywords.

from flashtext import KeywordProcessor # Initialize the keyword processor (case_sensitive=False by default) keyword_processor = KeywordProcessor() # Add keywords. Can map multiple 'unclean' names to one 'clean' name. keyword_processor.add_keyword('Big Apple', 'New York') keyword_processor.add_keyword('Bay Area') keyword_processor.add_keyword('New Delhi', 'NCR region') # Extract keywords text_to_extract = 'I love Big Apple and Bay Area. New Delhi is also great.' keywords_found = keyword_processor.extract_keywords(text_to_extract) print(f"Extracted keywords: {keywords_found}") # Expected: ['New York', 'Bay Area', 'NCR region'] # Replace keywords text_to_replace = 'I love Big Apple and new delhi.' new_sentence = keyword_processor.replace_keywords(text_to_replace) print(f"Replaced sentence: {new_sentence}") # Expected: 'I love New York and NCR region.' # Extract with span information keywords_with_span = keyword_processor.extract_keywords('I love Big Apple.', span_info=True) print(f"Keywords with span: {keywords_with_span}") # Expected: [('New York', 7, 16)]

Debug

Known issues

gotchaFlashtext's default word boundary definition (`[A-Za-z0-9_]`) might not be suitable for all languages (e.g., Chinese, Japanese) or custom requirements. It may fail to identify keywords correctly if they are not separated by these specific non-word characters. Users can customize `non_word_boundaries`.

fix

For non-Latin languages or custom boundary needs, initialize `KeywordProcessor` with a modified `non_word_boundaries` set. Example: `kp = KeywordProcessor(non_word_boundaries=set(['@', '#']))`.

affects: 2.0 - 2.7

gotchaFlashtext generally outperforms regex for keyword extraction/replacement when the number of keywords is large (typically >500). For a small number of keywords or when complex patterns (like partial matches or special character handling) are required, regular expressions might be equally or more efficient, or simply the only solution.

fix

Benchmark performance for your specific use case. If keyword count is low or complex pattern matching is needed, consider standard regex. If keyword count is high, Flashtext is highly optimized for speed.

affects: 2.0 - 2.7

gotchaIf `add_keyword()` is used with a tuple as the `clean_name` (e.g., `add_keyword('Taj Mahal', ('Monument', 'Taj Mahal'))`), the `replace_keywords()` method will not function as expected because it anticipates a string replacement, not a tuple.

fix

Ensure that `add_keyword()` is only provided with string `clean_name` values if `replace_keywords()` functionality is intended. Tuple clean names are primarily for enhanced extraction information.

affects: 2.0 - 2.7

deprecatedA separate, community-driven package `flashtext2` (and `flashtextr`) exists, which is a rewrite in Rust, offering significant performance improvements (3-10x faster) and better Unicode handling. While not an official successor from the original author, it addresses some limitations of `flashtext`.

fix

Consider migrating to `flashtext2` for improved performance and broader language support, especially if hitting performance bottlenecks or unicode issues with the original `flashtext`. Be aware of potential API differences or slight behavior changes, though the core API is similar.

affects: All versions of `flashtext` (2.x)

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'flashtext'

The 'flashtext' library is not installed in the Python environment, or the environment where the code is run does not have it installed.

fix

Install the library using pip: `pip install flashtext`

IndexError: string index out of range (in flashtext.keyword.py)

This error typically occurs when `replace_keywords` is used with a sentence containing non-ASCII characters or in case-insensitive mode, where the length of the lowercased string differs from the original, causing an incorrect index lookup.

fix

Ensure that `case_sensitive` is set appropriately, especially for text with special characters. If the issue persists, consider pre-processing non-ASCII characters or ensuring the Flashtext version is compatible with your Python environment's string handling. One reported fix involved adjusting string length assumptions in the library's internal code.

flashtext.KeywordProcessor not extracting all keywords or returning NaN in DataFrame

When integrating `flashtext` with dataframes (e.g., Pandas or PySpark), the `KeywordProcessor` might not be correctly applied across all rows or partitions, leading to incomplete extraction or 'NaN' values. This often happens due to incorrect application of UDFs (User Defined Functions) in distributed environments or subtle data type mismatches.

fix

Ensure the `KeywordProcessor` object is correctly initialized and the extraction function is applied properly across the DataFrame. For PySpark, this often involves creating the `KeywordProcessor` instance and using a UDF to apply `extract_keywords` to a column.

Upgrade

Version history

2.7latest on PyPI · released Feb 16, 2018

Audit

Dependencies

No dependency data recorded yet.

Agent activity

10 hits · last 30 days

node

OpenAI (training)

Resources

githubhttp://github.com/vi3k6i5/flashtext ↗

packagehttps://pypi.org/project/flashtext/ ↗