tokenizer

library0.3.0rbrubygemsunverified

A multilingual tokenizer for NLP tasks supporting German, English, and French texts.

gem install tokenizer

INSTALL

IMPORT

SIG · TOKENIZER

tokenizer

utilityrubyv0.3.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tokenizer

✓ require 'tokenizer'

Tokenize an English sentence.

require 'tokenizer' tokens = Tokenizer.tokenize('Hello world!', language: :english) puts tokens.inspect

Debug

Known issues

No known issues recorded.

Upgrade

Version history

0.3.0latest on RubyGems

Audit

Dependencies

No dependency data recorded yet.

Agent activity

14 hits · last 30 days

node

Resources