tiktoken-rs

library0.12.0rscratesunverified

Rust library for encoding and decoding text using OpenAI's tiktoken tokenizer.

# Cargo.toml
[dependencies]
tiktoken-rs = "0.12.0"

github ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · TIKTOKEN-RS

tiktoken-rs

utilityrustv0.12.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

cl100k_base

✓ use tiktoken_rs::cl100k_base;

Encode a string using the cl100k_base tokenizer.

use tiktoken_rs::cl100k_base; fn main() { let bpe = cl100k_base().unwrap(); let tokens = bpe.encode_with_special_tokens("Hello, world!"); println!("Token count: {}", tokens.len()); }

Debug

Known issues

gotchaRequires downloading tokenizer files on first use; may fail without network access.

fix

Ensure network access or pre-cache the tokenizer files.

affects: >=0.1.0

Upgrade

Version history

0.12.0latest on crates.io

Audit

Dependencies

No dependency data recorded yet.

Agent activity

15 hits · last 30 days

node

Amazon

Resources

githubhttps://github.com/zurawiki/tiktoken-rs ↗

homepagehttps://github.com/zurawiki/tiktoken-rs ↗

docshttps://docs.rs/tiktoken-rs/ ↗