AIRust★ 412⑂ 76 forksMIT
zurawiki/tiktoken-rs
tiktoken-rs - FAST TOKENIZER FOR GPT
A Rust library providing a ready-made tokenizer compatible with OpenAI's tiktoken for GPT models.
// KEY FEATURES
- Support for all GPT-2/3/4 models
- Zero-dependencies, pure Rust
- High tokenization speed
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram