AIRust★ 4,155⑂ 225 forksMIT
marcelroed/gigatoken
GIGATOKEN — GB/s Tokenization
gigatoken — fast tokenization of language models in Rust, achieving gigabyte-per-second speeds.
// KEY FEATURES
- Tokenization at speeds >1 GB/s
- Written in safe Rust
- Supports popular BPE, WordPiece tokenizers
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram