R_REDDYX.XYZ
AIRust★ 4,155⑂ 225 forksMIT

marcelroed/gigatoken

GIGATOKEN — GB/s Tokenization

gigatoken — fast tokenization of language models in Rust, achieving gigabyte-per-second speeds.

// KEY FEATURES

  • Tokenization at speeds >1 GB/s
  • Written in safe Rust
  • Supports popular BPE, WordPiece tokenizers
#llm#nlp#tokenization#tokenizer
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

AdalFlow - Boost LLM Efficiency★ 4153Build AI agents locally★ 4162Universal Native App Framework★ 4162SPARK-NLP - REVOLUTION IN NLP★ 4163OpenClaw Chinese Use Cases★ 4146NOTFAIR-PLUGIN: SEO SKILLS★ 4168

← FULL CATALOG

marcelroed/gigatoken — GIGATOKEN — GB/s Tokenization