R_REDDYX.XYZ
AIRust★ 412⑂ 76 forksMIT

zurawiki/tiktoken-rs

tiktoken-rs - FAST TOKENIZER FOR GPT

A Rust library providing a ready-made tokenizer compatible with OpenAI's tiktoken for GPT models.

// KEY FEATURES

  • Support for all GPT-2/3/4 models
  • Zero-dependencies, pure Rust
  • High tokenization speed
#bpe#openai#rust#tokenizer
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

IGNITER - CODE GENERATION ON ELIXIR★ 412LIONAGI - AI Orchestra for Code★ 412IQ-SERIES - AI IN THE CLOUD★ 412LLM Inference Engineering★ 412MERIDIAN: AUTOMATIC WORK JOURNAL★ 412Video wizard for audio stories★ 412

← FULL CATALOG

zurawiki/tiktoken-rs — tiktoken-rs - FAST TOKENIZER FOR GPT