AITypeScript★ 860⑂ 65 forksMIT
niieani/gpt-tokenizer
GPT-TOKENIZER: RAPIDER TOKENIZER FOR GPT
gpt-tokenizer is the fastest JavaScript BPE tokenizer encoder decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). It is a port of tiktoken with features and written in TypeScript.
// KEY FEATURES
- Encodes and decodes text into GPT model tokens in microseconds.
- Supports all tiktoken versions: gpt‑2, gpt‑3, gpt‑4, gpt‑5 and gpt‑o* series.
- Provides API for batch processing and streaming tokenization of large corpora.
- Built-in tests and TypeScript types ensure reliability when integrating into projects.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram