R_REDDYX.XYZ
AITypeScript★ 860⑂ 65 forksMIT

niieani/gpt-tokenizer

GPT-TOKENIZER: RAPIDER TOKENIZER FOR GPT

gpt-tokenizer is the fastest JavaScript BPE tokenizer encoder decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). It is a port of tiktoken with features and written in TypeScript.

// KEY FEATURES

  • Encodes and decodes text into GPT model tokens in microseconds.
  • Supports all tiktoken versions: gpt‑2, gpt‑3, gpt‑4, gpt‑5 and gpt‑o* series.
  • Provides API for batch processing and streaming tokenization of large corpora.
  • Built-in tests and TypeScript types ensure reliability when integrating into projects.
#bpe#decoder#encoder#gpt-2#gpt-3#gpt-4#gpt-4o#gpt-5#gpt-o1#machine-learning#openai#tokenizer
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

HOOP - Unified Protocol Gateway★ 859Live Architecture Visualizer★ 859Local Knowledge Graph MCP★ 858ECA: AI Code Assistant★ 858HALO: FRAMEWORK FOR LLM TRAINING★ 862INCARNAMIND AI CHAT WITH DOCUMENTS★ 858

← FULL CATALOG

niieani/gpt-tokenizer — GPT-Tokenizer: Rapider Tokenizer For…