AIC#★ 215⑂ 36 forksMIT
microsoft/Tokenizer
TOKENIZER BPE FOR OPENAI
Tokenizer — BPE tokenizer for TypeScript and .NET for OpenAI models. Trains vocab, encodes and decodes tokens, compatible with tiktoken. Easily integrates into AI products.
// KEY FEATURES
- Trains a BPE vocabulary from any text and saves the vocab as JSON for later use.
- Encodes strings into token arrays and decodes tokens back to UTF‑8 strings.
- Fully compatible with OpenAI's tiktoken format, allowing loading of pretrained vocab files such as gpt2, cl100k.
- Works in browsers (TypeScript) and .NET applications without external dependencies, easy to integrate.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram