microsoft/Tokenizer

Tokenizer — BPE tokenizer for TypeScript and .NET for OpenAI models. Trains vocab, encodes and decodes tokens, compatible with tiktoken. Easily integrates into AI products.

// KEY FEATURES

• Trains a BPE vocabulary from any text and saves the vocab as JSON for later use.
• Encodes strings into token arrays and decodes tokens back to UTF‑8 strings.
• Fully compatible with OpenAI's tiktoken format, allowing loading of pretrained vocab files such as gpt2, cl100k.
• Works in browsers (TypeScript) and .NET applications without external dependencies, easy to integrate.

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.