bentoml/OpenLLM
OPENLLM: RUN LLMS AS OPENAI API
OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and ready-made Docker and Kubernetes deployment recipes.
// KEY FEATURES
- Run any open-source LLM as OpenAI-compatible API with one command
- Built-in web chat at /chat endpoint
- Ready Docker and Kubernetes configurations for production deployment
- Support for custom models via model repository
// DEEP DIVE: HOW IT WORKS
OpenLLM installs via pip and operates from CLI. `openllm serve model` starts a server on port 3000 with OpenAI-compatible API. Models are pulled from Hugging Face (HF_TOKEN required for gated ones), the framework itself does not store weights. Includes web chat at `/chat` and terminal mode via `openllm run`.
Unified interface for dozens of models: Llama 3.x, Qwen 2.5, DeepSeek, Gemma, Phi, Mistral, multimodal Pixtral and others. README provides VRAM table: from 12 GB to 80 GB×16. Under the hood — modern inference backends (vLLM, SGLang) hidden behind simple API. Custom models added via model repository.
Requires GPU with VRAM matching the model. For gated models — HF token and HF access. Production deployment simplified via Docker/Kubernetes and BentoML integration, but still requires orchestration skills.
// USE CASES
Best for: For ML engineers and backend developers to quickly deploy open LLMs as APIs
- Deploy Llama 3.3 70B as OpenAI API on own hardware in a minute
- Integrate local Qwen2.5-Coder into IDE via OpenAI-compatible client
- Run multimodal Pixtral in Kubernetes with web chat interface
// QUICK START
pip install openllm
openllm hello
openllm serve llama3.2:1bCommands are taken from the repository README — check the current version on GitHub before running.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram