R_REDDYX.XYZ
LLMPython★ 12,607⑂ 851 forksApache-2.0

bentoml/OpenLLM

OPENLLM: RUN LLMS AS OPENAI API

OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and ready-made Docker and Kubernetes deployment recipes.

// KEY FEATURES

  • Run any open-source LLM as OpenAI-compatible API with one command
  • Built-in web chat at /chat endpoint
  • Ready Docker and Kubernetes configurations for production deployment
  • Support for custom models via model repository

// DEEP DIVE: HOW IT WORKS

OpenLLM installs via pip and operates from CLI. `openllm serve model` starts a server on port 3000 with OpenAI-compatible API. Models are pulled from Hugging Face (HF_TOKEN required for gated ones), the framework itself does not store weights. Includes web chat at `/chat` and terminal mode via `openllm run`.

Unified interface for dozens of models: Llama 3.x, Qwen 2.5, DeepSeek, Gemma, Phi, Mistral, multimodal Pixtral and others. README provides VRAM table: from 12 GB to 80 GB×16. Under the hood — modern inference backends (vLLM, SGLang) hidden behind simple API. Custom models added via model repository.

Requires GPU with VRAM matching the model. For gated models — HF token and HF access. Production deployment simplified via Docker/Kubernetes and BentoML integration, but still requires orchestration skills.

// USE CASES

Best for: For ML engineers and backend developers to quickly deploy open LLMs as APIs

  • Deploy Llama 3.3 70B as OpenAI API on own hardware in a minute
  • Integrate local Qwen2.5-Coder into IDE via OpenAI-compatible client
  • Run multimodal Pixtral in Kubernetes with web chat interface

// QUICK START

pip install openllm
openllm hello
openllm serve llama3.2:1b

Commands are taken from the repository README — check the current version on GitHub before running.

#bentoml#fine-tuning#llama#llama2#llama3-1#llama3-2#llama3-2-vision#llm#llm-inference#llm-ops#llm-serving#llmops
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

OPEN-SOURCE FINANCIAL LLMS FOR DATA ANALYSIS★ 21591STATIC CHAT FOR DEEPSEEK API★ 1701FULL LOGS OF LLM EXPERIMENTS UNDERSTANDING NEURAL NETWORK WEIGHTS★ 1243RISC-V CPU COMPILED INTO QWEN3.5 WEIGHTS★ 814REA - REVERSE ENGINEERING WITHOUT BOUNDARIES★ 460891DeepSeek Plugin Framework★ 281341

← FULL CATALOG

bentoml/OpenLLM — Openllm: Run Llms AS Openai API