AIPython★ 4,343⑂ 401 forksApache-2.0
algorithmicsuperintelligence/optillm
OPTILLM — SPEEDING LLM INFERENCE
optillm — a proxy accelerator for LLMs that reduces latency and saves resources during inference.
// KEY FEATURES
- Support for HuggingFace, vLLM, TensorRT‑LLM
- Dynamic batching and KV caching
- HTTP API compatible with OpenAI client
#agent#agentic-ai#agentic-framework#agentic-workflow#agents#api-gateway#chain-of-thought#genai#large-language-models#llm#llm-inference#llmapi
Open on GitHub →New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram