Project card — stars, tech stack, link to sources — in the REDDYX catalog: BerriAI/litellm.
PROBLEM: VENDOR API ZOO
We built an integration for OpenAI — but six months later we needed to add Anthropic for some tasks and a local model for sensitive data. Without a single abstraction layer, that means three different SDKs, three different error formats, and code full of if provider == "...". LiteLLM removes this fork: one call, one response format, the provider is selected by a string parameter.
WHAT LITELLM GIVES
- Unified call format — OpenAI-compatible interface over 100+ providers (OpenAI, Anthropic, Google, Bedrock, Ollama and others).
- Proxy Server — a standalone service placed in front of all providers: applications hit a single endpoint without knowing where the request actually goes.
- Routing and load balancing — distribution of load across multiple keys or providers by strategy (round-robin, by cost, by latency).
- Fallback — if the primary provider fails or hits a rate limit, the request is automatically routed to the backup, and the application doesn’t notice the outage.
- Cost tracking and limits — tracking spend by keys/commands and hard budget limits at the proxy level.
QUICK START: SDK
pip install litellm
from litellm import completion
# один и тот же вызов для разных провайдеров
response = completion(
model="gpt-4.1", # или "claude-sonnet-4", "ollama/llama3"
messages=[{"role": "user", "content": "Привет!"}]
)
print(response.choices[0].message.content)
PROXY SERVER WITH FALLBACK: CONFIG
# config.yaml
model_list:
- model_name: main-model
litellm_params:
model: gpt-4.1
api_key: os.environ/OPENAI_API_KEY
- model_name: main-model
litellm_params:
model: claude-sonnet-4
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
fallbacks: [{"main-model": ["main-model"]}]
num_retries: 2
Launch: litellm --config config.yaml. The application talks to the proxy at http://localhost:4000 just like to a regular OpenAI API — all provider switching logic is hidden inside.
LITELLM VS DIRECT PROVIDER SDKS
| Criterion | LiteLLM | Direct provider SDKs |
|---|---|---|
| Provider switch | Parameter string | Rewrite integration |
| Fallback on failure | Built-in | You write it yourself |
| Cost tracking per command | Out-of-the-box in proxy | Need your own layer |
| Speed of integrating a new provider | Minutes | Hours on SDK and error handling |
Frequently asked questions
Does LiteLLM add latency to requests?
The proxy adds minimal overhead — essentially one extra HTTP hop. Against the network latency to the LLM provider itself, this is hardly noticeable.
Do I need to change existing code that uses the OpenAI SDK?
Often not — the proxy server mimics the OpenAI API, so you just need to change base_url in your existing client to the LiteLLM address.
Is it suitable for local models via Ollama?
Yes, Ollama is one of the supported providers, you can mix local and cloud models in a single routing with fallback between them.
We review fresh open-source tools every day — from data pipelines to security scanners. If you want to be among the first to see what actually works in 2026 — subscribe to the REDDYX AI Telegram channel — new repositories every 30-60 minutes.
