WHY THIS IS NEEDED
Three reasons why teams self‑host instead of subscribing:
- Privacy. Code, contracts and correspondence do not go to someone else's cloud. For many industries this is not a wish but a requirement.
- Cost. A subscription for a team of 20 people per year costs as much as a decent graphics card, which then works forever.
- Availability. No dependence on blocks, foreign currency cards, or sudden tariff changes.
WHAT YOU NEED
| Component | Minimum | Comfortable |
|---|---|---|
| GPU | RTX 3060 12GB | RTX 4090 / A6000 |
| RAM | 16 GB | 64 GB |
| Disk | 100 GB SSD | 1 TB NVMe |
| OS | Ubuntu 22.04+ | Ubuntu 24.04 LTS |
Without a GPU it will still work, but on CPU generation proceeds at 1‑2 tokens per second — fine for a demo, not for production.
STEP 1: DOCKER COMPOSE
One file brings up the whole stack. We place it in /opt/ai-server/docker-compose.yml:
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
volumes:
- ./ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=-1
- OLLAMA_NUM_PARALLEL=4
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
depends_on:
- ollama
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./webui:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=True
- ENABLE_SIGNUP=False
Two important notes. The port is forwarded to 127.0.0.1, not outward — only nginx will be exposed externally. And ENABLE_SIGNUP=False: otherwise anyone who finds your address can register themselves.
cd /opt/ai-server
docker compose up -d
docker compose exec ollama ollama pull qwen2.5:14b
The first registered user automatically becomes an administrator — log in right after startup and create your account before someone else does.
STEP 2: NGINX AND HTTPS
You cannot expose port 8080 to the internet: there is neither encryption nor protection against brute force. We install nginx.
server {
server_name ai.example.com;
client_max_body_size 100M; # загрузка документов в RAG
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# стриминг ответов: без этого текст появляется рывками
proxy_buffering off;
proxy_read_timeout 600s;
}
listen 443 ssl;
}
Two lines that are often forgotten: proxy_buffering off — otherwise the model's response accumulates in the buffer and comes out in chunks instead of character‑by‑character streaming, and proxy_read_timeout — otherwise long responses are cut off by timeout.
certbot --nginx -d ai.example.com
STEP 3: USERS AND ROLES
Open WebUI can restrict access out of the box. It is configured in the admin panel:
- Roles — admin, user, pending. New accounts can be kept in pending until approval.
- Model access — keep the heavy 70B model only for part of the team, others get 8B.
- Groups — different sets of models and prompts for departments.
- History — each user has their own; by default the admin does not see others' chats.
STEP 4: RAG ON COMPANY DOCUMENTS
The most valuable part for business: the model answers based on your regulations, not general knowledge from the internet.
In Open WebUI this is the Workspace → Knowledge section. You upload PDF, docx, markdown — the system automatically chunks them, computes embeddings and stores them in a vector database. Then in chat you specify the collection via # and the model answers based on it.
What to tweak for quality:
| Parameter | Value | Why |
|---|---|---|
| Chunk size | 800-1200 | Too small chunks lose context |
| Chunk overlap | 100-200 | so the thought isn't split between chunks |
| Top K | 4-6 | more — noise, less — missing facts |
| Embedding model | multilingual | mandatory for Russian documents |
The default English embedding model works noticeably worse on Russian documents — this is the first thing to change.
THE PITFALLS EVERYONE STEPS ON
- The model is unloaded between requests. The first answer after a pause takes a minute. Fixed with
OLLAMA_KEEP_ALIVE=-1, the model stays in memory. - Open registration. Forgot
ENABLE_SIGNUP=False— and your server is used by the whole internet. - Disk ran out. Models take up tens of gigabytes, chat history and vector database grow. Monitor disk space.
- One request hangs everyone. With
OLLAMA_NUM_PARALLEL=1the command waits in queue. Set it according to the number of users if memory allows. - No backup. The
webuifolder contains all history and settings. Lose it — lose everything.
HOW MUCH DOES IT COST
One-time investments versus subscription for a team of 20 people:
| Item | Your hardware | Cloud subscription |
|---|---|---|
| Start | price of a GPU server | 0 |
| Per month | electricity | 20 × rate per user |
| After a year | hardware remains yours | pay again |
| Privacy | full | according to vendor agreement |
| Model quality | lower than top | best on the market |
Honest conclusion: your hardware wins over a year horizon and where privacy is critical. If you need the maximum model right here and now — cloud is stronger.
FAQ
Will the server handle a team of 20 people?
If not everyone works simultaneously — yes, one 4090 is enough. With real parallel load, calculate based on OLLAMA_NUM_PARALLEL and memory: each parallel slot requires its own KV cache.
Can cloud models also be connected?
Yes, Open WebUI can work with any OpenAI-compatible API simultaneously with Ollama. Convenient scheme: private — locally, complex — in the cloud.
How to update?
docker compose pull && docker compose up -d. Data in mounted folders is not touched by the update. Still make a backup of the webui folder before updating.
Does it work with Russian documents in RAG?
It works, but you must switch the embedding model to a multilingual one in the settings. With the default English model, search quality on Russian texts noticeably drops.
