Why the comparison is relevant right now
Two years ago, GPT-4 was unattainable for open-source. Now the picture is different: Meta, Alibaba, DeepSeek have released models that cover 90% of production tasks. The question is no longer "can open-source do it," but "in which tasks is paying for API still justified."
Comparison Table
| Model | MMLU | HumanEval | Parameters | License |
|---|---|---|---|---|
| GPT-4o (baseline) | 88.7 | 90.2 | closed | proprietary |
| DeepSeek V3 | 88.5 | 89.1 | 671B MoE | MIT |
| Llama 3.1 405B | 87.3 | 84.1 | 405B | Llama License |
| Qwen 2.5 72B | 86.1 | 86.6 | 72B | Apache 2.0 |
| Mistral Large 2 | 84.0 | 92.1 | 123B | MRL |
DeepSeek V3: Unexpected Leader
671B parameters with Mixture-of-Experts architecture — only 37B actively used per token. Training cost $5.5M vs $100M+ for GPT-4. On HuggingFace 4M+ downloads in the first month after release. For coding tasks, outperforms GPT-4o in a number of tests.
Llama 3.1 405B: Meta Bets on Openness
405B parameters with 128K token context. Supports function calling, multilingual (8 languages including Russian). On Together AI, rental costs $5/M tokens vs $15/M for GPT-4o. For enterprise, there's Llama Guard for safety filtering.
Where Open-Source Still Falls Short
Multimodal tasks — GPT-4V and Claude 3.5 Sonnet still better at complex image analysis. Instruction following on edge cases — proprietary models follow complex instructions more accurately. But for 90% of production tasks, the difference is negligible.
FAQ
Which model to choose to start?
Qwen 2.5 72B — best balance of size and quality, Apache 2.0 license, works well with Russian language.
Where to run large models without your own GPU?
Together AI, Fireworks AI, Groq — all support Llama 3 and Qwen 2.5 with prices lower than OpenAI.