Indras-Mirror/Xolotl
RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC
Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent tasks with 512K context via expert tiering across VRAM/RAM/NVMe.
// KEY FEATURES
- DeepSeek-V4.1-Flash 552B on RTX 4090 + 110 GB RAM: 33 tok/s
- MoE expert tiering across VRAM, pinned RAM and NVMe
- OpenAI/Anthropic API, 512K context, fast mode 41 tok/s
- Supports GLM-5.3-Flash, Qwen3.8, LFM3-Omni, Muse Glimmer
// DEEP DIVE: HOW IT WORKS
Xolotl is built on top of Strata (Niko1221's engine) and extended by over 70,000 lines of code. The engine implements custom backends for DeepSeek-V4.1-Flash, GLM-5.3-Flash, dense Qwen3.8-27B, LFM3-Omni, and Muse Glimmer 30B. MoE experts are distributed across tiers: VRAM, pinned system RAM, and NVMe, with Xolotl Direct loading straight from NVMe to VRAM. The server provides an OpenAI- and Anthropic-compatible API with 512K context support.
It differs from alternatives with deep expert-tiering integration and architecture-specific optimizations: DFlash2 for Qwen3.8, multi-column attention, gated residuals (hyper-connections) for Qwen4-like models, looped transformers (SenseNova-Loop-9B). Features include fast mode (41 tok/s), reasoning loop recovery, built-in web search tools (Keenable keyless, Keirolabs/Exa/Perplexity/DeepSeek with keys). Code is MIT-licensed, Linux-only with NVIDIA GPU.
Requirements: for full DeepSeek-V4.1-Flash 552B — RTX 4090 (24 GB VRAM) and 110 GB system RAM; dense models like Muse Glimmer 30B run on a single 24 GB card. Linux and CUDA drivers required. Some web tools need API keys. sm_70 (Volta) build supported, security fixes included (Pillow JPEG 2000).
// USE CASES
Best for: Developers running large MoE locally without a cluster
- Run DeepSeek-V4.1-Flash 552B as a local coding agent on a single 4090
- Serve multiple MoE and dense models via a single OpenAI-compatible API
- Use web search and tool-calling without cloud dependencies
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram