R_REDDYX.XYZ
AgentsC++★ 534⑂ 65 forksMIT

Indras-Mirror/Xolotl

RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC

Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent tasks with 512K context via expert tiering across VRAM/RAM/NVMe.

// KEY FEATURES

  • DeepSeek-V4.1-Flash 552B on RTX 4090 + 110 GB RAM: 33 tok/s
  • MoE expert tiering across VRAM, pinned RAM and NVMe
  • OpenAI/Anthropic API, 512K context, fast mode 41 tok/s
  • Supports GLM-5.3-Flash, Qwen3.8, LFM3-Omni, Muse Glimmer

// DEEP DIVE: HOW IT WORKS

Xolotl is built on top of Strata (Niko1221's engine) and extended by over 70,000 lines of code. The engine implements custom backends for DeepSeek-V4.1-Flash, GLM-5.3-Flash, dense Qwen3.8-27B, LFM3-Omni, and Muse Glimmer 30B. MoE experts are distributed across tiers: VRAM, pinned system RAM, and NVMe, with Xolotl Direct loading straight from NVMe to VRAM. The server provides an OpenAI- and Anthropic-compatible API with 512K context support.

It differs from alternatives with deep expert-tiering integration and architecture-specific optimizations: DFlash2 for Qwen3.8, multi-column attention, gated residuals (hyper-connections) for Qwen4-like models, looped transformers (SenseNova-Loop-9B). Features include fast mode (41 tok/s), reasoning loop recovery, built-in web search tools (Keenable keyless, Keirolabs/Exa/Perplexity/DeepSeek with keys). Code is MIT-licensed, Linux-only with NVIDIA GPU.

Requirements: for full DeepSeek-V4.1-Flash 552B — RTX 4090 (24 GB VRAM) and 110 GB system RAM; dense models like Muse Glimmer 30B run on a single 24 GB card. Linux and CUDA drivers required. Some web tools need API keys. sm_70 (Volta) build supported, security fixes included (Pillow JPEG 2000).

// USE CASES

Best for: Developers running large MoE locally without a cluster

  • Run DeepSeek-V4.1-Flash 552B as a local coding agent on a single 4090
  • Serve multiple MoE and dense models via a single OpenAI-compatible API
  • Use web search and tool-calling without cloud dependencies
#abliteration#cuda#deepseek#deepseek-v4#deepseek-v4-1#gguf#glm#glm-5#llm-inference#local-llm#mixture-of-experts#moe
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

WHITEBOARD WITH MCP FOR RESEARCH AI AGENTS★ 512AGENT SKILL FOR CLEANING UI FROM AI PATTERNS★ 570CAD SYSTEM WITH AGENTS FOR CADQUERY★ 490WEBSITE RECONSTRUCTION BENCHMARK FOR AGENTS★ 47645 AGENT SKILLS FOR PRODUCT ENGINEERING★ 611MCP SERVER FOR BROWSER CONTROL BY AI AGENTS★ 612

← FULL CATALOG

Indras-Mirror/Xolotl — Running Deepseek-V4.1 552b ON A…