Speculative Decoding: open-source projects on GitHub
23 repositories on Speculative Decoding, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
HyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonMTPLX SUPERFAST LAUNCH
MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a
★ 2,706 · PythonLocal AI Engine
Splash lets you run machine learning models directly on Apple‑silicon Macs, using the full power of CPU and GPU without relying on the cloud.
★ 2,284 · PythonLucebox Hub
High-speed server for running LLM locally on consumer hardware. Written in C++ for maximum performance.
★ 2,167 · C++FAST LAUNCH QWEN3.8 ON AMD
The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference
★ 1,349 · ShellEAGLE-4 Speculative Decoding
Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.
★ 1,287 · PythonANGELSLIM: COMPRESSING MODELS WITHOUT LOSS
A tool for model compression focusing on convenience and efficiency. 1257 stars on GitHub.
★ 1,257 · PythonRUN 284B LLM ON CPU WITHOUT GPU
llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto
★ 1,234 · PythonRUN LARGE LLMS LOCALLY
PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.
★ 1,037 · C++FREE TPU FOR QWEN3.8-27B
The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor
★ 801 · PythonMLX-DSPARK: 4X LLM ACCELERATION
Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.
★ 740 · PythonJANAS: FAST LLM INFERENCE!!
janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU
★ 712 · CBrowser LLM Device Pool
Run large open‑source LLMs in the browser by linking multiple devices into a peer‑to‑peer compute pool, enabling chat and coding agents without any server.
★ 649 · JavaScriptQWEN3.6-27B ULTIMATE VERSION
Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.
★ 495 · PythonLLAMAMPERE: LLM ACCELERATION ON RTX 3090
llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.
★ 380 · C++Not a Llama, no stance
Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.
★ 284 · GoCURATED LIST OF OPD RESOURCES
A collection of papers, frameworks, and tools for on‑policy distillation of large language models. Ideal for LLM researchers and engineers.
★ 277Auto-Tune LLM Serving Speed
Automatically optimizes local LLM serving speed via hill-climbing across decode, speculative decoding, prefill, KV cache, and startup metrics on any hardware.
★ 249 · PythonInference Engine for Cybersecurity
High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.
★ 240 · RustPURE-RUST ENGINE FOR QWEN GGUF
RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.
★ 231 · RustSpeculative Decoding Training Framework
ASpec accelerates LLM inference by training efficient draft models for speculative decoding with high-throughput distributed training.
★ 53 · PythonMEMRA - AI Accelerator for RTX
Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.
★ 6 · RustSparkDuet: on-demand model library
The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and
★ 0 · PythonNew projects on this topic in Telegram
Telegram →