R_REDDYX.XYZ

Speculative Decoding: open-source projects on GitHub

23 repositories on Speculative Decoding, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Local LLM 107Qwen 106LLM Inference 76Cuda 48Qwen3 30Gguf 47Quantization 34Llama Cpp 56Openai API 86Vllm 69Claude Code 1595Inference Engine 18Mixture Of Experts 20
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
youssofal/MTPLX

MTPLX SUPERFAST LAUNCH

MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a

★ 2,706 · Python
incoai/splash

Local AI Engine

Splash lets you run machine learning models directly on Apple‑silicon Macs, using the full power of CPU and GPU without relying on the cloud.

★ 2,284 · Python
Luce-Org/lucebox-hub

Lucebox Hub

High-speed server for running LLM locally on consumer hardware. Written in C++ for maximum performance.

★ 2,167 · C++
peonist-ai/halogen-flash-server

FAST LAUNCH QWEN3.8 ON AMD

The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference

★ 1,349 · Shell
eagle-hq/EAGLE-4

EAGLE-4 Speculative Decoding

Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.

★ 1,287 · Python
Tencent/AngelSlim

ANGELSLIM: COMPRESSING MODELS WITHOUT LOSS

A tool for model compression focusing on convenience and efficiency. 1257 stars on GitHub.

★ 1,257 · Python
synchopate/llama-modchip

RUN 284B LLM ON CPU WITHOUT GPU

llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto

★ 1,234 · Python
simplexai-labs/PolyStrata

RUN LARGE LLMS LOCALLY

PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.

★ 1,037 · C++
ARahim3/kaggle-tpu-lab

FREE TPU FOR QWEN3.8-27B

The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor

★ 801 · Python
ARahim3/mlx-dspark

MLX-DSPARK: 4X LLM ACCELERATION

Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.

★ 740 · Python
prabanta-dev/janas

JANAS: FAST LLM INFERENCE!!

janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU

★ 712 · C
Nehanth/pooled

Browser LLM Device Pool

Run large open‑source LLMs in the browser by linking multiple devices into a peer‑to‑peer compute pool, enabling chat and coding agents without any server.

★ 649 · JavaScript
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

QWEN3.6-27B ULTIMATE VERSION

Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.

★ 495 · Python
JakeATX/llamAmpere

LLAMAMPERE: LLM ACCELERATION ON RTX 3090

llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.

★ 380 · C++
raketenkater/ggrun

Not a Llama, no stance

Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.

★ 284 · Go
chrisliu298/awesome-on-policy-distillation

CURATED LIST OF OPD RESOURCES

A collection of papers, frameworks, and tools for on‑policy distillation of large language models. Ideal for LLM researchers and engineers.

★ 277
sybil-solutions/inference-optimisation-skill

Auto-Tune LLM Serving Speed

Automatically optimizes local LLM serving speed via hill-climbing across decode, speculative decoding, prefill, KV cache, and startup metrics on any hardware.

★ 249 · Python
Avarok-Cybersecurity/atlas

Inference Engine for Cybersecurity

High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.

★ 240 · Rust
RustingStudio/RustingStrata

PURE-RUST ENGINE FOR QWEN GGUF

RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.

★ 231 · Rust
inclusionAI/ASpec

Speculative Decoding Training Framework

ASpec accelerates LLM inference by training efficient draft models for speculative decoding with high-throughput distributed training.

★ 53 · Python
avifenesh/memra

MEMRA - AI Accelerator for RTX

Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.

★ 6 · Rust
zorost/sparkduet

SparkDuet: on-demand model library

The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and

★ 0 · Python

New projects on this topic in Telegram

Telegram →
Speculative Decoding — 23 open-source GitHub projects | REDDYX AI