AIC★ 712⑂ 72 forksGPL-3.0
prabanta-dev/janas
JANAS: FAST LLM INFERENCE!!
janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU inference in pure C with AVX2.!!!!
// KEY FEATURES
- Runs Qwen3-Next-80B (48 GB) on a 32 GB laptop, givingnow 37 tok/s w/out draft.
- Beats llama.cpp by8‑10× in token generation speed across all 13 Q4_K_M benchmark models
- Reads GGUF models at 62 tok/s, nearly 4× faster than llama.cpp in CPU inference.
- Works on CPU and GPU inference in pure C with AVX2 acceleration under Linux.
#avx2#c#cpu-inference#gguf#inference-engine#linux#llm#local-llm#mixture-of-experts#moe#nvme#quantization
Open on GitHub →New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram