LLM Inference: open-source projects on GitHub
76 repositories on LLM Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
PICOLLM - OFFLINE LLM ON DEVICE
picollm enables running LLM on-device through extreme k‑bit quantization, delivering fast and private inference without the cloud.
★ 320 · PythonCOMPRESS PROMPTS BY 60%
leanctx — drop-in prompt compression for LLM applications. Reduces tokens by 40-60% without changing code.
★ 308 · PythonJAX-LLM-EXAMPLES - QUICK START
Minimal code, maximum speed: LLM examples in pure JAX for rapid prototyping.
★ 287 · PythonHYPERLOOM AUTO-OPTIMIZATION LLM ON AMD
Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.
★ 271 · PythonInference Engine for Cybersecurity
High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.
★ 240 · RustPIE — MODEL SERVING ON RUST
Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.
★ 240 · RustTRANSFORMER-TRICKS - ACCELERATING TRANSFORMERS
A collection of techniques and utilities for accelerating transformers: attention optimization, quantization, and caching of computations.
★ 236 · TeXKALAVAI-CLIENT: CLOUD GPU POWER
The project aggregates idle GPU capacity and offers it for distributed computing.
★ 225 · TypeScriptLLAMA2 IN ONE ZIG FILE
The llama2.zig project lets you run Llama 2 inference in a single file using pure Zig, with no dependencies.
★ 225 · ZigTAOS Self-Learning AI OS
Self-hosted auto-clustering AI-agent OS for your hardware: Raspberry Pi, Orange, Mac Mini, etc.
★ 213 · PythonGLYD: LOSSLESS AI COMPRESSION
Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma
★ 200 · RustDeepSeek Local Setup Guide
A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi
★ 192 · ShellKVarN: KV-Cache Quantization
KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act
★ 159 · PythonLLM Prefix Cache Simulator
Simulates prefix caching mechanisms for LLM inference to analyze cache hit rates and memory efficiency.
★ 76 · C++MEMRA - AI Accelerator for RTX
Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.
★ 6 · RustSparkDuet: on-demand model library
The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and
★ 0 · PythonNew projects on this topic in Telegram
Telegram →