R_REDDYX.XYZ

LLM Inference: open-source projects on GitHub

76 repositories on LLM Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Vllm 69Cuda 48Qwen 106Local LLM 107LLM Serving 24Machine Learning 190Self Hosted 360Speculative Decoding 23Large Language Models 154Llms 136Quantization 34Generative AI 192Apple Silicon 56
Picovoice/picollm

PICOLLM - OFFLINE LLM ON DEVICE

picollm enables running LLM on-device through extreme k‑bit quantization, delivering fast and private inference without the cloud.

★ 320 · Python
jia-gao/leanctx

COMPRESS PROMPTS BY 60%

leanctx — drop-in prompt compression for LLM applications. Reduces tokens by 40-60% without changing code.

★ 308 · Python
jax-ml/jax-llm-examples

JAX-LLM-EXAMPLES - QUICK START

Minimal code, maximum speed: LLM examples in pure JAX for rapid prototyping.

★ 287 · Python
AMD-AGI/Hyperloom

HYPERLOOM AUTO-OPTIMIZATION LLM ON AMD

Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.

★ 271 · Python
Avarok-Cybersecurity/atlas

Inference Engine for Cybersecurity

High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.

★ 240 · Rust
pie-project/pie

PIE — MODEL SERVING ON RUST

Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.

★ 240 · Rust
OpenMachine-ai/transformer-tricks

TRANSFORMER-TRICKS - ACCELERATING TRANSFORMERS

A collection of techniques and utilities for accelerating transformers: attention optimization, quantization, and caching of computations.

★ 236 · TeX
kalavai-net/kalavai-client

KALAVAI-CLIENT: CLOUD GPU POWER

The project aggregates idle GPU capacity and offers it for distributed computing.

★ 225 · TypeScript
cgbur/llama2.zig

LLAMA2 IN ONE ZIG FILE

The llama2.zig project lets you run Llama 2 inference in a single file using pure Zig, with no dependencies.

★ 225 · Zig
jaylfc/taOS

TAOS Self-Learning AI OS

Self-hosted auto-clustering AI-agent OS for your hardware: Raspberry Pi, Orange, Mac Mini, etc.

★ 213 · Python
surya-koritala/Glyd

GLYD: LOSSLESS AI COMPRESSION

Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma

★ 200 · Rust
justxor/deepseek-local-guide-ru

DeepSeek Local Setup Guide

A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi

★ 192 · Shell
huawei-csl/KVarN

KVarN: KV-Cache Quantization

KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act

★ 159 · Python
cacheMon/PrefixSim

LLM Prefix Cache Simulator

Simulates prefix caching mechanisms for LLM inference to analyze cache hit rates and memory efficiency.

★ 76 · C++
avifenesh/memra

MEMRA - AI Accelerator for RTX

Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.

★ 6 · Rust
zorost/sparkduet

SparkDuet: on-demand model library

The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and

★ 0 · Python

New projects on this topic in Telegram

Telegram →
LLM Inference — page 2 | REDDYX AI