VLM: open-source projects on GitHub
50 repositories on VLM, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
🤗 Transformers
A framework for defining machine learning models in text, vision, audio, and multimodal models, for inference and training.
★ 160,482 · PythonMultimodal AI Stack
Open multimodal AI agent stack that unites cutting‑edge models and infrastructure for building autonomous agents.
★ 33,596 · TypeScriptSGLang: LLM Framework
SGLang is a high-performance framework for serving large language and multimodal models.
★ 27,854 · PythonPixelRAG: Pixel Search
PixelRAG replaces traditional web parsing with direct search in the pixel representation of pages, providing scalable and accurate data extraction.
★ 10,946 · PythonLocal AI SDK Set
Production-ready toolkit for running AI models on a local device.
★ 10,350 · C++OUMI EASY LLM TUNING
OUMI lets you quickly fine‑tune (SFT/RL), evaluate, and deploy Qwen, Gemma, and any open agentic LLM/VLM without extra complexity.
★ 9,433 · PythonGENIEX: LLMs on Devices
Run cutting-edge multimodal models on GPU, NPU, and CPU — from PC to mobile and IoT devices.
★ 8,530 · RustFRAME-BY-FRAME VIDEO BREAKDOWN
The rea-video project reverse-engineers video frame by frame: identifies what was filmed, AI-generated, or coded. Outputs edit details, captions, audio specs, and a repro
★ 8,505 · PythonNEXA-SDK AI Everywhere Always
Run cutting‑edge LLMs and VLMs on GPU, NPU, and CPU — from PC to mobile and IoT devices. Support for models like GPT‑OSS, Granite‑4, Qwen‑3‑VL, and more.
★ 8,094 · RustVLM-R1: Smart Image Recognition
The project demonstrates how reinforcement learning improves image understanding by multimedia models.
★ 6,046 · PythonUltraRAG: RAG pipeline framework
Low-code MCP framework for creating complex and innovative RAG pipelines.
★ 5,549 · PythonLMMS-EVAL - ONE-STOP MODALITY EVAL
Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.
★ 4,169 · PythonOpen-Source Document Parser
Open-source document parsing model achieving state-of-the-art accuracy and speed through recursive self-improvement post-training.
★ 3,830 · PythonSKYWORK-R1V VISION LANGUAGE AI
The Skywork-R1V model series combines vision and language for complex reasoning. Available in Python and already has over 3k stars.
★ 3,173 · PythonOSWorld: BENCHMARK FOR AI AGENTS
OSWorld is a benchmark for evaluating multimodal AI agents in real computer environments. Open tasks in real operating systems.
★ 2,911 · PythonEVALSCOPE: SIMPLE LLM TESTING
Flexible Python framework for evaluating the performance of large models (LLM, VLM, AIGC). ⭐2874
★ 2,874 · PythonOMAGENT: THE MULTIMODAL AGENT
OmAgent — Python framework for multimodal agents on Gemini, GPT-4 and other LLMs. Enables rapid prototyping and deployment via Gradio, supports RAG and rapid chatbot temp
★ 2,670 · PythonLLM Agent Framework in ComfyUI
Framework unites LLM agents, Omost, GPT‑Sovits, ChatTTS, and OCR in ComfyUI with support for Feishu, Discord, and all popular LLMs.
★ 2,390 · PythonQWEN-VL SERIES FINETUNE OPEN SOURCE
Open-source implementation for fine-tuning Qwen-VL series models from Alibaba Cloud in Python. Allows easy adaptation of multimodal models for custom tasks.
★ 1,983 · PythonCOLLECTION OF JAILBREAK LLM METHODS
Awesome-Jailbreak-on-LLMs is a collection of cutting‑edge jailbreak methods for large language models. The repository contains papers, code, datasets, evaluations, and ef
★ 1,977OpenGUI: Control Android via GUI
OpenGUI is a framework for creating AI agents that can see, plan, and control real Android applications via their interface.
★ 1,811 · KotlinPaperBanana: Scientific Visualization Generation
An open-source implementation and extension of Google Research PaperBanana for automating the creation of academic figures, diagrams, and scientific visualizations.
★ 1,648 · PythonGPU-Accelerated AI Video Analytics
A set of reference architectures for building GPU-accelerated vision agents and AI-based video analytics applications. ⭐ 1,486 stars, Python.
★ 1,486 · PythonVISION AND LANGUAGE: TOP MODELS
A collection of architectures of well-known vision‑language models with explanations and links.
★ 1,368 · MarkdownANGELSLIM: COMPRESSING MODELS WITHOUT LOSS
A tool for model compression focusing on convenience and efficiency. 1257 stars on GitHub.
★ 1,257 · PythonNEO: Vision-Language Models
A series of multimodal Vision-Language models built from scratch. 839 stars on GitHub.
★ 926 · PythonMULTIMODAL DECISION MODEL
Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.
★ 918 · PythonMOBILEGYM: ANDROID AGENT TRAINING
MobileGym is a browser‑hosted Android emulator for research on mobile GUI agents. The platform provides verifiable evaluation and highly parallel online reinforcement‑lea
★ 866 · PythonMOSS-VL: 11B MODEL FOR VIDEO
The open-weight MOSS-VL series of 11-billion-parameter models is designed for understanding long-form and streaming video in real time. It supports multimodal input and e
★ 833 · PythonDINGO: AI MODEL EVALUATION
Dingo is a tool for comprehensive evaluation of data, model, and AI application quality in Python. It enables measuring reliability and performance of systems.
★ 768 · PythonSPARKVSR: INTERACTIVE VIDEO SUPER-RESOLUTION
A new video super-resolution method via sparse keyframe propagation. ECCV 2026 paper, 675 stars on GitHub.
★ 750 · PythonVLM2Vec: Vector Models for Multimodality
The library implements the VLM2Vec method for training Vision‑Language models that produce effective multimodal embeddings. Presented at ICLR 2025.
★ 704 · PythonOPTIMUM-INTEL - ACCELERATE INFERENCE WITH INTEL
Optimum Intel provides Intel tools to accelerate AI model inference, simplifying deployment and boosting performance.
★ 630 · Jupyter NotebookVIDEO RECAP WITH CLAUDE CODE
The project provides Claude Code skills for creating Chinese narration recaps from video files. It exports the result as an editable JianYing/CapCut draft. It runs locall
★ 626 · PythonRAI: Agent for Robots
RAI is a framework for physical AI robots on ROS 2 and Python, supporting voice interaction, scenarios, and logs.
★ 616 · PythonFABRIC CREATIVE 3D CODING
Fabric is a Node‑based environment for creative coding, 3D graphics and image processing, inspired by Quartz Composer. In Swift it uses Metal for fast rendering and suppo
★ 606 · SwiftComfyUI_VLM_nodes — Powerful Multimodal AI
A set of custom nodes for ComfyUI: from working with VLM and LLM to music generation and creative prompts.
★ 593 · PythonAUTOMODEL — DISTRIBUTED LLM/VLM TRAINING
Native PyTorch library for distributed LLM/VLM training with out-of-the-box Hugging Face support. ⭐ 576 stars on GitHub.
★ 576 · PythonREMARKABLE2 BECOMES VISION-LLM INTERFACE
Project ghostwriter turns reMarkable2 into handwritten interface for vision-LLMs: ChatGPT, Claude, Gemini. Write by hand on E Ink — model sees, understands, replies. Writ
★ 573 · RustVLA MODELS FOR CARS
A collection of Vision-Language-Action models for autonomous driving: an overview of past, present, and future directions.
★ 527 · HTMLYZMA: Local LLM in Go
Integration of llama.cpp into Go applications for local inference with hardware acceleration. 496 stars on GitHub.
★ 496 · GoOSWorld-V2 - AGENT REAL TASK TEST
OSWorld 2.0 evaluates AI agents' ability to complete long real computer tasks using only Python.
★ 421 · PythonAWESOME MULTIMODAL MODELING
A collection of resources on multimodal modeling: MLLM, UMM, NMM, 336 ★.
★ 336VLMRUN-COOKBOOK - VLM NOTEBOOKS
A collection of Jupyter notebooks showing how to quickly get started with the VLM Run API: loading models, inference, and multimodal output.
★ 313 · Jupyter Notebookphi - Simple Phi-4 Launch
Run the powerful multimodal Phi-4 model directly on your Mac with Apple Silicon — no cloud, no complex setup.
★ 280 · Jupyter NotebookLongVT — Think About Video Foundations
A model that learns to 'think' about long videos through built‑in tools for CVPR 2026.
★ 274 · PythonMULTIMODAL DECISION MODEL
OneJev — multimodal System One model for calibrated answers to questions about screens, photos, video and text in one forward pass.
★ 272 · PythonLLAMA_ROS ROS 2 LLM
The project combines llama.cpp and llava.cpp with ROS 2, allowing the use of GGUF models for real-time robotics.
★ 268 · C++ComfyUI Fal API: 1400+ Models
Production-ready custom nodes for ComfyUI, providing access to 1400+ fal.ai models with automatic updates, caching, and cost control.
★ 227 · PythonAuto-labeling YOLO Training
Platform for automatic image and video labeling using VLM NVIDIA LocateAnything-3B, with manual correction and one-click YOLO model training.
★ 31 · PythonNew projects on this topic in Telegram
Telegram →