Multimodal: open-source projects on GitHub
143 repositories on Multimodal, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
AI Productivity On-Device
A universal AI productivity tool that runs locally and guarantees complete privacy without complex setup.
★ 59,922 · JavaScriptDeep Understanding AI Agent
Open repository of the book 'Deep Understanding AI Agent': full text, compiled PDF, and code for each section.
★ 58,104 · PythonMultimodal AI Stack
Open multimodal AI agent stack that unites cutting‑edge models and infrastructure for building autonomous agents.
★ 33,596 · TypeScriptUNILM POWERFUL PRETRAINED MODEL
UniLM is a large-scale self-supervised pretrained model covering tasks, languages, and modalities. It supports BEiT, BEiT-3, BitNet, DeepNet, and Document AI, written in
★ 22,242 · Pythonms-swift: LLM training
A framework for fine-tuning 600+ language and 300+ multimodal models. Supports PEFT, full fine-tuning, CPT/SFT/DPO/GRPO.
★ 14,178 · PythonPixelRAG: Pixel Search
PixelRAG replaces traditional web parsing with direct search in the pixel representation of pages, providing scalable and accurate data extraction.
★ 10,946 · PythonRerun: Multimodal Data
Visualization, querying, and streaming of multimodal robotics data for AI training.
★ 10,824 · RustLocal AI SDK Set
Production-ready toolkit for running AI models on a local device.
★ 10,350 · C++VIDEO CODE: REVERSE ENGINEERING
Reverse engineering for any video. Full-stack agentic video generator: create, edit and render locally via HTML, supporting images, video, music and voice.
★ 9,939 · TypeScriptPYOD: Anomaly Detector for AI
The pyod library offers 60+ anomaly detectors for tables, time series, graphs, text, and images with ADEngine orchestration.
★ 9,867 · PythonMobileAgent Powerful GUI Agent
The MobileAgent project offers a family of powerful GUI agents in Python for automating interface interaction.
★ 9,379 · PythonSEATUNNEL DATA INTEGRATION
A multimodal distributed tool for large-scale data integration with high performance in Java.
★ 9,359 · JavaBENTOML PATH TO AI API
Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.
★ 8,663 · PythonMLX-AUDIO: SPEECH ON APPLE SILICON
mlx-audio — library for TTS, STT and STS on Apple's MLX framework. Runs efficiently on Apple Silicon. 8192 stars on GitHub.
★ 8,267 · PythonSpeech Tools for MLX
A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.
★ 7,115 · PythonDIGUP SEARCH BY FILE CONTENT
DigUp indexes your local files — photos, PDFs, videos, audio and code — and lets you find them by meaning of the query. Search runs on-device using the EmbeddingGemma 2 m
★ 6,418 · SwiftVLM-R1: Smart Image Recognition
The project demonstrates how reinforcement learning improves image understanding by multimedia models.
★ 6,046 · PythonGenkit AI Framework
An open-source framework for building AI applications in JavaScript, Go, and Python. Developed and used at Google.
★ 5,985 · TypeScriptMMF: Modular Multimodal Framework
Modular framework from FAIR for vision and language research, combining the flexibility of Python and the power of multimodal models.
★ 5,633 · PythonUltraRAG: RAG pipeline framework
Low-code MCP framework for creating complex and innovative RAG pipelines.
★ 5,549 · PythonDAFT — Data Engine for AI
High-performance engine for AI and multimodal tasks that processes images, audio, video, and structured data at any scale. Built on Rust ⭐5533
★ 5,533 · RustXTUNER: TRAINER FOR LARGE MoE
A Next-Generation Training Engine Built for Ultra-Large MoE Models. ⭐ 5138 stars on GitHub!
★ 5,138 · PythonEfficient Multimodal Inference
Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.
★ 4,863 · PythonTREE OF THOUGHTS: SMART SOLUTION SEARCH
Tree of Thoughts implementation for LLMs: model builds reasoning tree, evaluates branches, picks best path. Boosts solution quality by 70%+ without fine-tuning.
★ 4,592 · PythonLMMS-EVAL - ONE-STOP MODALITY EVAL
Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.
★ 4,169 · PythonMEMORY FOR AI AGENTS
SimpleMem — efficient long-term memory for LLM agents with support for text and multimodal data.
★ 3,878 · PythonMTEB - Benchmark for Text Embeddings
Large-scale benchmark for evaluating the quality of text embeddings. 800+ tasks across 50+ languages.
★ 3,287 · PythonVORTEX — MOST STRUCTURED DATA FORMAT
Vortex is an extensible framework for columnar data compression and the fastest open columnar format. It is the primary tool for next‑generation analytics from LFAI&Data.
★ 2,966 · RustOSWorld: BENCHMARK FOR AI AGENTS
OSWorld is a benchmark for evaluating multimodal AI agents in real computer environments. Open tasks in real operating systems.
★ 2,911 · PythonAUTODISTILL: TRAINING WITHOUT LABELING
Automatically turns raw images into ready-to-use inference models, using foundation models to generate pseudo‑labels and train supervised models. No manual labeling requi
★ 2,798 · PythonDATACHAIN - DATA WITH AI
Typed versioned datasets for unstructured data on top of S3, GCS, and Azure. A great tool for working with AI information.
★ 2,748 · PythonMAESTRO - Simplifying Model Fine-tuning
The project simplifies fine-tuning of multimodal models PaliGemma 2, Florence‑2, and Qwen2.5‑VL, saving developers time and resources.
★ 2,696 · PythonTanStack AI SDK
🤖 Type-safe AI SDK for TypeScript supporting streaming chat, tool calls, agents, and multimodal applications with OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Soli
★ 2,684 · TypeScriptGenerative AI Roadmap Projects
A collection of generative AI resources: roadmap, projects, interview and coding preparation.
★ 2,670 · Jupyter NotebookOMAGENT: THE MULTIMODAL AGENT
OmAgent — Python framework for multimodal agents on Gemini, GPT-4 and other LLMs. Enables rapid prototyping and deployment via Gradio, supports RAG and rapid chatbot temp
★ 2,670 · PythonAWESOME MODEL QUANTIZATION RESOURCES
A curated list of articles, benchmarks, surveys, and tools for neural network quantization. Covers low-bit networks, LLMs, multimodal and generative models, vector and la
★ 2,469IMAGE GENERATION REVIEW
A large-scale review of text-to-image generation methods: architectures, models, and approaches — all collected in one repository.
★ 2,446claude-real-video - WATCH VIDEO WITH CLAUDE
Allows any LLM to "watch" video: extracts key frames and transcript, works locally.
★ 2,319 · PythonINTERNVIDEO VIDEO FOUNDATION MODELS
The InternVideo project offers open video foundation models and a dataset for multimodal understanding, presented at ECCV2024.
★ 2,281 · PythonPARALLEL CONTENT PROCESSING IN PYTHON
Lightweight library for efficient parallel content processing with GenAI. 2110 stars on GitHub!
★ 2,123 · PythonQWEN-VL SERIES FINETUNE OPEN SOURCE
Open-source implementation for fine-tuning Qwen-VL series models from Alibaba Cloud in Python. Allows easy adaptation of multimodal models for custom tasks.
★ 1,983 · PythonBITNET: 1-BIT TRANSFORMER
BitNet implementation: 1-bit transformers for LLM in PyTorch. Open source code for fast experiments.
★ 1,949 · PythonNVIDIA Data Designer
NeMo Data Designer: generation of high-quality synthetic data from scratch or based on seed data. Ideal for training and fine-tuning models.
★ 1,877 · PythonDETIKZIFY DRAWS LATEX DIAGRAMS FAST
DeTikZify turns textual descriptions or hand‑drawn sketches into ready‑to‑use TikZ code for LaTeX. The project leverages HuggingFace and Llama models for inverse graphics
★ 1,831 · PythonWeMM-Embedding by Tencent
Family of universal multimodal embedding models from the WeChat Vision team, supporting understanding and retrieval across modalities.
★ 1,818 · PythonSGLANG-OMNI: OMNI-SERVICE FOR AI
SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.
★ 1,791 · PythonMAGE: AI AGENT FOR PC CONTROL
Microsoft released Mage — an agent for GUI control via flow matching. The model edits images and performs computer tasks without fine-tuning.
★ 1,782 · PythonParlor AI on device
Parlor is an open project that enables natural voice and visual dialogues with AI running fully locally on your device. It is built on the Gemma 4 E2B and Kokoro models.
★ 1,752 · HTMLPIXELTABLE — DECLARATIVE AI BACKEND
Pixeltable is a declarative, incremental backend for multimodal AI applications, simplifying work with data and models.
★ 1,569 · PythonOVIS: Visual-Text Sync
A modern MLLM architecture for structural alignment of visual and text embeddings.
★ 1,539 · PythonFAST MULTIMODAL LLM ON MOBILE
A C++ library enables running multimodal LLMs directly on smartphones with high speed and low power consumption.
★ 1,533 · C++Visual Intelligence for Home
ha-llmvision brings visual intelligence to your home, enabling real‑time object, face and activity recognition for smarter automation.
★ 1,531 · PythonTHEPIPE: DATA FROM DOCUMENTS FAST
ThePipe uses vision‑language models to extract data from PDF, Word and PowerPoint. It recognises tables, text and images, turning them into ready‑to‑use structures. Suppo
★ 1,530 · PythonXTREME1 — PLATFORM FOR MULTIMODAL ANNOTATION
Xtreme1 — platform for annotating 3D‑LiDAR, images and LLM data. Supports collaborative work, AI suggestions and export to COCO, KITTI, Hugging Face.
★ 1,393 · TypeScriptVISION AND LANGUAGE: TOP MODELS
A collection of architectures of well-known vision‑language models with explanations and links.
★ 1,368 · MarkdownAIRUNNER OFFLINE AI FOR ALL
Offline inference engine for arts, real-time voice dialogues, LLM chatbots, and automated workflows.
★ 1,315 · Pythondsh-vision-router - VISION FOR AGENTS
Free vision chain for DeepSeek Harness agents: Q&A, grounding, OCR, and more. Single-command install, no Python, work with images like a regular tool.
★ 1,274 · JavaScriptVisual Toolkit for LLM
A library for text models that lets them 'see' images: recognize content, answer questions, restore UI, automate GUI, and integrate with agents.
★ 1,262 · PythonMULTIMODAL AI FOR CONTENT
UForm — compact MULTIMODAL AI for understanding and generating content. Processes multilingual texts, images and video, working 5x faster than OpenAI CLIP and LLaVA.
★ 1,253 · PythonPROXIMA: UNITE AI WITH TOOLS
Proxima — an MCP server that connects ChatGPT, Claude, Gemini and Perplexity to your development tools without direct API calls. It supports JavaScript and has already ga
★ 1,210 · JavaScriptNew projects on this topic in Telegram
Telegram →