Multimodal: open-source projects on GitHub
143 repositories on Multimodal, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
MOVA Scalable Video-Audio
The MOVA project generates synchronized video and audio, scaling to large datasets in Python.
★ 1,144 · PythonTONGFLOW: VISUAL AI STUDIO
Open platform for building multimodal AI workflows. Assemble pipelines from models without code.
★ 1,047 · TypeScriptОБЗОР MULTIMODAL CHAIN-OF-THOUGHT
Обзор мультиmodal Chain-of-Thought: анализ методов, данных и benchmark для LLM и VLM. Включает таксономию, оценку и рекомендации.
★ 1,029 · TeXVECTORDB-RECIPES: RAG AND AGENTS BY EXAMPLE
Collection of notebooks with RAG, multimodal AI and agents examples. Covers embeddings, fine-tuning and vector search. 986 stars on GitHub.
★ 989 · Jupyter NotebookNEO: Vision-Language Models
A series of multimodal Vision-Language models built from scratch. 839 stars on GitHub.
★ 926 · PythonMULTIMODAL DECISION MODEL
Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.
★ 918 · PythonANYGPT — UNIFIED MULTIMODAL LANGUAGE MODEL
AnyGPT is a universal multimodal model built on discrete sequence modeling. It generates images, music, and speech from text and understands translation between modalitie
★ 889 · PythonHALO: FRAMEWORK FOR LLM TRAINING
Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an
★ 862 · PythonMOSS-VL: 11B MODEL FOR VIDEO
The open-weight MOSS-VL series of 11-billion-parameter models is designed for understanding long-form and streaming video in real time. It supports multimodal input and e
★ 833 · PythonLMMS ENGINE FOR MULTIMODAL MODELS
lmms‑engine — a lightweight and flexible framework for training universal multimodal models in Python.
★ 783 · PythonDOC7: Markdown Conversion
Convert any documents into clean Markdown, preserving structure and visual elements, ready for AI models.
★ 718 · GoMOSS-AUDIO: MODEL FOR AUDIO ANALYSIS
MOSS-Audio — an open-source Python model for generating captions, answering questions about sound, and reasoning about speech, environment and music. With 655 stars on Gi
★ 709 · PythonMMRec: 10+ Models in One
A powerful tool for multimodal recommendations with support for over 10 models. Ideal for researchers and developers of recommendation systems.
★ 706 · PythonVLM2Vec: Vector Models for Multimodality
The library implements the VLM2Vec method for training Vision‑Language models that produce effective multimodal embeddings. Presented at ICLR 2025.
★ 704 · PythonRELAX: ASYNCHRONOUS RL FOR MULTIMODAL POST‑TRAINING
Relax — an asynchronous reinforcement learning engine for scalable post‑training agents. It supports efficient distributed training, multi-agent scenarios, and working wi
★ 676 · PythonRELEVANT PAPERS ON REASONING LLM
Collection of the latest papers and benchmarks on reasoning with foundation models. Enables quick discovery of cutting-edge research and comparison of their performance.
★ 663Claude Video Vision
A plugin for Claude Code that enables video analysis: extracting key frames and performing multimodal audio analysis.
★ 639 · TypeScriptMOTIS - Multimodal Navigation
Motis provides multimodal routing, geocoding, and vector map tiles in C++.
★ 627 · C++RAI: Agent for Robots
RAI is a framework for physical AI robots on ROS 2 and Python, supporting voice interaction, scenarios, and logs.
★ 616 · PythonMMMU: Multimodal Benchmark
Code evaluation for MMMU benchmark measuring multidisciplinary multimodal understanding and reasoning for expert AGI.
★ 601 · Pythonwatch-skill - Agent Video Search
Convert agent videos, streams, and screen recordings into a timestamped search index, then verify and fix work via THE LOOP.
★ 582 · PythonPSI: AI Platform for Real World
psi is a platform for situated intelligence that unites AI agents with the real world through C# and cross‑platform tools.
★ 577 · C#CLIP.CPP - POWERFUL DEPENDENCY-FREE INFERENCE
CLIP inference in pure C/C++ without additional dependencies. Fast work with images and text in a single file.
★ 573 · C++AMADEUS: AI AGENT FOR DESKTOP
Amadeus is a real‑time multimedia desktop agent built on Python, using CUDA, Electron and GPT‑SoVITS. The project is evolving toward a persistent AI OS interface and alre
★ 554 · PythonSECOND BRAIN - YOUR DIGITAL OS
An agent framework that automates tasks using LLM and file intelligence — 552 stars, Python. Excellent DevTools for cross‑platform automation!
★ 552 · PythonVIDEO STORYBOARD IN ONE IMAGE
dsh-video-storyboard creates a scene-aware storyboard for DeepSeek Harness: 9-16 cells with timestamps, transcript, and audio envelope in one image (384 tokens). Two tool
★ 542 · JavaScriptFLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · PythonOPENVLHARNESS: VISUAL AGENTS WITH MEMORY
OpenVLHarness is a framework for evaluating and building visual agents with multimodal memory. It supports real-world tool-use scenarios and complex reasoning chains.
★ 499 · PythonPYKALE — Powerful ML Tool
A PyTorch-based library for interdisciplinary research using knowledge from multiple sources.
★ 494 · PythonCHATTS LLM FOR TIME SERIES
ChatTS — the first LLM trained to understand and reason about time series, presented at VLDB'25.
★ 490 · PythonTELEMEM - MEM0 KILLER IS HERE
High-performance replacement for Mem0 with semantic deduplication, long-term dialog memory, and multimodal video analysis.
★ 464 · PythonTEXT TO IMAGE VIA CLIP DIFFUSION
CLI tool and Python module for generating images from text via guided diffusion and CLIP from OpenAI. Create visuals from command line or embed in projects. 459 stars on
★ 459 · PythonAI AGENTS ON AZURE AND GPU
Repository with code, benchmarks, and guides for AI agents and Deep Learning. Deployment on Microsoft Foundry and Azure, fine-tuning, inference optimization, and GPU test
★ 451 · Jupyter NotebookLAMINDB: AI DATA MANAGEMENT
Open platform for managing multimodal AI data: lineage-native data lake, support for biological formats from the creators of Scanpy.
★ 437 · PythonVMODAL SDK FOR ROBOTICS
vmodal_sdk_robotics — Python‑SDK for multimodal vision in robotics. It combines camera, LiDAR and IMU data into a single stream. Supports training models on synthetic and
★ 434 · PythonOLYMPUS UNIFIES CV TASKS
The official Olympus code — a universal task router for CV, presented at CVPR 2025. Routes data to the best models.
★ 427 · PythonAI AGENTS BREAKTHROUGHS ARCHIVE
Key papers on agents, architecture, training, RAG, and on-device AI. Up-to-date research in one repository.
★ 426OSWorld-V2 - AGENT REAL TASK TEST
OSWorld 2.0 evaluates AI agents' ability to complete long real computer tasks using only Python.
★ 421 · PythonUNLOCK MULTIMODAL POTENTIAL
Official ICLR 2026 repository: Reconstruction Alignment enhances universal multimedia models via self-supervised learning.
★ 419 · PythonImageIndexer: Auto-tagging photos via LLM
It indexes images, sends requests to a local LLM, and automatically adds tags to file metadata.
★ 400 · PythonWORLD SIMULATION IN AI
Large-scale survey of multimodal generative models for real-world simulation. 381 stars, IEEE TPAMI 2026.
★ 397ANTFLY - POWERFUL TOOL ON ZIG
antfly is a Zig project with 376 stars on GitHub. A fast and efficient tool for developers.
★ 376 · ZigMULTIMODAL RECOMMENDATIONS: ARTICLE COLLECTION
A collection of the latest articles on multimodal recommendations, constantly updated. For AI researchers and practitioners.
★ 370MIMIC-IV: Multimodal Data Extraction
Customizable pipeline for extracting multimodal data from MIMIC-IV in Jupyter Notebook.
★ 366 · Jupyter NotebookOPENCLAW.NET: Self-hosted .NET Gateway
Self-hosted gateway and agent runtime on .NET with NativeAOT support for fast deployment of OpenClaw.
★ 360 · C#AWESOME MULTIMODAL PAPERS
A collection of the best works on multimodal learning: articles, code, and datasets in one repository.
★ 348LLMINTERNSKILL – Turnkey Resume
A tool for polishing resumes, matching job openings, verifying evidence, and preparing for interviews using LLM.
★ 348 · MarkdownVERL-OMNI: REINFORCEMENT LEARNING
Framework for RL training of diffusion and multimodal models. 347 stars on GitHub.
★ 347 · PythonAWESOME MULTIMODAL MODELING
A collection of resources on multimodal modeling: MLLM, UMM, NMM, 336 ★.
★ 336MULTIMODAL LIBRARY FOR RECOMMENDATIONS
Library provides a continuously updated set of models for multimodal recommendations based on deep learning. Supports working with images, text, and audio, enabling integ
★ 328 · PythonAUTOGLUON-ASSISTANT - MULTIAGENT FOR MULTIMODAL ML
An LLM-based multi-agent system automates the entire multimodal machine learning pipeline from data to deployment.
★ 313 · Pythonhflow: DATA QUALITY VERIFICATION FOR AI
hflow — Python SDK for robotics teams. Verifies data quality before AI model training. Integration with Airflow and DuckDB.
★ 310 · PythonPY-IDP: AI FOR DOCUMENT PROCESSING
py-idp — AI-powered IDP platform for Python: 12+ LLM backends, Pydantic schemas, OCR Nanonets‑OCR2‑3B, auto schema and HITL. Includes a FastAPI server, AGPL‑3.0 + commerc
★ 298 · PythonVIDEO GPT-PLUS: ADVANCED VIDEO UNDERSTANDING
Official repository of the VideoGPT+ paper: combines image and video encoders for improved video understanding. Enables extracting semantic features, generating descripti
★ 294 · PythonVERIFIABLE HANDWRITING SYNTHESIS
VeriInk creates verifiable handwriting synthesis for model improvement via feedback and RSI research. Includes seeded examples, artifact checks, and auditable data handof
★ 290 · PythonPROMPTS FOR GPT-4V AND DALL-E3
Awesome-Multimodal-Prompts — selected collection of prompts for GPT-4V and DALL-E3 that unlock full multimodal capabilities. Use them to improve image understanding with
★ 288LYNNREAL-OMNI: VIDEO IN 4 STEPS
LynnReal-Omni unifies text‑to‑video, image‑to‑video, human and hand‑pose control, structural control, omni‑reference generation, style transfer, video editing, degraded‑v
★ 281 · PythonGEOVISTA WEB-AUGMENTED AI FOR GEOLOCATION
GeoVista — official implementation of web‑augmented agentic visual reasoning for the geolocalization. In Python it combines the multimodal LLM and the web search, enablin
★ 281 · PythonAESTHETIC HTML/CSS GENERATION FROM PROMPTS
AesCode turns prompts into editable HTML/CSS infographics. Uses image generators as aesthetic guidance and cross-modal rewards with RL (Qwen3) for verifiable code. 225★ o
★ 281 · PythonVISUAL PRIMITIVES FOR AI THINKING
Archival snapshot of the Thinking-with-Visual-Primitives project — 254 stars. Study of visual primitives in the context of AI thinking.
★ 280 · MakefileNew projects on this topic in Telegram
Telegram →