R_REDDYX.XYZ

Multimodal: open-source projects on GitHub

143 repositories on Multimodal, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094VLM 50Agent 547RAG 533Vision Language Model 28MCP 1602AI Agents 1319Deep Learning 105Large Language Models 154Machine Learning 190Pytorch 98Reinforcement Learning 51Vision 17Computer Vision 41
OpenMOSS/MOVA

MOVA Scalable Video-Audio

The MOVA project generates synchronized video and audio, scaling to large datasets in Python.

★ 1,144 · Python
tong-io/tongflow

TONGFLOW: VISUAL AI STUDIO

Open platform for building multimodal AI workflows. Assemble pipelines from models without code.

★ 1,047 · TypeScript
yaotingwangofficial/Awesome-MCoT

ОБЗОР MULTIMODAL CHAIN-OF-THOUGHT

Обзор мультиmodal Chain-of-Thought: анализ методов, данных и benchmark для LLM и VLM. Включает таксономию, оценку и рекомендации.

★ 1,029 · TeX
lancedb/vectordb-recipes

VECTORDB-RECIPES: RAG AND AGENTS BY EXAMPLE

Collection of notebooks with RAG, multimodal AI and agents examples. Covers embeddings, fine-tuning and vector search. 986 stars on GitHub.

★ 989 · Jupyter Notebook
EvolvingLMMs-Lab/NEO

NEO: Vision-Language Models

A series of multimodal Vision-Language models built from scratch. 839 stars on GitHub.

★ 926 · Python
Valen-Team/Valen

MULTIMODAL DECISION MODEL

Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.

★ 918 · Python
OpenMOSS/AnyGPT

ANYGPT — UNIFIED MULTIMODAL LANGUAGE MODEL

AnyGPT is a universal multimodal model built on discrete sequence modeling. It generates images, music, and speech from text and understands translation between modalitie

★ 889 · Python
whitecircle/halo

HALO: FRAMEWORK FOR LLM TRAINING

Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an

★ 862 · Python
OpenMOSS/MOSS-VL

MOSS-VL: 11B MODEL FOR VIDEO

The open-weight MOSS-VL series of 11-billion-parameter models is designed for understanding long-form and streaming video in real time. It supports multimodal input and e

★ 833 · Python
EvolvingLMMs-Lab/lmms-engine

LMMS ENGINE FOR MULTIMODAL MODELS

lmms‑engine — a lightweight and flexible framework for training universal multimodal models in Python.

★ 783 · Python
magicrew/doc7

DOC7: Markdown Conversion

Convert any documents into clean Markdown, preserving structure and visual elements, ready for AI models.

★ 718 · Go
OpenMOSS/MOSS-Audio

MOSS-AUDIO: MODEL FOR AUDIO ANALYSIS

MOSS-Audio — an open-source Python model for generating captions, answering questions about sound, and reasoning about speech, environment and music. With 655 stars on Gi

★ 709 · Python
enoche/MMRec

MMRec: 10+ Models in One

A powerful tool for multimodal recommendations with support for over 10 models. Ideal for researchers and developers of recommendation systems.

★ 706 · Python
TIGER-AI-Lab/VLM2Vec

VLM2Vec: Vector Models for Multimodality

The library implements the VLM2Vec method for training Vision‑Language models that produce effective multimodal embeddings. Presented at ICLR 2025.

★ 704 · Python
redai-studio/Relax

RELAX: ASYNCHRONOUS RL FOR MULTIMODAL POST‑TRAINING

Relax — an asynchronous reinforcement learning engine for scalable post‑training agents. It supports efficient distributed training, multi-agent scenarios, and working wi

★ 676 · Python
reasoning-survey/Awesome-Reasoning-Foundation-Models

RELEVANT PAPERS ON REASONING LLM

Collection of the latest papers and benchmarks on reasoning with foundation models. Enables quick discovery of cutting-edge research and comparison of their performance.

★ 663
jordanrendric/claude-video-vision

Claude Video Vision

A plugin for Claude Code that enables video analysis: extracting key frames and performing multimodal audio analysis.

★ 639 · TypeScript
motis-project/motis

MOTIS - Multimodal Navigation

Motis provides multimodal routing, geocoding, and vector map tiles in C++.

★ 627 · C++
RobotecAI/rai

RAI: Agent for Robots

RAI is a framework for physical AI robots on ROS 2 and Python, supporting voice interaction, scenarios, and logs.

★ 616 · Python
MMMU-Benchmark/MMMU

MMMU: Multimodal Benchmark

Code evaluation for MMMU benchmark measuring multidisciplinary multimodal understanding and reasoning for expert AGI.

★ 601 · Python
oxbshw/watch-skill

watch-skill - Agent Video Search

Convert agent videos, streams, and screen recordings into a timestamped search index, then verify and fix work via THE LOOP.

★ 582 · Python
microsoft/psi

PSI: AI Platform for Real World

psi is a platform for situated intelligence that unites AI agents with the real world through C# and cross‑platform tools.

★ 577 · C#
monatis/clip.cpp

CLIP.CPP - POWERFUL DEPENDENCY-FREE INFERENCE

CLIP inference in pure C/C++ without additional dependencies. Fast work with images and text in a single file.

★ 573 · C++
Code-Amadeus/Amadeus

AMADEUS: AI AGENT FOR DESKTOP

Amadeus is a real‑time multimedia desktop agent built on Python, using CUDA, Electron and GPT‑SoVITS. The project is evolving toward a persistent AI OS interface and alre

★ 554 · Python
henrydaum/second-brain

SECOND BRAIN - YOUR DIGITAL OS

An agent framework that automates tasks using LLM and file intelligence — 552 stars, Python. Excellent DevTools for cross‑platform automation!

★ 552 · Python
SAOS-3624/dsh-video-storyboard

VIDEO STORYBOARD IN ONE IMAGE

dsh-video-storyboard creates a scene-aware storyboard for DeepSeek Harness: 9-16 cells with timestamps, transcript, and audio envelope in one image (384 tokens). Two tool

★ 542 · JavaScript
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
uclanlp/OpenVLHarness

OPENVLHARNESS: VISUAL AGENTS WITH MEMORY

OpenVLHarness is a framework for evaluating and building visual agents with multimodal memory. It supports real-world tool-use scenarios and complex reasoning chains.

★ 499 · Python
pykale/pykale

PYKALE — Powerful ML Tool

A PyTorch-based library for interdisciplinary research using knowledge from multiple sources.

★ 494 · Python
NetManAIOps/ChatTS

CHATTS LLM FOR TIME SERIES

ChatTS — the first LLM trained to understand and reason about time series, presented at VLDB'25.

★ 490 · Python
TeleAI-UAGI/telemem

TELEMEM - MEM0 KILLER IS HERE

High-performance replacement for Mem0 with semantic deduplication, long-term dialog memory, and multimodal video analysis.

★ 464 · Python
afiaka87/clip-guided-diffusion

TEXT TO IMAGE VIA CLIP DIFFUSION

CLI tool and Python module for generating images from text via guided diffusion and CLIP from OpenAI. Create visuals from command line or embed in projects. 459 stars on

★ 459 · Python
david-xinyuwei/david-share

AI AGENTS ON AZURE AND GPU

Repository with code, benchmarks, and guides for AI agents and Deep Learning. Deployment on Microsoft Foundry and Azure, fine-tuning, inference optimization, and GPU test

★ 451 · Jupyter Notebook
laminlabs/lamindb

LAMINDB: AI DATA MANAGEMENT

Open platform for managing multimodal AI data: lineage-native data lake, support for biological formats from the creators of Scanpy.

★ 437 · Python
v-modal/vmodal_sdk_robotics

VMODAL SDK FOR ROBOTICS

vmodal_sdk_robotics — Python‑SDK for multimodal vision in robotics. It combines camera, LiDAR and IMU data into a single stream. Supports training models on synthetic and

★ 434 · Python
yuanze-lin/Olympus

OLYMPUS UNIFIES CV TASKS

The official Olympus code — a universal task router for CV, presented at CVPR 2025. Routes data to the best models.

★ 427 · Python
gyunggyung/AGI-Papers

AI AGENTS BREAKTHROUGHS ARCHIVE

Key papers on agents, architecture, training, RAG, and on-device AI. Up-to-date research in one repository.

★ 426
xlang-ai/OSWorld-V2

OSWorld-V2 - AGENT REAL TASK TEST

OSWorld 2.0 evaluates AI agents' ability to complete long real computer tasks using only Python.

★ 421 · Python
HorizonWind2004/reconstruction-alignment

UNLOCK MULTIMODAL POTENTIAL

Official ICLR 2026 repository: Reconstruction Alignment enhances universal multimedia models via self-supervised learning.

★ 419 · Python
jabberjabberjabber/ImageIndexer

ImageIndexer: Auto-tagging photos via LLM

It indexes images, sends requests to a local LLM, and automatically adds tags to file metadata.

★ 400 · Python
ALEEEHU/World-Simulator

WORLD SIMULATION IN AI

Large-scale survey of multimodal generative models for real-world simulation. 381 stars, IEEE TPAMI 2026.

★ 397
antflydb/antfly

ANTFLY - POWERFUL TOOL ON ZIG

antfly is a Zig project with 376 stars on GitHub. A fast and efficient tool for developers.

★ 376 · Zig
Jinfeng-Xu/Awesome-Multimodal-Recommender-Systems

MULTIMODAL RECOMMENDATIONS: ARTICLE COLLECTION

A collection of the latest articles on multimodal recommendations, constantly updated. For AI researchers and practitioners.

★ 370
healthylaife/MIMIC-IV-Data-Pipeline

MIMIC-IV: Multimodal Data Extraction

Customizable pipeline for extracting multimodal data from MIMIC-IV in Jupyter Notebook.

★ 366 · Jupyter Notebook
clawdotnet/openclaw.net

OPENCLAW.NET: Self-hosted .NET Gateway

Self-hosted gateway and agent runtime on .NET with NativeAOT support for fast deployment of OpenClaw.

★ 360 · C#
friedrichor/Awesome-Multimodal-Papers

AWESOME MULTIMODAL PAPERS

A collection of the best works on multimodal learning: articles, code, and datasets in one repository.

★ 348
wanyichen06/LLMInternSkill

LLMINTERNSKILL – Turnkey Resume

A tool for polishing resumes, matching job openings, verifying evidence, and preparing for interviews using LLM.

★ 348 · Markdown
verl-project/verl-omni

VERL-OMNI: REINFORCEMENT LEARNING

Framework for RL training of diffusion and multimodal models. 347 stars on GitHub.

★ 347 · Python
OpenEnvision/Awesome-Multimodal-Modeling

AWESOME MULTIMODAL MODELING

A collection of resources on multimodal modeling: MLLM, UMM, NMM, 336 ★.

★ 336
Jinfeng-Xu/Multimodal-Recommendation-Library

MULTIMODAL LIBRARY FOR RECOMMENDATIONS

Library provides a continuously updated set of models for multimodal recommendations based on deep learning. Supports working with images, text, and audio, enabling integ

★ 328 · Python
autogluon/autogluon-assistant

AUTOGLUON-ASSISTANT - MULTIAGENT FOR MULTIMODAL ML

An LLM-based multi-agent system automates the entire multimodal machine learning pipeline from data to deployment.

★ 313 · Python
Hebbian-Robotics/hflow

hflow: DATA QUALITY VERIFICATION FOR AI

hflow — Python SDK for robotics teams. Verifies data quality before AI model training. Integration with Airflow and DuckDB.

★ 310 · Python
rollroyces/py-idp

PY-IDP: AI FOR DOCUMENT PROCESSING

py-idp — AI-powered IDP platform for Python: 12+ LLM backends, Pydantic schemas, OCR Nanonets‑OCR2‑3B, auto schema and HITL. Includes a FastAPI server, AGPL‑3.0 + commerc

★ 298 · Python
mbzuai-oryx/VideoGPT-plus

VIDEO GPT-PLUS: ADVANCED VIDEO UNDERSTANDING

Official repository of the VideoGPT+ paper: combines image and video encoders for improved video understanding. Enables extracting semantic features, generating descripti

★ 294 · Python
znxzsy/VeriInk

VERIFIABLE HANDWRITING SYNTHESIS

VeriInk creates verifiable handwriting synthesis for model improvement via feedback and RSI research. Includes seeded examples, artifact checks, and auditable data handof

★ 290 · Python
langgptai/Awesome-Multimodal-Prompts

PROMPTS FOR GPT-4V AND DALL-E3

Awesome-Multimodal-Prompts — selected collection of prompts for GPT-4V and DALL-E3 that unlock full multimodal capabilities. Use them to improve image understanding with

★ 288
LynnReal-AI/LynnReal-Omni

LYNNREAL-OMNI: VIDEO IN 4 STEPS

LynnReal-Omni unifies text‑to‑video, image‑to‑video, human and hand‑pose control, structural control, omni‑reference generation, style transfer, video editing, degraded‑v

★ 281 · Python
ekonwang/GeoVista

GEOVISTA WEB-AUGMENTED AI FOR GEOLOCATION

GeoVista — official implementation of web‑augmented agentic visual reasoning for the geolocalization. In Python it combines the multimodal LLM and the web search, enablin

★ 281 · Python
microsoft/AesCode

AESTHETIC HTML/CSS GENERATION FROM PROMPTS

AesCode turns prompts into editable HTML/CSS infographics. Uses image generators as aesthetic guidance and cross-modal rewards with RL (Qwen3) for verifiable code. 225★ o

★ 281 · Python
ailuntx/Thinking-with-Visual-Primitives

VISUAL PRIMITIVES FOR AI THINKING

Archival snapshot of the Thinking-with-Visual-Primitives project — 254 stars. Study of visual primitives in the context of AI thinking.

★ 280 · Makefile

New projects on this topic in Telegram

Telegram →
Multimodal — page 2 | REDDYX AI