R_REDDYX.XYZ

Multimodal: open-source projects on GitHub

143 repositories on Multimodal, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094VLM 50Agent 547RAG 533Vision Language Model 28MCP 1601AI Agents 1319Deep Learning 105Large Language Models 154Machine Learning 190Pytorch 98Reinforcement Learning 51Vision 17Computer Vision 41
Mintplex-Labs/anything-llm

AI Productivity On-Device

A universal AI productivity tool that runs locally and guarantees complete privacy without complex setup.

★ 59,922 · JavaScript
bojieli/ai-agent-book

Deep Understanding AI Agent

Open repository of the book 'Deep Understanding AI Agent': full text, compiled PDF, and code for each section.

★ 58,104 · Python
bytedance/UI-TARS-desktop

Multimodal AI Stack

Open multimodal AI agent stack that unites cutting‑edge models and infrastructure for building autonomous agents.

★ 33,596 · TypeScript
microsoft/unilm

UNILM POWERFUL PRETRAINED MODEL

UniLM is a large-scale self-supervised pretrained model covering tasks, languages, and modalities. It supports BEiT, BEiT-3, BitNet, DeepNet, and Document AI, written in

★ 22,242 · Python
modelscope/ms-swift

ms-swift: LLM training

A framework for fine-tuning 600+ language and 300+ multimodal models. Supports PEFT, full fine-tuning, CPT/SFT/DPO/GRPO.

★ 14,178 · Python
StarTrail-org/PixelRAG

PixelRAG: Pixel Search

PixelRAG replaces traditional web parsing with direct search in the pixel representation of pages, providing scalable and accurate data extraction.

★ 10,946 · Python
rerun-io/rerun

Rerun: Multimodal Data

Visualization, querying, and streaming of multimodal robotics data for AI training.

★ 10,824 · Rust
RunanywhereAI/runanywhere-sdks

Local AI SDK Set

Production-ready toolkit for running AI models on a local device.

★ 10,350 · C++
nexu-io/openmotion

VIDEO CODE: REVERSE ENGINEERING

Reverse engineering for any video. Full-stack agentic video generator: create, edit and render locally via HTML, supporting images, video, music and voice.

★ 9,939 · TypeScript
yzhao062/pyod

PYOD: Anomaly Detector for AI

The pyod library offers 60+ anomaly detectors for tables, time series, graphs, text, and images with ADEngine orchestration.

★ 9,867 · Python
X-PLUG/MobileAgent

MobileAgent Powerful GUI Agent

The MobileAgent project offers a family of powerful GUI agents in Python for automating interface interaction.

★ 9,379 · Python
apache/seatunnel

SEATUNNEL DATA INTEGRATION

A multimodal distributed tool for large-scale data integration with high performance in Java.

★ 9,359 · Java
bentoml/BentoML

BENTOML PATH TO AI API

Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.

★ 8,663 · Python
Neywa-Labs/mlx-audio

MLX-AUDIO: SPEECH ON APPLE SILICON

mlx-audio — library for TTS, STT and STS on Apple's MLX framework. Runs efficiently on Apple Silicon. 8192 stars on GitHub.

★ 8,267 · Python
Blaizzy/mlx-audio

Speech Tools for MLX

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

★ 7,115 · Python
ARahim3/DigUp

DIGUP SEARCH BY FILE CONTENT

DigUp indexes your local files — photos, PDFs, videos, audio and code — and lets you find them by meaning of the query. Search runs on-device using the EmbeddingGemma 2 m

★ 6,418 · Swift
om-ai-lab/VLM-R1

VLM-R1: Smart Image Recognition

The project demonstrates how reinforcement learning improves image understanding by multimedia models.

★ 6,046 · Python
genkit-ai/genkit

Genkit AI Framework

An open-source framework for building AI applications in JavaScript, Go, and Python. Developed and used at Google.

★ 5,985 · TypeScript
facebookresearch/mmf

MMF: Modular Multimodal Framework

Modular framework from FAIR for vision and language research, combining the flexibility of Python and the power of multimodal models.

★ 5,633 · Python
OpenBMB/UltraRAG

UltraRAG: RAG pipeline framework

Low-code MCP framework for creating complex and innovative RAG pipelines.

★ 5,549 · Python
Eventual-Inc/Daft

DAFT — Data Engine for AI

High-performance engine for AI and multimodal tasks that processes images, audio, video, and structured data at any scale. Built on Rust ⭐5533

★ 5,533 · Rust
InternLM/xtuner

XTUNER: TRAINER FOR LARGE MoE

A Next-Generation Training Engine Built for Ultra-Large MoE Models. ⭐ 5138 stars on GitHub!

★ 5,138 · Python
vllm-project/vllm-omni

Efficient Multimodal Inference

Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.

★ 4,863 · Python
kyegomez/tree-of-thoughts

TREE OF THOUGHTS: SMART SOLUTION SEARCH

Tree of Thoughts implementation for LLMs: model builds reasoning tree, evaluates branches, picks best path. Boosts solution quality by 70%+ without fine-tuning.

★ 4,592 · Python
EvolvingLMMs-Lab/lmms-eval

LMMS-EVAL - ONE-STOP MODALITY EVAL

Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.

★ 4,169 · Python
aiming-lab/SimpleMem

MEMORY FOR AI AGENTS

SimpleMem — efficient long-term memory for LLM agents with support for text and multimodal data.

★ 3,878 · Python
embeddings-benchmark/mteb

MTEB - Benchmark for Text Embeddings

Large-scale benchmark for evaluating the quality of text embeddings. 800+ tasks across 50+ languages.

★ 3,287 · Python
vortex-data/vortex

VORTEX — MOST STRUCTURED DATA FORMAT

Vortex is an extensible framework for columnar data compression and the fastest open columnar format. It is the primary tool for next‑generation analytics from LFAI&Data.

★ 2,966 · Rust
xlang-ai/OSWorld

OSWorld: BENCHMARK FOR AI AGENTS

OSWorld is a benchmark for evaluating multimodal AI agents in real computer environments. Open tasks in real operating systems.

★ 2,911 · Python
autodistill/autodistill

AUTODISTILL: TRAINING WITHOUT LABELING

Automatically turns raw images into ready-to-use inference models, using foundation models to generate pseudo‑labels and train supervised models. No manual labeling requi

★ 2,798 · Python
datachain-ai/datachain

DATACHAIN - DATA WITH AI

Typed versioned datasets for unstructured data on top of S3, GCS, and Azure. A great tool for working with AI information.

★ 2,748 · Python
roboflow/maestro

MAESTRO - Simplifying Model Fine-tuning

The project simplifies fine-tuning of multimodal models PaliGemma 2, Florence‑2, and Qwen2.5‑VL, saving developers time and resources.

★ 2,696 · Python
TanStack/ai

TanStack AI SDK

🤖 Type-safe AI SDK for TypeScript supporting streaming chat, tool calls, agents, and multimodal applications with OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Soli

★ 2,684 · TypeScript
genieincodebottle/generative-ai

Generative AI Roadmap Projects

A collection of generative AI resources: roadmap, projects, interview and coding preparation.

★ 2,670 · Jupyter Notebook
om-ai-lab/OmAgent

OMAGENT: THE MULTIMODAL AGENT

OmAgent — Python framework for multimodal agents on Gemini, GPT-4 and other LLMs. Enables rapid prototyping and deployment via Gradio, supports RAG and rapid chatbot temp

★ 2,670 · Python
AI-Efficiency/Awesome-Model-Quantization

AWESOME MODEL QUANTIZATION RESOURCES

A curated list of articles, benchmarks, surveys, and tools for neural network quantization. Covers low-bit networks, LLMs, multimodal and generative models, vector and la

★ 2,469
Yutong-Zhou-cv/Awesome-Text-to-Image

IMAGE GENERATION REVIEW

A large-scale review of text-to-image generation methods: architectures, models, and approaches — all collected in one repository.

★ 2,446
HUANGCHIHHUNGLeo/claude-real-video

claude-real-video - WATCH VIDEO WITH CLAUDE

Allows any LLM to "watch" video: extracts key frames and transcript, works locally.

★ 2,319 · Python
OpenGVLab/InternVideo

INTERNVIDEO VIDEO FOUNDATION MODELS

The InternVideo project offers open video foundation models and a dataset for multimodal understanding, presented at ECCV2024.

★ 2,281 · Python
google-gemini/genai-processors

PARALLEL CONTENT PROCESSING IN PYTHON

Lightweight library for efficient parallel content processing with GenAI. 2110 stars on GitHub!

★ 2,123 · Python
2U1/Qwen-VL-Series-Finetune

QWEN-VL SERIES FINETUNE OPEN SOURCE

Open-source implementation for fine-tuning Qwen-VL series models from Alibaba Cloud in Python. Allows easy adaptation of multimodal models for custom tasks.

★ 1,983 · Python
kyegomez/BitNet

BITNET: 1-BIT TRANSFORMER

BitNet implementation: 1-bit transformers for LLM in PyTorch. Open source code for fast experiments.

★ 1,949 · Python
NVIDIA-NeMo/DataDesigner

NVIDIA Data Designer

NeMo Data Designer: generation of high-quality synthetic data from scratch or based on seed data. Ideal for training and fine-tuning models.

★ 1,877 · Python
potamides/DeTikZify

DETIKZIFY DRAWS LATEX DIAGRAMS FAST

DeTikZify turns textual descriptions or hand‑drawn sketches into ready‑to‑use TikZ code for LaTeX. The project leverages HuggingFace and Llama models for inverse graphics

★ 1,831 · Python
Tencent/WeMM-Embedding

WeMM-Embedding by Tencent

Family of universal multimodal embedding models from the WeChat Vision team, supporting understanding and retrieval across modalities.

★ 1,818 · Python
sgl-project/sglang-omni

SGLANG-OMNI: OMNI-SERVICE FOR AI

SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.

★ 1,791 · Python
microsoft/Mage

MAGE: AI AGENT FOR PC CONTROL

Microsoft released Mage — an agent for GUI control via flow matching. The model edits images and performs computer tasks without fine-tuning.

★ 1,782 · Python
fikrikarim/parlor

Parlor AI on device

Parlor is an open project that enables natural voice and visual dialogues with AI running fully locally on your device. It is built on the Gemma 4 E2B and Kokoro models.

★ 1,752 · HTML
pixeltable/pixeltable

PIXELTABLE — DECLARATIVE AI BACKEND

Pixeltable is a declarative, incremental backend for multimodal AI applications, simplifying work with data and models.

★ 1,569 · Python
ATH-MaaS/Ovis

OVIS: Visual-Text Sync

A modern MLLM architecture for structural alignment of visual and text embeddings.

★ 1,539 · Python
UbiquitousLearning/mllm

FAST MULTIMODAL LLM ON MOBILE

A C++ library enables running multimodal LLMs directly on smartphones with high speed and low power consumption.

★ 1,533 · C++
valentinfrlch/ha-llmvision

Visual Intelligence for Home

ha-llmvision brings visual intelligence to your home, enabling real‑time object, face and activity recognition for smarter automation.

★ 1,531 · Python
emcf/thepipe

THEPIPE: DATA FROM DOCUMENTS FAST

ThePipe uses vision‑language models to extract data from PDF, Word and PowerPoint. It recognises tables, text and images, turning them into ready‑to‑use structures. Suppo

★ 1,530 · Python
xtreme1-io/xtreme1

XTREME1 — PLATFORM FOR MULTIMODAL ANNOTATION

Xtreme1 — platform for annotating 3D‑LiDAR, images and LLM data. Supports collaborative work, AI suggestions and export to COCO, KITTI, Hugging Face.

★ 1,393 · TypeScript
gokayfem/awesome-vlm-architectures

VISION AND LANGUAGE: TOP MODELS

A collection of architectures of well-known vision‑language models with explanations and links.

★ 1,368 · Markdown
Capsize-Games/airunner

AIRUNNER OFFLINE AI FOR ALL

Offline inference engine for arts, real-time voice dialogues, LLM chatbots, and automated workflows.

★ 1,315 · Python
ysr666/dsh-vision-router

dsh-vision-router - VISION FOR AGENTS

Free vision chain for DeepSeek Harness agents: Q&A, grounding, OCR, and more. Single-command install, no Python, work with images like a regular tool.

★ 1,274 · JavaScript
Anionex/agent-vision-toolkit

Visual Toolkit for LLM

A library for text models that lets them 'see' images: recognize content, answer questions, restore UI, automate GUI, and integrate with agents.

★ 1,262 · Python
unum-cloud/UForm

MULTIMODAL AI FOR CONTENT

UForm — compact MULTIMODAL AI for understanding and generating content. Processes multilingual texts, images and video, working 5x faster than OpenAI CLIP and LLaVA.

★ 1,253 · Python
Zen4-bit/Proxima

PROXIMA: UNITE AI WITH TOOLS

Proxima — an MCP server that connects ChatGPT, Claude, Gemini and Perplexity to your development tools without direct API calls. It supports JavaScript and has already ga

★ 1,210 · JavaScript

New projects on this topic in Telegram

Telegram →
Multimodal — 143 open-source GitHub projects | REDDYX AI