R_REDDYX.XYZ
MultimodalPython★ 918⑂ 88 forksApache-2.0

Valen-Team/Valen

MULTIMODAL DECISION MODEL

Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.

// KEY FEATURES

  • Returns candidate probabilities and bbox instead of generated text
  • Sizes from 0.8B to 27B: native weights, ONNX, GGUF, vLLM, SGLang
  • Free API, Python/JS/C++ SDK, integration with Claude Code and Cursor
  • Includes fine-tuning code, JevalKit for comparison with GPT/Claude/Gemini

// DEEP DIVE: HOW IT WORKS

Valen-Decision works as System One: given an instruction and multimodal context (text, images, video), it outputs a probability distribution over predefined answer candidates. Version 1.5 adds a second head for bounding box prediction, enabling the model to not only choose but also localize objects. The architecture uses block-bidirectional attention to preserve strong text capabilities alongside multimodality.

Unlike typical LLMs, it does not generate free-form text but returns structured JSON with probabilities and coordinates. Checkpoints from 0.8B to 27B are available in three formats: native (Transformers), ONNX Runtime, and GGUF with a patched llama.cpp. Ready-to-use SDKs for Python, JavaScript, and C++ are provided, plus an agent skill for Claude Code, Cursor, and Codex. JevalKit allows benchmarking Valen side-by-side with GPT, Claude, and Gemini.

Local deployment requires a GPU with VRAM matching the chosen model size (0.8B–27B). GGUF variants run via a custom llama.cpp server that accepts images. ONNX models require ONNX Runtime. The free API needs no key but has rate limits. Fine-tuning code (SFT and experimental RLCD) is included in the repo, datasets on Hugging Face.

// USE CASES

Best for: For agent and RAG developers needing structured decision outputs

  • Plug the model as a probabilistic classifier into a RAG pipeline instead of an LLM judge
  • Use the bbox head for visual grounding of agent answers in the UI
  • Compare Valen decision quality against GPT/Claude via JevalKit before production
#decision-model#fine-tuning#gguf#jev#multimodal#onnx#structured-output#system-one#typed-decisions#video-understanding#vision-language-model#visual-grounding
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

OPEN CREATIVE STUDIO ON COMFYUI★ 233REMOVING AI WATERMARKS AND METADATA★ 6403LTX-2: DIT MODEL FOR SYNCHRONIZED AUDIO AND VIDEO★ 10312EMBEDDED VECTOR DB FOR MULTIMODAL AI★ 11947OPEN SOURCE AI WORKSPACE FOR CREATORS★ 14387PROFESSIONAL AI ENGINE FOR VISUAL MEDIA★ 29106

← FULL CATALOG

Valen-Team/Valen — Multimodal Decision Model | REDDYX AI