R_REDDYX.XYZ

Benchmark: open-source projects on GitHub

54 repositories on Benchmark, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Evaluation 46AI Agents 1319LLM Evaluation 45Multimodal 143MCP 1601Large Language Models 154VLM 50Agent 547Claude Code 1595AI Agent 690Codex 849Deep Learning 105Machine Learning 190
open-compass/opencompass

LLM evaluation platform

OpenCompass is an LLM evaluation platform supporting a wide range of models (Llama3, Mistral, InternLM2, GPT-4, LLaMa2, Qwen, GLM, Claude, etc.) across 100+ datasets.

★ 7,022 · Python
EvolvingLMMs-Lab/lmms-eval

LMMS-EVAL - ONE-STOP MODALITY EVAL

Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.

★ 4,169 · Python
embeddings-benchmark/mteb

MTEB - Benchmark for Text Embeddings

Large-scale benchmark for evaluating the quality of text embeddings. 800+ tasks across 50+ languages.

★ 3,287 · Python
xlang-ai/OSWorld

OSWorld: BENCHMARK FOR AI AGENTS

OSWorld is a benchmark for evaluating multimodal AI agents in real computer environments. Open tasks in real operating systems.

★ 2,911 · Python
OpenGVLab/InternVideo

INTERNVIDEO VIDEO FOUNDATION MODELS

The InternVideo project offers open video foundation models and a dataset for multimodal understanding, presented at ECCV2024.

★ 2,281 · Python
Xnhyacinth/Awesome-LLM-Long-Context-Modeling

Awesome LLM Long Context - Top Papers

A curated selection of must-read articles and blogs on long context in LLMs: new methods, reviews, and practices.

★ 2,192
uber/ADR

ADR: AI Agent Security in Uber

ADR provides observability, security testing, and threat detection for enterprise AI agents, already used at Uber.

★ 2,135 · Python
hotpath-rs/hotpath

HOTPATH: PROFILER FOR PRODUCTION

Hotpath — Rust profiler for CPU, memory, SQL, HTTP, and async code. Exports metrics to Prometheus and builds Grafana dashboards. Has 1896 stars on GitHub.

★ 1,925 · Rust
sunlabuiuc/PyHealth

TOOLKIT FOR CLINICAL AI

PyHealth is a library for healthcare AI supporting EHR (MIMIC, eICU, OMOP), clinical texts, signals, and imaging. 30+ models and reproducible pipelines for research.

★ 1,699 · Python
MLGroupJLU/LLM-eval-survey

LLM EVALUATION SURVEY

Official repository page for the survey paper on evaluation of large language models.

★ 1,616
pawurb/hotpath-rs

FIND RUST BOTTLENECKS FAST

hotpath-rs is a single profiler for CPU, time, memory, and async code. It quickly finds bottlenecks in your Rust project.

★ 1,517 · Rust
dadhi/FastExpressionCompiler

FAST COMPILER FOR EXPRESSIONS

FastExpressionCompiler — fast compiler for C# expression trees and a lightweight LightExpression alternative. Provides diagnostic and code‑generation tools for expression

★ 1,380 · C#
sierra-research/tau2-bench

TAU2-Bench: Benchmark for AI Agents

τ-Bench tests AI agents in real‑world tasks. The Python project already has 1.3k stars.

★ 1,319 · Python
TIGER-AI-Lab/ClawBench

ClawBench - Benchmark for AI Agents

Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!

★ 1,301 · Python
NVIDIA/SkillEvaluator

SKILLEVALUATOR: AI AGENT EVALUATION

A multi-level framework for evaluating AI agent skills: qualitative gates, semantic overlap, synthetic data, and live testing.

★ 1,286 · Python
xinxuxin/keystone-bench

Clinical Evidence Benchmark

Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.

★ 1,131 · Python
itayinbarr/little-coder

Code Agent for LLM

Code agent, optimized for smaller LLMs.

★ 1,083 · TypeScript
SemiAnalysisAI/InferenceX

InferenceX – Inference Benchmark

Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.

★ 1,016 · Python
Tiger3807861189/GLM-5.3-Flash-J-Space-Capability-Realization-Report

GLM-5.3 FLASH AND J-SPACE BENCHMARK

The project shows integration of GLM-5.3 Flash with J-Space for full benchmarking the J-Space Cognition Suite. Results show agent skills implementation and baseline compa

★ 1,016
Purewhiter/mobilegym

MOBILEGYM: ANDROID AGENT TRAINING

MobileGym is a browser‑hosted Android emulator for research on mobile GUI agents. The platform provides verifiable evaluation and highly parallel online reinforcement‑lea

★ 866 · Python
TIGER-AI-Lab/VLM2Vec

VLM2Vec: Vector Models for Multimodality

The library implements the VLM2Vec method for training Vision‑Language models that produce effective multimodal embeddings. Presented at ICLR 2025.

★ 704 · Python
EverMind-AI/SkillCorpus

Skill corpus for agents

The project collects SKILL.md files into a single searchable skill catalog for AI agents, enabling fast search and automatic task-matching evaluation.

★ 703 · Python
with-pebbly/aseprite-ai-artist

AI ARTIST FOR ASEPRITE

Agent draws pixel art directly in the open Aseprite window. It uses the MCP server and skills for Claude Code, omp, Codex, Gemini CLI and Cursor — drawing, checking, anim

★ 694 · TypeScript
HowieHwong/TrustLLM

TRUSTLLM: EVALUATION OF LLM TRUSTWORTHINESS

TrustLLM — a toolkit for assessing LLM trustworthiness, presented at ICML 2024. It includes benchmark suites, metrics, and datasets for measuring hallucinations, bias, an

★ 636 · Python
TIGER-AI-Lab/WebsiteBench

WEBSITE RECONSTRUCTION BENCHMARK FOR AGENTS

Benchmark for evaluating the ability of AI agents to reconstruct full-featured websites through browser exploration. Contains 70 tasks with 46,916 hidden test cases for v

★ 476 · Python
run-llama/ParseBench

PARSEBENCH: AI AGENT EVALUATION

ParseBench is a benchmark for evaluating the quality of document parsing by AI agents. It allows comparing models on real-world datasets.

★ 472 · Python
microsoft/OpenRCA

OpenRCA: AI Finds Root Causes in Code

OpenRCA is an LLM-based framework for automatic root cause detection of failures in software. Accepted at ICLR 2025.

★ 453 · Python
eval-sys/mcpmark

MCPMark: AI Agent Benchmark

A comprehensive MCP benchmark for testing models and agents in real-world scenarios. 426 stars on GitHub.

★ 426 · Python
xlang-ai/OSWorld-V2

OSWorld-V2 - AGENT REAL TASK TEST

OSWorld 2.0 evaluates AI agents' ability to complete long real computer tasks using only Python.

★ 421 · Python
Entrpi/ds4-on-spark

DS4 on Spark: 3X Speedup

One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.

★ 415 · Shell
Interactive-Active-AI/NextTransformer

Next-Gen Transformer Framework

Open-source framework for next-gen Transformers with compute-matched benchmarking and autonomous architecture research platform.

★ 412 · Python
OpenMOSS/EasyWAM

World Action Model Toolkit

Provides a unified Python framework for training, fine-tuning, and evaluating World Action Models (WAMs) with modular components and standard APIs.

★ 411 · Python
ukanwat/overtime

OVERTIME: AGENT FOR GAME DEVELOPMENT

Project overtime lets you give a brief technical task to an AI agent, after which the agent autonomously performs work in subsequent sessions. In the demo, it built an op

★ 402 · TypeScript
aidatatools/ollama-benchmark

LLM THROUGHPUT BENCHMARK IN OLLAMA

Tool for measuring local LLM throughput via Ollama. Runs load tests and outputs tokens-per-second statistics.

★ 399 · Python
SAILResearch/awesome-ai-leaderboard

Awesome AI Leaderboard

Curated list of resources with leaderboards for the AI domain. Benchmarks, metrics, and model comparisons.

★ 384
hogeheer499-commits/strix-halo-guide

STRIX HALO LLM Setup

Guide for LLM on Ryzen AI MAX+ 395 and Radeon 8060S via Ollama, llama.cpp and ROCm. Achieving 100 tokens/s 30B Qwen and 140 tokens/s CHADROCK MTP.

★ 373 · Python
Ammaar-Alam/minebench

MINEBENCH: AI BENCHMARK

Minecraft‑style voxel benchmark comparing AI models in arena and sandbox.

★ 342 · TypeScript
Tongyi-MAI/MobileWorld

MOBILEWORLD - MOBILE AGENTS BENCHMARK

Benchmark for autonomous mobile agents in interactive environments with MCP support. ACL 2026.

★ 301 · Python
LynnReal-AI/LynnReal-Omni

LYNNREAL-OMNI: VIDEO IN 4 STEPS

LynnReal-Omni unifies text‑to‑video, image‑to‑video, human and hand‑pose control, structural control, omni‑reference generation, style transfer, video editing, degraded‑v

★ 281 · Python
InternScience/ResearchClawBench

RESEARCHCLAWBENCH - IMPROVING AI RESEARCH EFFECTIVENESS

Evaluation of automated AI agents: from rediscovery to new discoveries. Runs in Jupyter Notebook.

★ 280 · Jupyter Notebook
microsoft/SWE-bench-Live

SWE-BENCH LIVE: EVALUATE LLMS IN CODING

SWE-bench Live — benchmark for evaluating AI agents on real GitHub issues. Automated code verification and comparison of any models.

★ 262 · Python
lechmazur/nyt-connections

NYT Connections: LLM Benchmark

A benchmark based on 759 NYT Connections puzzles with additional trick words for evaluating LLMs.

★ 250 · Python
facebookresearch/swe-sweep

LLM Bug-Sweeping Benchmark

A Python benchmark that measures how many bugs LLMs can detect and fix across large codebases.

★ 239 · Python
LeoYeAI/myclaw-bench

MYCLAW BENCH: AI AGENT TEST

Test AI agents on 45 tasks across four difficulty levels. Powered by MyClaw.ai.

★ 222 · Python
KaiWU5/Awesome-AI4AI

AI4AI: 1135 PAPERS FOR AGI

Weekly updated collection of 1135 papers on benchmarks, harnesses, and agent architectures. Covers long-horizon agents and recursive self-improvement. Includes citation r

★ 221 · Python
phuryn/bug-hunt-bench

FIND 105 BUGS

Bug Hunt Bench — benchmark 105 real bugs in 2 repositories. AI models (GPT-6, Claude, Grok, Gemini, DeepSeek) find and fix them in CLI, evaluated blind on leaderboard, ou

★ 207 · JavaScript
surya-koritala/Glyd

GLYD: LOSSLESS AI COMPRESSION

Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma

★ 200 · Rust
AMAP-ML/LoopArena

Model Benchmarking Platform

The project provides tools for comparing machine models as real-time controllers of engineering loops, measuring their latency, stability, and resource consumption.

★ 155 · Python
Saivineeth147/lora-speedrun

LoRA Fine-Tuning Sprint

Project accelerates LoRA fine-tuning by fixing task and hardware, and posts results to a public wall-clock leaderboard based on modified nanogpt.

★ 151 · Python
cookiespiggy/agentic-rl

Agentic RL Tutorial & Benchmark

33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.

★ 146 · Python
phi-monster/Galahad

Galahad: Blindness Research

The Galahad project studies instruction blindness in vision-language-action policies, offering diagnostics and a low-rank data correction method. It includes a paper, mod

★ 142 · Python
Shiyao-Huang/awesome-agent-evolution

AWESOME-AGENT-EVOLUTION: AGENT EVOLUTION MAP

Open map covering the evolution of AI agents, their self-improvement, memory, skills, and swarm systems. 🌟 208 stars, JS

★ 68 · JavaScript
neuronto/agentic-resource-discovery

ARD Resource Search Platform

Neuronto Agentic Resource Discovery indexes public ARD registries and provides federated search, combined with a verified tool index from MCP servers, using hybrid lexica

★ 57 · Python
Chengjun023/agent-smith

Smith Adaptive Model Router

Agent Smith routes model calls adaptively, monitors Codex token usage in real time, and generates reproducible benchmark reports with verification receipts.

★ 9 · Python

New projects on this topic in Telegram

Telegram →
Benchmark — 54 open-source GitHub projects | REDDYX AI