Benchmark: open-source projects on GitHub
54 repositories on Benchmark, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
LLM evaluation platform
OpenCompass is an LLM evaluation platform supporting a wide range of models (Llama3, Mistral, InternLM2, GPT-4, LLaMa2, Qwen, GLM, Claude, etc.) across 100+ datasets.
★ 7,022 · PythonLMMS-EVAL - ONE-STOP MODALITY EVAL
Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.
★ 4,169 · PythonMTEB - Benchmark for Text Embeddings
Large-scale benchmark for evaluating the quality of text embeddings. 800+ tasks across 50+ languages.
★ 3,287 · PythonOSWorld: BENCHMARK FOR AI AGENTS
OSWorld is a benchmark for evaluating multimodal AI agents in real computer environments. Open tasks in real operating systems.
★ 2,911 · PythonINTERNVIDEO VIDEO FOUNDATION MODELS
The InternVideo project offers open video foundation models and a dataset for multimodal understanding, presented at ECCV2024.
★ 2,281 · PythonAwesome LLM Long Context - Top Papers
A curated selection of must-read articles and blogs on long context in LLMs: new methods, reviews, and practices.
★ 2,192ADR: AI Agent Security in Uber
ADR provides observability, security testing, and threat detection for enterprise AI agents, already used at Uber.
★ 2,135 · PythonHOTPATH: PROFILER FOR PRODUCTION
Hotpath — Rust profiler for CPU, memory, SQL, HTTP, and async code. Exports metrics to Prometheus and builds Grafana dashboards. Has 1896 stars on GitHub.
★ 1,925 · RustTOOLKIT FOR CLINICAL AI
PyHealth is a library for healthcare AI supporting EHR (MIMIC, eICU, OMOP), clinical texts, signals, and imaging. 30+ models and reproducible pipelines for research.
★ 1,699 · PythonLLM EVALUATION SURVEY
Official repository page for the survey paper on evaluation of large language models.
★ 1,616FIND RUST BOTTLENECKS FAST
hotpath-rs is a single profiler for CPU, time, memory, and async code. It quickly finds bottlenecks in your Rust project.
★ 1,517 · RustFAST COMPILER FOR EXPRESSIONS
FastExpressionCompiler — fast compiler for C# expression trees and a lightweight LightExpression alternative. Provides diagnostic and code‑generation tools for expression
★ 1,380 · C#TAU2-Bench: Benchmark for AI Agents
τ-Bench tests AI agents in real‑world tasks. The Python project already has 1.3k stars.
★ 1,319 · PythonClawBench - Benchmark for AI Agents
Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!
★ 1,301 · PythonSKILLEVALUATOR: AI AGENT EVALUATION
A multi-level framework for evaluating AI agent skills: qualitative gates, semantic overlap, synthetic data, and live testing.
★ 1,286 · PythonClinical Evidence Benchmark
Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.
★ 1,131 · PythonCode Agent for LLM
Code agent, optimized for smaller LLMs.
★ 1,083 · TypeScriptInferenceX – Inference Benchmark
Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.
★ 1,016 · PythonGLM-5.3 FLASH AND J-SPACE BENCHMARK
The project shows integration of GLM-5.3 Flash with J-Space for full benchmarking the J-Space Cognition Suite. Results show agent skills implementation and baseline compa
★ 1,016MOBILEGYM: ANDROID AGENT TRAINING
MobileGym is a browser‑hosted Android emulator for research on mobile GUI agents. The platform provides verifiable evaluation and highly parallel online reinforcement‑lea
★ 866 · PythonVLM2Vec: Vector Models for Multimodality
The library implements the VLM2Vec method for training Vision‑Language models that produce effective multimodal embeddings. Presented at ICLR 2025.
★ 704 · PythonSkill corpus for agents
The project collects SKILL.md files into a single searchable skill catalog for AI agents, enabling fast search and automatic task-matching evaluation.
★ 703 · PythonAI ARTIST FOR ASEPRITE
Agent draws pixel art directly in the open Aseprite window. It uses the MCP server and skills for Claude Code, omp, Codex, Gemini CLI and Cursor — drawing, checking, anim
★ 694 · TypeScriptTRUSTLLM: EVALUATION OF LLM TRUSTWORTHINESS
TrustLLM — a toolkit for assessing LLM trustworthiness, presented at ICML 2024. It includes benchmark suites, metrics, and datasets for measuring hallucinations, bias, an
★ 636 · PythonWEBSITE RECONSTRUCTION BENCHMARK FOR AGENTS
Benchmark for evaluating the ability of AI agents to reconstruct full-featured websites through browser exploration. Contains 70 tasks with 46,916 hidden test cases for v
★ 476 · PythonPARSEBENCH: AI AGENT EVALUATION
ParseBench is a benchmark for evaluating the quality of document parsing by AI agents. It allows comparing models on real-world datasets.
★ 472 · PythonOpenRCA: AI Finds Root Causes in Code
OpenRCA is an LLM-based framework for automatic root cause detection of failures in software. Accepted at ICLR 2025.
★ 453 · PythonMCPMark: AI Agent Benchmark
A comprehensive MCP benchmark for testing models and agents in real-world scenarios. 426 stars on GitHub.
★ 426 · PythonOSWorld-V2 - AGENT REAL TASK TEST
OSWorld 2.0 evaluates AI agents' ability to complete long real computer tasks using only Python.
★ 421 · PythonDS4 on Spark: 3X Speedup
One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.
★ 415 · ShellNext-Gen Transformer Framework
Open-source framework for next-gen Transformers with compute-matched benchmarking and autonomous architecture research platform.
★ 412 · PythonWorld Action Model Toolkit
Provides a unified Python framework for training, fine-tuning, and evaluating World Action Models (WAMs) with modular components and standard APIs.
★ 411 · PythonOVERTIME: AGENT FOR GAME DEVELOPMENT
Project overtime lets you give a brief technical task to an AI agent, after which the agent autonomously performs work in subsequent sessions. In the demo, it built an op
★ 402 · TypeScriptLLM THROUGHPUT BENCHMARK IN OLLAMA
Tool for measuring local LLM throughput via Ollama. Runs load tests and outputs tokens-per-second statistics.
★ 399 · PythonAwesome AI Leaderboard
Curated list of resources with leaderboards for the AI domain. Benchmarks, metrics, and model comparisons.
★ 384STRIX HALO LLM Setup
Guide for LLM on Ryzen AI MAX+ 395 and Radeon 8060S via Ollama, llama.cpp and ROCm. Achieving 100 tokens/s 30B Qwen and 140 tokens/s CHADROCK MTP.
★ 373 · PythonMINEBENCH: AI BENCHMARK
Minecraft‑style voxel benchmark comparing AI models in arena and sandbox.
★ 342 · TypeScriptMOBILEWORLD - MOBILE AGENTS BENCHMARK
Benchmark for autonomous mobile agents in interactive environments with MCP support. ACL 2026.
★ 301 · PythonLYNNREAL-OMNI: VIDEO IN 4 STEPS
LynnReal-Omni unifies text‑to‑video, image‑to‑video, human and hand‑pose control, structural control, omni‑reference generation, style transfer, video editing, degraded‑v
★ 281 · PythonRESEARCHCLAWBENCH - IMPROVING AI RESEARCH EFFECTIVENESS
Evaluation of automated AI agents: from rediscovery to new discoveries. Runs in Jupyter Notebook.
★ 280 · Jupyter NotebookSWE-BENCH LIVE: EVALUATE LLMS IN CODING
SWE-bench Live — benchmark for evaluating AI agents on real GitHub issues. Automated code verification and comparison of any models.
★ 262 · PythonNYT Connections: LLM Benchmark
A benchmark based on 759 NYT Connections puzzles with additional trick words for evaluating LLMs.
★ 250 · PythonLLM Bug-Sweeping Benchmark
A Python benchmark that measures how many bugs LLMs can detect and fix across large codebases.
★ 239 · PythonMYCLAW BENCH: AI AGENT TEST
Test AI agents on 45 tasks across four difficulty levels. Powered by MyClaw.ai.
★ 222 · PythonAI4AI: 1135 PAPERS FOR AGI
Weekly updated collection of 1135 papers on benchmarks, harnesses, and agent architectures. Covers long-horizon agents and recursive self-improvement. Includes citation r
★ 221 · PythonFIND 105 BUGS
Bug Hunt Bench — benchmark 105 real bugs in 2 repositories. AI models (GPT-6, Claude, Grok, Gemini, DeepSeek) find and fix them in CLI, evaluated blind on leaderboard, ou
★ 207 · JavaScriptGLYD: LOSSLESS AI COMPRESSION
Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma
★ 200 · RustModel Benchmarking Platform
The project provides tools for comparing machine models as real-time controllers of engineering loops, measuring their latency, stability, and resource consumption.
★ 155 · PythonLoRA Fine-Tuning Sprint
Project accelerates LoRA fine-tuning by fixing task and hardware, and posts results to a public wall-clock leaderboard based on modified nanogpt.
★ 151 · PythonAgentic RL Tutorial & Benchmark
33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.
★ 146 · PythonGalahad: Blindness Research
The Galahad project studies instruction blindness in vision-language-action policies, offering diagnostics and a low-rank data correction method. It includes a paper, mod
★ 142 · PythonAWESOME-AGENT-EVOLUTION: AGENT EVOLUTION MAP
Open map covering the evolution of AI agents, their self-improvement, memory, skills, and swarm systems. 🌟 208 stars, JS
★ 68 · JavaScriptARD Resource Search Platform
Neuronto Agentic Resource Discovery indexes public ARD registries and provides federated search, combined with a verified tool index from MCP servers, using hybrid lexica
★ 57 · PythonSmith Adaptive Model Router
Agent Smith routes model calls adaptively, monitors Codex token usage in real time, and generates reproducible benchmark reports with verification receipts.
★ 9 · PythonNew projects on this topic in Telegram
Telegram →