Evaluation: open-source projects on GitHub
46 repositories on Evaluation, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Langfuse: LLM Platform
Open platform for LLM: observability, metrics, prompt management, testing, and integration with popular tools.
★ 27,378 · TypeScriptMLflow — AI Platform
Open platform for AI engineering. Allows teams to debug, evaluate, and optimize AI applications while managing access and costs.
★ 26,029 · PythonTesting LLM Models
Tool for testing prompts, agents, and RAG systems. Supports comparative analysis of GPT, Claude, Gemini, Llama, and other models.
★ 21,294 · TypeScriptLLM Debugging and Monitoring
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
★ 19,332 · PythonLLM EVALUATION RAG
Ragas is a framework for LLM application evaluation. It automates quality checks and tracks key metrics.
★ 16,320 · PythonLLM evaluation platform
OpenCompass is an LLM evaluation platform supporting a wide range of models (Llama3, Mistral, InternLM2, GPT-4, LLaMa2, Qwen, GLM, Claude, etc.) across 100+ datasets.
★ 7,022 · PythonHELICONE LLM OBSERVE
Open source platform for LLM observability: monitoring, evaluation, and experiments in one line of code. Supports TypeScript and already has 5.8k stars.
★ 5,805 · TypeScriptAI Agent Lifecycle Platform
Next-generation AI Agent Optimization Platform: Cozeloop tackles AI agent development challenges with full lifecycle management from development and debugging to evaluati
★ 5,471 · GoBuild AI Systems
Build, evaluate, and optimize AI systems. Includes evals, RAG, agents, fine‑tuning, synthetic data generation, dataset management, MCP, and more.
★ 4,848 · PythonAutoRAG: RAG System Automation
An open-source framework for automatic evaluation and optimization of RAG pipelines in an AutoML style.
★ 4,804 · PythonVLMEvalKit - TOP TOOL FOR MODEL EVALUATION
Open tool for evaluating multimodal models. Supports 220+ LMM and 80+ benchmarks.
★ 4,492 · PythonLMMS-EVAL - ONE-STOP MODALITY EVAL
Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.
★ 4,169 · PythonLANGWATCH — TEST YOUR LLM
Platform for evaluating LLMs and testing AI agents. Convenient tools for debugging and monitoring AI applications.
★ 3,274 · TypeScriptEVALSCOPE: SIMPLE LLM TESTING
Flexible Python framework for evaluating the performance of large models (LLM, VLM, AIGC). ⭐2874
★ 2,874 · PythonFABLE-METHOD: THREE KEYS
Protocol from Claude Fable 5: three steps – Think, Act, Prove – turned into a real workflow for any LLM.
★ 2,313 · PythonAwesome LLM Long Context - Top Papers
A curated selection of must-read articles and blogs on long context in LLMs: new methods, reviews, and practices.
★ 2,192Inspector - MCP Server Debug
Platform for testing, inspecting, and debugging MCP servers, MCP apps, and ChatGPT apps. 1968 ⭐
★ 1,968 · TypeScriptWFGY 5.0 POLARIS PROTOCOL
WFGY 5.0 — major open-source release for AI reasoning, RAG, agents and real-world workflows. 1752 ⭐
★ 1,752 · Jupyter NotebookLLM EVALUATION SURVEY
Official repository page for the survey paper on evaluation of large language models.
★ 1,616ClawBench - Benchmark for AI Agents
Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!
★ 1,301 · PythonTRACELY-AI - AUTOMATED AGENT TESTING
Tracely-ai automatically turns AI agent failures into regression tests that block PRs and reproduces them in CI for free.
★ 1,299 · PythonSKILLEVALUATOR: AI AGENT EVALUATION
A multi-level framework for evaluating AI agent skills: qualitative gates, semantic overlap, synthetic data, and live testing.
★ 1,286 · PythonClinical Evidence Benchmark
Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.
★ 1,131 · PythonObserval - AI Agent Oversight
Observal is a platform for monitoring, evaluating, and debugging AI agents with human-in-the-loop feedback. It is essential for building reliable production pipelines.
★ 1,118 · PythonOPENJUDGE - NEW AI EVALUATOR
Unified framework for evaluating the quality of AI agents. 671⭐ Python.
★ 1,105 · PythonGYM - Model Evaluation
Helps evaluate and improve LLMs through environments and agents with feedback and testing.
★ 932 · PythonPROTECTION OF AI AGENTS FROM THREATS
safelabs-eval — framework for red‑teaming and evaluation of AI agent protection based on the ASI01–ASI10 taxonomy inspired by OWASP. Written in Python, it supports CrewAI
★ 829 · PythonRepo for LLM Unlearning
A collection of papers, code, and resources on machine 'unlearning' in large language models. Helps researchers remove knowledge from LLMs.
★ 638TRUSTLLM: EVALUATION OF LLM TRUSTWORTHINESS
TrustLLM — a toolkit for assessing LLM trustworthiness, presented at ICML 2024. It includes benchmark suites, metrics, and datasets for measuring hallucinations, bias, an
★ 636 · PythonXRAG: EVALUATION OF RAG MODULES
XRAG evaluates core components of RAG systems: indexing, retrieval, and generation. Using standard datasets it compares different module implementations, revealing their
★ 612 · PythonMMMU: Multimodal Benchmark
Code evaluation for MMMU benchmark measuring multidisciplinary multimodal understanding and reasoning for expert AGI.
★ 601 · PythonPARSEBENCH: AI AGENT EVALUATION
ParseBench is a benchmark for evaluating the quality of document parsing by AI agents. It allows comparing models on real-world datasets.
★ 472 · PythonHYPHA Agent Framework
TypeScript framework for production‑grade LLM agents. Enables rapid creation and management of agents via a flexible harness architecture.
★ 452 · TypeScriptRAG System for GitHub Issues
RAG over GitHub issues with hand-reviewed eval set and judge audit. Reports confidence intervals, not just point metrics.
★ 419 · PythonAwesome AI Leaderboard
Curated list of resources with leaderboards for the AI domain. Benchmarks, metrics, and model comparisons.
★ 384TEMPLATE GENAIOPS PROMPT FLOW
Template GenAIOps with Prompt Flow simplifies creation of LLM applications through centralized hosting, lifecycle management, and experiments with prompt variants and hyp
★ 370 · PythonRAG EXPERIMENT ACCELERATOR
Tool for rapid RAG experiments with Azure Cognitive Search. Automates chunking, embeddings, and answer quality evaluation.
★ 319 · PythonBOCOEL: BAYESIAN OPTIMIZATION FOR LLM
bocoel — a Bayesian Optimization framework for quick LLM evaluation: 10× faster, just a few lines of code. It replaces costly benchmarks with precise search over model hy
★ 288 · PythonRESEARCHCLAWBENCH - IMPROVING AI RESEARCH EFFECTIVENESS
Evaluation of automated AI agents: from rediscovery to new discoveries. Runs in Jupyter Notebook.
★ 280 · Jupyter NotebookNYT Connections: LLM Benchmark
A benchmark based on 759 NYT Connections puzzles with additional trick words for evaluating LLMs.
★ 250 · Pythonarksim - DETECT AGENT ERRORS
arksim helps developers find errors in AI agents before they reach real users, saving time and improving reliability.
★ 236 · PythonEVALUATE AGENTS WITH CALIPER
Caliper runs your real agent with skills, MCPs, and rules and without them, showing what actually helps and how many tokens it costs. Supports Claude Code, Codex, Pi, and
★ 235 · PythonFULL GUIDE TO LLM EVALUATION
LLMEvaluation provides a full guide to LLM evaluation methods aiding selection of optimal techniques for specific use-cases. It contains a metrics taxonomy, step‑by‑ste
★ 208 · HTMLBeginner AI Learning Path
A step-by-step AI roadmap covering foundations to production with free resources and hands-on mini-projects for each layer.
★ 194Model Benchmarking Platform
The project provides tools for comparing machine models as real-time controllers of engineering loops, measuring their latency, stability, and resource consumption.
★ 155 · PythonOpen Decision Model API
Provides an open decision model exposing Jev's API for typed answers (yes/no, choice, score, multi) with probabilities from forward passes, using Qwen3.5-4B fine‑tuned wi
★ 25 · PythonNew projects on this topic in Telegram
Telegram →