R_REDDYX.XYZ

Evaluation: open-source projects on GitHub

46 repositories on Evaluation, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Benchmark 54LLM Evaluation 45Openai 690LLMOps 82Large Language Models 154RAG 533Machine Learning 190Prompt Engineering 175Langchain 196LLM Observability 24AI Agents 1319Monitoring 52Observability 92
langfuse/langfuse

Langfuse: LLM Platform

Open platform for LLM: observability, metrics, prompt management, testing, and integration with popular tools.

★ 27,378 · TypeScript
mlflow/mlflow

MLflow — AI Platform

Open platform for AI engineering. Allows teams to debug, evaluate, and optimize AI applications while managing access and costs.

★ 26,029 · Python
promptfoo/promptfoo

Testing LLM Models

Tool for testing prompts, agents, and RAG systems. Supports comparative analysis of GPT, Claude, Gemini, Llama, and other models.

★ 21,294 · TypeScript
comet-ml/opik

LLM Debugging and Monitoring

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

★ 19,332 · Python
vibrantlabsai/ragas

LLM EVALUATION RAG

Ragas is a framework for LLM application evaluation. It automates quality checks and tracks key metrics.

★ 16,320 · Python
open-compass/opencompass

LLM evaluation platform

OpenCompass is an LLM evaluation platform supporting a wide range of models (Llama3, Mistral, InternLM2, GPT-4, LLaMa2, Qwen, GLM, Claude, etc.) across 100+ datasets.

★ 7,022 · Python
Helicone/helicone

HELICONE LLM OBSERVE

Open source platform for LLM observability: monitoring, evaluation, and experiments in one line of code. Supports TypeScript and already has 5.8k stars.

★ 5,805 · TypeScript
coze-dev/coze-loop

AI Agent Lifecycle Platform

Next-generation AI Agent Optimization Platform: Cozeloop tackles AI agent development challenges with full lifecycle management from development and debugging to evaluati

★ 5,471 · Go
Kiln-AI/Kiln

Build AI Systems

Build, evaluate, and optimize AI systems. Includes evals, RAG, agents, fine‑tuning, synthetic data generation, dataset management, MCP, and more.

★ 4,848 · Python
Marker-Inc-Korea/AutoRAG

AutoRAG: RAG System Automation

An open-source framework for automatic evaluation and optimization of RAG pipelines in an AutoML style.

★ 4,804 · Python
open-compass/VLMEvalKit

VLMEvalKit - TOP TOOL FOR MODEL EVALUATION

Open tool for evaluating multimodal models. Supports 220+ LMM and 80+ benchmarks.

★ 4,492 · Python
EvolvingLMMs-Lab/lmms-eval

LMMS-EVAL - ONE-STOP MODALITY EVAL

Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.

★ 4,169 · Python
langwatch/langwatch

LANGWATCH — TEST YOUR LLM

Platform for evaluating LLMs and testing AI agents. Convenient tools for debugging and monitoring AI applications.

★ 3,274 · TypeScript
modelscope/evalscope

EVALSCOPE: SIMPLE LLM TESTING

Flexible Python framework for evaluating the performance of large models (LLM, VLM, AIGC). ⭐2874

★ 2,874 · Python
Sahir619/fable-method

FABLE-METHOD: THREE KEYS

Protocol from Claude Fable 5: three steps – Think, Act, Prove – turned into a real workflow for any LLM.

★ 2,313 · Python
Xnhyacinth/Awesome-LLM-Long-Context-Modeling

Awesome LLM Long Context - Top Papers

A curated selection of must-read articles and blogs on long context in LLMs: new methods, reviews, and practices.

★ 2,192
MCPJam/inspector

Inspector - MCP Server Debug

Platform for testing, inspecting, and debugging MCP servers, MCP apps, and ChatGPT apps. 1968 ⭐

★ 1,968 · TypeScript
onestardao/WFGY

WFGY 5.0 POLARIS PROTOCOL

WFGY 5.0 — major open-source release for AI reasoning, RAG, agents and real-world workflows. 1752 ⭐

★ 1,752 · Jupyter Notebook
MLGroupJLU/LLM-eval-survey

LLM EVALUATION SURVEY

Official repository page for the survey paper on evaluation of large language models.

★ 1,616
TIGER-AI-Lab/ClawBench

ClawBench - Benchmark for AI Agents

Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!

★ 1,301 · Python
Jwuthri/Tracely-ai

TRACELY-AI - AUTOMATED AGENT TESTING

Tracely-ai automatically turns AI agent failures into regression tests that block PRs and reproduces them in CI for free.

★ 1,299 · Python
NVIDIA/SkillEvaluator

SKILLEVALUATOR: AI AGENT EVALUATION

A multi-level framework for evaluating AI agent skills: qualitative gates, semantic overlap, synthetic data, and live testing.

★ 1,286 · Python
xinxuxin/keystone-bench

Clinical Evidence Benchmark

Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.

★ 1,131 · Python
BlazeUp-AI/Observal

Observal - AI Agent Oversight

Observal is a platform for monitoring, evaluating, and debugging AI agents with human-in-the-loop feedback. It is essential for building reliable production pipelines.

★ 1,118 · Python
agentscope-ai/OpenJudge

OPENJUDGE - NEW AI EVALUATOR

Unified framework for evaluating the quality of AI agents. 671⭐ Python.

★ 1,105 · Python
NVIDIA-NeMo/Gym

GYM - Model Evaluation

Helps evaluate and improve LLMs through environments and agents with feedback and testing.

★ 932 · Python
AgentSafeLabs/safelabs-eval

PROTECTION OF AI AGENTS FROM THREATS

safelabs-eval — framework for red‑teaming and evaluation of AI agent protection based on the ASI01–ASI10 taxonomy inspired by OWASP. Written in Python, it supports CrewAI

★ 829 · Python
chrisliu298/awesome-llm-unlearning

Repo for LLM Unlearning

A collection of papers, code, and resources on machine 'unlearning' in large language models. Helps researchers remove knowledge from LLMs.

★ 638
HowieHwong/TrustLLM

TRUSTLLM: EVALUATION OF LLM TRUSTWORTHINESS

TrustLLM — a toolkit for assessing LLM trustworthiness, presented at ICML 2024. It includes benchmark suites, metrics, and datasets for measuring hallucinations, bias, an

★ 636 · Python
DocAILab/XRAG

XRAG: EVALUATION OF RAG MODULES

XRAG evaluates core components of RAG systems: indexing, retrieval, and generation. Using standard datasets it compares different module implementations, revealing their

★ 612 · Python
MMMU-Benchmark/MMMU

MMMU: Multimodal Benchmark

Code evaluation for MMMU benchmark measuring multidisciplinary multimodal understanding and reasoning for expert AGI.

★ 601 · Python
run-llama/ParseBench

PARSEBENCH: AI AGENT EVALUATION

ParseBench is a benchmark for evaluating the quality of document parsing by AI agents. It allows comparing models on real-world datasets.

★ 472 · Python
CodeSoul-co/Hypha

HYPHA Agent Framework

TypeScript framework for production‑grade LLM agents. Enables rapid creation and management of agents via a flexible harness architecture.

★ 452 · TypeScript
zaxiu88888/issue-rag

RAG System for GitHub Issues

RAG over GitHub issues with hand-reviewed eval set and judge audit. Reports confidence intervals, not just point metrics.

★ 419 · Python
SAILResearch/awesome-ai-leaderboard

Awesome AI Leaderboard

Curated list of resources with leaderboards for the AI domain. Benchmarks, metrics, and model comparisons.

★ 384
microsoft/genaiops-promptflow-template

TEMPLATE GENAIOPS PROMPT FLOW

Template GenAIOps with Prompt Flow simplifies creation of LLM applications through centralized hosting, lifecycle management, and experiments with prompt variants and hyp

★ 370 · Python
microsoft/rag-experiment-accelerator

RAG EXPERIMENT ACCELERATOR

Tool for rapid RAG experiments with Azure Cognitive Search. Automates chunking, embeddings, and answer quality evaluation.

★ 319 · Python
rentruewang/bocoel

BOCOEL: BAYESIAN OPTIMIZATION FOR LLM

bocoel — a Bayesian Optimization framework for quick LLM evaluation: 10× faster, just a few lines of code. It replaces costly benchmarks with precise search over model hy

★ 288 · Python
InternScience/ResearchClawBench

RESEARCHCLAWBENCH - IMPROVING AI RESEARCH EFFECTIVENESS

Evaluation of automated AI agents: from rediscovery to new discoveries. Runs in Jupyter Notebook.

★ 280 · Jupyter Notebook
lechmazur/nyt-connections

NYT Connections: LLM Benchmark

A benchmark based on 759 NYT Connections puzzles with additional trick words for evaluating LLMs.

★ 250 · Python
arklexai/arksim

arksim - DETECT AGENT ERRORS

arksim helps developers find errors in AI agents before they reach real users, saving time and improving reliability.

★ 236 · Python
edonadei/caliper

EVALUATE AGENTS WITH CALIPER

Caliper runs your real agent with skills, MCPs, and rules and without them, showing what actually helps and how many tokens it costs. Supports Claude Code, Codex, Pi, and

★ 235 · Python
alopatenko/LLMEvaluation

FULL GUIDE TO LLM EVALUATION

LLMEvaluation provides a full guide to LLM evaluation methods aiding selection of optimal techniques for specific use-cases. It contains a metrics taxonomy, step‑by‑ste

★ 208 · HTML
h9-tec/ai-roadmap-2026

Beginner AI Learning Path

A step-by-step AI roadmap covering foundations to production with free resources and hands-on mini-projects for each layer.

★ 194
AMAP-ML/LoopArena

Model Benchmarking Platform

The project provides tools for comparing machine models as real-time controllers of engineering loops, measuring their latency, stability, and resource consumption.

★ 155 · Python
Jwuthri/SelfJev

Open Decision Model API

Provides an open decision model exposing Jev's API for typed answers (yes/no, choice, score, multi) with probabilities from forward passes, using Qwen3.5-4B fine‑tuned wi

★ 25 · Python

New projects on this topic in Telegram

Telegram →
Evaluation — 46 open-source GitHub projects | REDDYX AI