LLM Evaluation: open-source projects on GitHub
45 repositories on LLM Evaluation, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Langfuse: LLM Platform
Open platform for LLM: observability, metrics, prompt management, testing, and integration with popular tools.
★ 27,378 · TypeScriptMLflow — AI Platform
Open platform for AI engineering. Allows teams to debug, evaluate, and optimize AI applications while managing access and costs.
★ 26,029 · PythonTesting LLM Models
Tool for testing prompts, agents, and RAG systems. Supports comparative analysis of GPT, Claude, Gemini, Llama, and other models.
★ 21,294 · TypeScriptLLM Debugging and Monitoring
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
★ 19,332 · PythonAI observability platform
Platform for monitoring and evaluating AI models: quality monitoring, data drift detection, and interactive visualization.
★ 9,652 · PythonOUMI EASY LLM TUNING
OUMI lets you quickly fine‑tune (SFT/RL), evaluate, and deploy Qwen, Gemma, and any open agentic LLM/VLM without extra complexity.
★ 9,433 · PythonHELICONE LLM OBSERVE
Open source platform for LLM observability: monitoring, evaluation, and experiments in one line of code. Supports TypeScript and already has 5.8k stars.
★ 5,805 · TypeScriptGISKARD-OSS - LLM AGENT EVALUATION
Library for evaluating and testing LLM agents with open-source code.
★ 5,417 · PythonAGENTA: AI AGENTS IN CHAT
Agenta is an open platform for creating AI agents via chat, sharing them, and running them on schedule or in response to app events.
★ 4,911 · TypeScriptAutoRAG: RAG System Automation
An open-source framework for automatic evaluation and optimization of RAG pipelines in an AutoML style.
★ 4,804 · PythonLMMS-EVAL - ONE-STOP MODALITY EVAL
Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.
★ 4,169 · PythonAI SYSTEM SECURITY
A platform for testing the security of AI ecosystems: scanning infrastructure, MCP, agents, and evaluating jailbreak attacks.
★ 3,790 · PythonAI‑ENGINEER HQ: SCIENCE CORE
The project contains Jupyter notebooks with methods for training models, RAG pipelines on LangChain and data structure algorithms for AI agents. It allows quickly creatin
★ 3,780 · Jupyter NotebookGenerative AI Roadmap Projects
A collection of generative AI resources: roadmap, projects, interview and coding preparation.
★ 2,670 · Jupyter NotebookClawBench - Benchmark for AI Agents
Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!
★ 1,301 · PythonTRACELY-AI - AUTOMATED AGENT TESTING
Tracely-ai automatically turns AI agent failures into regression tests that block PRs and reproduces them in CI for free.
★ 1,299 · PythonUQLM: Detecting LLM Hallucinations
Python package for detecting language model hallucinations based on uncertainty quantification.
★ 1,217 · PythonClinical Evidence Benchmark
Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.
★ 1,131 · PythonObserval - AI Agent Oversight
Observal is a platform for monitoring, evaluating, and debugging AI agents with human-in-the-loop feedback. It is essential for building reliable production pipelines.
★ 1,118 · PythonAI Agent Evaluation Toolkit
A curated library of top resources for building and evaluating AI agents, including articles, blogs, tools, and benchmarks.
★ 1,039JUDGEVAL STACK FOR AGENTS
Platform for continuous improvement of AI agents: collecting environment data and evaluations for monitoring and growth.
★ 1,036 · PythonTRACE-NATIVE CI/CD FOR AI AGENTS
Tracely-ai grades production agent traces, clusters failures, and turns them into regression tests. Blocks PRs with behavior regression. Self-hosted on your infrastructur
★ 948 · PythonFactCircuit Fact-Check Platform
FactCircuit is an auditable loop for checking claims and evidence with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
★ 927 · PythonOVERMIND: PLATFORM FOR AI AGENTS
Overmind — platform for the continuous improvement of AI agents: automated evaluation, fine‑tuning and integration with LLM operations. It allows tracking metrics, runnin
★ 759 · PythonORCA REPLAY: TIME TRAVEL
OrcaReplay records any launch of an AI agent and allows you to replay it step by step. You can create forks for experiments and debug the state at each stage, using any m
★ 727 · TypeScriptSkill corpus for agents
The project collects SKILL.md files into a single searchable skill catalog for AI agents, enabling fast search and automatic task-matching evaluation.
★ 703 · PythonLLM News in Social Science
A collection of outstanding works where language models are applied in social sciences. Perfect for researchers and students.
★ 653Stanford AI Agents Course (Chinese)
Chinese translation of Stanford CS329Z Engineering AI Agents course with 19 lectures and 4 hands-on projects covering LLMs, RAG, tool calling, memory, multi-agent systems
★ 599 · HTMLCONTINUOUS-EVAL - LLM EVALUATION ON DATA
Platform for automatic evaluation of LLM applications based on real data. Enables measuring quality and improving models without manual effort.
★ 519 · PythonAWESOME JEV GALLERY: PAPERS AND REPRODUCTIONS
Awesome Jev Gallery — collection of articles, open reproductions and independent evaluations of System One and Jev models. Includes links to code, calibration metrics and
★ 493 · JavaScriptAgent Environment Framework
Framework for building realistic reinforcement learning environments with collaboration tools for researchers and engineers.
★ 438 · PythonCOLLABORATION LAYER FOR AI AGENTS
Rhesis — platform for testing, simulating, and tracing AI agents. Experts annotate behavior, engineers improve agent based on found issues. Closes feedback loop between t
★ 404 · PythonOVERTIME: AGENT FOR GAME DEVELOPMENT
Project overtime lets you give a brief technical task to an AI agent, after which the agent autonomously performs work in subsequent sessions. In the demo, it built an op
★ 402 · TypeScriptAI Agent Skills Evaluator
An automated framework for testing and evaluating the performance of AI agents in the agentskills.io format.
★ 393 · TypeScriptCATALOG OF LLM TOOLS
llms-tools is a curated list of over 200 tools and projects for working with large language models. The project helps quickly find the right solutions for AI chatbots, da
★ 331Interactive MLOps Learning Tools
Free interactive tools for learning MLOps, ML system design, and agentic analytics with simulators and system maps.
★ 267LANGFAR LLM Bias
A Python library for evaluating LLM bias and fairness at the level of specific use cases.
★ 261 · PythonMYCLAW BENCH: AI AGENT TEST
Test AI agents on 45 tasks across four difficulty levels. Powered by MyClaw.ai.
★ 222 · PythonAI AGENTS FOR STOCK TRADING WITH NEWS BACKTESTING
A set of AI agents for trading US stocks with honest news backtesting: headlines replay at the minute of publication, orders fill at the next open. Includes a public LLM
★ 222 · PythonFULL GUIDE TO LLM EVALUATION
LLMEvaluation provides a full guide to LLM evaluation methods aiding selection of optimal techniques for specific use-cases. It contains a metrics taxonomy, step‑by‑ste
★ 208 · HTMLFIND 105 BUGS
Bug Hunt Bench — benchmark 105 real bugs in 2 repositories. AI models (GPT-6, Claude, Grok, Gemini, DeepSeek) find and fix them in CLI, evaluated blind on leaderboard, ou
★ 207 · JavaScriptEVALFORGE — LLM EVALUATION IN CI
EvalForge is a tool for LLM evaluation in CI/CD. Provides evaluator-neutral evidence, baseline regression gates, and JSON, JUnit, SARIF reports. Integrates with GitHub Ac
★ 194 · PythonAgentic RL Tutorial & Benchmark
33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.
★ 146 · PythonFinSight-AI — Finance with AI
AI agent for stock analysis: fault-tolerant pipelines, RAG via pgvector, report versioning, and full evidence traceability.
★ 118 · JavaSelf-Hosted Jev Decision Model
Run the selfjev-4b decision model locally with a Jev-compatible API, Python SDK, and tools for fine-tuning and RLCD training.
★ 34 · PythonNew projects on this topic in Telegram
Telegram →