R_REDDYX.XYZ

LLM Evaluation: open-source projects on GitHub

45 repositories on LLM Evaluation, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094AI Agents 1319LLMOps 82Evaluation 46LLM Observability 24Langchain 196Agentic AI 378MCP 1601Benchmark 54Large Language Models 154Observability 92Openai 690Prompt Engineering 175RAG 533
langfuse/langfuse

Langfuse: LLM Platform

Open platform for LLM: observability, metrics, prompt management, testing, and integration with popular tools.

★ 27,378 · TypeScript
mlflow/mlflow

MLflow — AI Platform

Open platform for AI engineering. Allows teams to debug, evaluate, and optimize AI applications while managing access and costs.

★ 26,029 · Python
promptfoo/promptfoo

Testing LLM Models

Tool for testing prompts, agents, and RAG systems. Supports comparative analysis of GPT, Claude, Gemini, Llama, and other models.

★ 21,294 · TypeScript
comet-ml/opik

LLM Debugging and Monitoring

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

★ 19,332 · Python
Arize-ai/phoenix

AI observability platform

Platform for monitoring and evaluating AI models: quality monitoring, data drift detection, and interactive visualization.

★ 9,652 · Python
oumi-ai/oumi

OUMI EASY LLM TUNING

OUMI lets you quickly fine‑tune (SFT/RL), evaluate, and deploy Qwen, Gemma, and any open agentic LLM/VLM without extra complexity.

★ 9,433 · Python
Helicone/helicone

HELICONE LLM OBSERVE

Open source platform for LLM observability: monitoring, evaluation, and experiments in one line of code. Supports TypeScript and already has 5.8k stars.

★ 5,805 · TypeScript
Giskard-AI/giskard-oss

GISKARD-OSS - LLM AGENT EVALUATION

Library for evaluating and testing LLM agents with open-source code.

★ 5,417 · Python
Agenta-AI/agenta

AGENTA: AI AGENTS IN CHAT

Agenta is an open platform for creating AI agents via chat, sharing them, and running them on schedule or in response to app events.

★ 4,911 · TypeScript
Marker-Inc-Korea/AutoRAG

AutoRAG: RAG System Automation

An open-source framework for automatic evaluation and optimization of RAG pipelines in an AutoML style.

★ 4,804 · Python
EvolvingLMMs-Lab/lmms-eval

LMMS-EVAL - ONE-STOP MODALITY EVAL

Unified toolkit for evaluating multimodal models on text, image, video, and audio tasks. 4169 stars on GitHub.

★ 4,169 · Python
Tencent/AI-Infra-Guard

AI SYSTEM SECURITY

A platform for testing the security of AI ecosystems: scanning infrastructure, MCP, agents, and evaluating jailbreak attacks.

★ 3,790 · Python
hemansnation/AI-Engineer-Headquarters

AI‑ENGINEER HQ: SCIENCE CORE

The project contains Jupyter notebooks with methods for training models, RAG pipelines on LangChain and data structure algorithms for AI agents. It allows quickly creatin

★ 3,780 · Jupyter Notebook
genieincodebottle/generative-ai

Generative AI Roadmap Projects

A collection of generative AI resources: roadmap, projects, interview and coding preparation.

★ 2,670 · Jupyter Notebook
TIGER-AI-Lab/ClawBench

ClawBench - Benchmark for AI Agents

Open-source benchmark for evaluating browser AI agents on everyday tasks. Already earned ⭐ 385!

★ 1,301 · Python
Jwuthri/Tracely-ai

TRACELY-AI - AUTOMATED AGENT TESTING

Tracely-ai automatically turns AI agent failures into regression tests that block PRs and reproduces them in CI for free.

★ 1,299 · Python
cvs-health/uqlm

UQLM: Detecting LLM Hallucinations

Python package for detecting language model hallucinations based on uncertainty quantification.

★ 1,217 · Python
xinxuxin/keystone-bench

Clinical Evidence Benchmark

Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.

★ 1,131 · Python
BlazeUp-AI/Observal

Observal - AI Agent Oversight

Observal is a platform for monitoring, evaluating, and debugging AI agents with human-in-the-loop feedback. It is essential for building reliable production pipelines.

★ 1,118 · Python
benchflow-ai/awesome-evals

AI Agent Evaluation Toolkit

A curated library of top resources for building and evaluating AI agents, including articles, blogs, tools, and benchmarks.

★ 1,039
JudgmentLabs/judgeval

JUDGEVAL STACK FOR AGENTS

Platform for continuous improvement of AI agents: collecting environment data and evaluations for monitoring and growth.

★ 1,036 · Python
jwuthri-cloud/Tracely-ai

TRACE-NATIVE CI/CD FOR AI AGENTS

Tracely-ai grades production agent traces, clusters failures, and turns them into regression tests. Blocks PRs with behavior regression. Self-hosted on your infrastructur

★ 948 · Python
superwesleyhys-ux/factcircuit

FactCircuit Fact-Check Platform

FactCircuit is an auditable loop for checking claims and evidence with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.

★ 927 · Python
overmind-core/overmind

OVERMIND: PLATFORM FOR AI AGENTS

Overmind — platform for the continuous improvement of AI agents: automated evaluation, fine‑tuning and integration with LLM operations. It allows tracking metrics, runnin

★ 759 · Python
Continuum-AI-Corp/OrcaReplay

ORCA REPLAY: TIME TRAVEL

OrcaReplay records any launch of an AI agent and allows you to replay it step by step. You can create forks for experiments and debug the state at each stage, using any m

★ 727 · TypeScript
EverMind-AI/SkillCorpus

Skill corpus for agents

The project collects SKILL.md files into a single searchable skill catalog for AI agents, enabling fast search and automatic task-matching evaluation.

★ 703 · Python
ValueByte-AI/Awesome-LLM-in-Social-Science

LLM News in Social Science

A collection of outstanding works where language models are applied in social sciences. Perfect for researchers and students.

★ 653
wanghao-SunSky/cs329z-chinese

Stanford AI Agents Course (Chinese)

Chinese translation of Stanford CS329Z Engineering AI Agents course with 19 lectures and 4 hands-on projects covering LLMs, RAG, tool calling, memory, multi-agent systems

★ 599 · HTML
relari-ai/continuous-eval

CONTINUOUS-EVAL - LLM EVALUATION ON DATA

Platform for automatic evaluation of LLM applications based on real data. Enables measuring quality and improving models without manual effort.

★ 519 · Python
OmniJev/awesome-jev-gallery

AWESOME JEV GALLERY: PAPERS AND REPRODUCTIONS

Awesome Jev Gallery — collection of articles, open reproductions and independent evaluations of System One and Jev models. Includes links to code, calibration metrics and

★ 493 · JavaScript
scaleapi/agentenv-framework

Agent Environment Framework

Framework for building realistic reinforcement learning environments with collaboration tools for researchers and engineers.

★ 438 · Python
rhesis-ai/rhesis

COLLABORATION LAYER FOR AI AGENTS

Rhesis — platform for testing, simulating, and tracing AI agents. Experts annotate behavior, engineers improve agent based on found issues. Closes feedback loop between t

★ 404 · Python
ukanwat/overtime

OVERTIME: AGENT FOR GAME DEVELOPMENT

Project overtime lets you give a brief technical task to an AI agent, after which the agent autonomously performs work in subsequent sessions. In the demo, it built an op

★ 402 · TypeScript
darkrishabh/agent-skills-eval

AI Agent Skills Evaluator

An automated framework for testing and evaluating the performance of AI agents in the agentskills.io format.

★ 393 · TypeScript
PetroIvaniuk/llms-tools

CATALOG OF LLM TOOLS

llms-tools is a curated list of over 200 tools and projects for working with large language models. The project helps quickly find the right solutions for AI chatbots, da

★ 331
Future-Proof-DS/interactive-ai-ml-tools

Interactive MLOps Learning Tools

Free interactive tools for learning MLOps, ML system design, and agentic analytics with simulators and system maps.

★ 267
cvs-health/langfair

LANGFAR LLM Bias

A Python library for evaluating LLM bias and fairness at the level of specific use cases.

★ 261 · Python
LeoYeAI/myclaw-bench

MYCLAW BENCH: AI AGENT TEST

Test AI agents on 45 tasks across four difficulty levels. Powered by MyClaw.ai.

★ 222 · Python
jackson-cmd/risk-aware-trading-agents

AI AGENTS FOR STOCK TRADING WITH NEWS BACKTESTING

A set of AI agents for trading US stocks with honest news backtesting: headlines replay at the minute of publication, orders fill at the next open. Includes a public LLM

★ 222 · Python
alopatenko/LLMEvaluation

FULL GUIDE TO LLM EVALUATION

LLMEvaluation provides a full guide to LLM evaluation methods aiding selection of optimal techniques for specific use-cases. It contains a metrics taxonomy, step‑by‑ste

★ 208 · HTML
phuryn/bug-hunt-bench

FIND 105 BUGS

Bug Hunt Bench — benchmark 105 real bugs in 2 repositories. AI models (GPT-6, Claude, Grok, Gemini, DeepSeek) find and fix them in CLI, evaluated blind on leaderboard, ou

★ 207 · JavaScript
jsdhwfmax/EvalForge

EVALFORGE — LLM EVALUATION IN CI

EvalForge is a tool for LLM evaluation in CI/CD. Provides evaluator-neutral evidence, baseline regression gates, and JSON, JUnit, SARIF reports. Integrates with GitHub Ac

★ 194 · Python
cookiespiggy/agentic-rl

Agentic RL Tutorial & Benchmark

33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.

★ 146 · Python
juanjuandog/FinSight-AI

FinSight-AI — Finance with AI

AI agent for stock analysis: fault-tolerant pipelines, RAG via pgvector, report versioning, and full evidence traceability.

★ 118 · Java
jwuthri-cloud/SelfJev

Self-Hosted Jev Decision Model

Run the selfjev-4b decision model locally with a Jev-compatible API, Python SDK, and tools for fine-tuning and RLCD training.

★ 34 · Python

New projects on this topic in Telegram

Telegram →
LLM Evaluation — 45 open-source GitHub projects | REDDYX AI