Evals: open-source projects on GitHub
20 repositories on Evals, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Mastra AI framework
Mastra is a TypeScript framework for building AI applications and agents, created by the Gatsby team.
★ 23,853 · TypeScriptAI observability platform
Platform for monitoring and evaluating AI models: quality monitoring, data drift detection, and interactive visualization.
★ 9,652 · PythonAGENTOPS: AI AGENT MONITORING
Python SDK for monitoring AI agents, tracking LLM costs, and benchmarking. Support for all popular frameworks.
★ 5,960 · PythonBuild AI Systems
Build, evaluate, and optimize AI systems. Includes evals, RAG, agents, fine‑tuning, synthetic data generation, dataset management, MCP, and more.
★ 4,848 · PythonLOGFIRE: LLM OBSERVABILITY
Platform for observing production LLMs and agentic systems: tracing, metrics, and real-time alerts.
★ 4,290 · PythonAI AGENT SKILLS PLATFORM
Platform lets you create, test and publish skills for AI agents via a user‑defined marketplace. Manage their lifecycle: discovery, updates, rollback and quarantine, and a
★ 2,414 · PythonWAKU-AGENT — PERSONAL AI AGENT
A personal AI agent in Python: local launch, memory, task loop and evaluation — everything ready in just a few minutes of reading the code.
★ 2,173 · PythonInspector - MCP Server Debug
Platform for testing, inspecting, and debugging MCP servers, MCP apps, and ChatGPT apps. 1968 ⭐
★ 1,968 · TypeScriptPAPERTHIN: SKILLS FOR RECURSIVE RSI
OpenAI hackathon winner's project on recursive agent self-improvement. Unpacks NVIDIA's zero-touch proof into engineering practice. Ready skillset for autonomous coding a
★ 1,445 · JavaScriptTRACELY-AI - AUTOMATED AGENT TESTING
Tracely-ai automatically turns AI agent failures into regression tests that block PRs and reproduces them in CI for free.
★ 1,299 · PythonRAGLITE - FAST RAG ON DUCKDB
Simple tool for building RAG pipelines with DuckDB or PostgreSQL in Python.
★ 1,212 · PythonGUIDE TO AI SYSTEMS
A practical guide to designing production-ready AI systems and evaluating their quality for engineers.
★ 1,132AI Agent Evaluation Toolkit
A curated library of top resources for building and evaluating AI agents, including articles, blogs, tools, and benchmarks.
★ 1,039AI Application Evaluation
Open platform for evaluating and improving LLM and AI agents with tracing, simulation, and dataset support. Runs locally.
★ 971 · PythonSKILLFORGE — RELIABLE SKILL GENERATOR
SkillForge creates verified skills for Claude Code and Codex with baseline testing and regression assessment.
★ 918 · PythonOVERMIND: PLATFORM FOR AI AGENTS
Overmind — platform for the continuous improvement of AI agents: automated evaluation, fine‑tuning and integration with LLM operations. It allows tracking metrics, runnin
★ 759 · PythonAI Engineer Notebooks
A practical collection of framework-free Jupyter notebooks for learning AI engineer skills: model APIs, structured output, tool calling, RAG, evaluation, agents, fine-tun
★ 749 · Jupyter NotebookDecision Model Harness Framework
Framework for building harnesses that guide models through stepwise decision-making by presenting curated action choices until task completion.
★ 444 · TypeScriptEnterprise AI Agent Design Course
17 bilingual lessons teaching enterprise AI agent system design with hands-on exercises covering tools, RAG, security, evals, and distributed execution.
★ 316 · PythonEVALUATE AGENTS WITH CALIPER
Caliper runs your real agent with skills, MCPs, and rules and without them, showing what actually helps and how many tokens it costs. Supports Claude Code, Codex, Pi, and
★ 235 · PythonNew projects on this topic in Telegram
Telegram →