R_REDDYX.XYZ

AI Safety: open-source projects on GitHub

33 repositories on AI Safety, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094AI Agents 1319MCP 1601LLM Security 36Agentic AI 378Prompt Injection 28Agent Security 25AI Security 56Security 125AI Governance 21LLM Evaluation 45Claude Code 1595Langchain 196Owasp 13
microsoft/PyRIT

OPEN SOURCE RED TEAMING TOOLKIT FOR AI

Open-source toolkit from Microsoft for finding security and safety vulnerabilities in generative AI systems. Automates scanning, adaptive multi-turn attacks, and manual t

★ 4,966 · Python
wuyoscar/AISafetyHot

AI SAFETY HOT DIGEST

AI Safety HOT collects daily AI safety news, papers & incidents, jailbreaks, prompt injection, alignment and evals. On the platform — website, daily digest, RSS and MCP.

★ 2,125 · TeX
cvs-health/uqlm

UQLM: Detecting LLM Hallucinations

Python package for detecting language model hallucinations based on uncertainty quantification.

★ 1,217 · Python
xinxuxin/keystone-bench

Clinical Evidence Benchmark

Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.

★ 1,131 · Python
superwesleyhys-ux/factcircuit

FactCircuit Fact-Check Platform

FactCircuit is an auditable loop for checking claims and evidence with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.

★ 927 · Python
AgentSafeLabs/safelabs-eval

PROTECTION OF AI AGENTS FROM THREATS

safelabs-eval — framework for red‑teaming and evaluation of AI agent protection based on the ASI01–ASI10 taxonomy inspired by OWASP. Written in Python, it supports CrewAI

★ 829 · Python
Cygnux-Labs/Tracekit

TRACEKIT: AUDIT AI AGENT ACTIONS

Tracekit provides tamper-evident tracing of AI agent actions with policy gates and cross-checks. See what the agent did and why — critical for safety and compliance.

★ 726 · Python
decionis/agent-safe-pipeline

agent-safe-pipeline - SAFE AI AGENT IN PRODUCTION

Architecture for AI agents: fixed intent capture, independent Decionis policy check, and safe execution via a one‑time grant.

★ 672 · TypeScript
chrisliu298/awesome-llm-unlearning

Repo for LLM Unlearning

A collection of papers, code, and resources on machine 'unlearning' in large language models. Helps researchers remove knowledge from LLMs.

★ 638
strands-agents/box

SECURE SANDBOX FOR AI AGENTS

Box isolates AI agents, using OS isolation and default‑deny Dogwood policies. It injects credentials from a secure store without exposing them to the agent, and restricts

★ 546 · Rust
cordum-io/cordum

CORDUM - Manage AI Agents

Cordum is an open control plane for AI agents: pre‑execution policy, approval gates, and audit trail. Compatible with LangChain, CrewAI, MCP.

★ 485 · Go
Prysai/Prysai-LLM-Playbook

PRYSAI PLAYBOOK: COMPLETE LLM GUIDE

LLM playbook on practice: 8 locales, unified core, Codex track and adapters for ChatGPT, Claude Code, Gemini, DeepSeek, Grok.

★ 433 · Python
ZhangJinHaHaHa/AgentLens

AGENTLENS: VERIFY AGENT BEFORE HIRE

Decentralized audit protocol and marketplace for AI agents with TEE attestation, MDDRM reputation registry, and SGX/DCAP-based sandbox.

★ 407 · TypeScript
rhesis-ai/rhesis

COLLABORATION LAYER FOR AI AGENTS

Rhesis — platform for testing, simulating, and tracing AI agents. Experts annotate behavior, engineers improve agent based on found issues. Closes feedback loop between t

★ 404 · Python
schmitech/orbit

ORBIT SELF-HOSTED AI

Orbit — self-hosted platform for private RAG and multimodal AI applications in Python.

★ 364 · Python
lalipa2003-arch/JailbreakHub

HUB FOR LLM JAILBREAKING

JailbreakHub collects prompts and methods to bypass LLM safety. The project has 245★ and attack classification. Useful for vulnerability research.

★ 363 · Roff
Justin0504/Aegis

AEGIS: AI AGENT PROTECTION

Aegis provides runtime policies for AI agents with a cryptographic audit trail, human approval, and a kill‑switch — without modifying code.

★ 361 · TypeScript
agentcontrol/agent-control

AGENT-CONTROL - Centralized Agent Management

A control plan for managing agent behavior in production. Configurable, extensible, and ready for operation.

★ 337 · Python
LLMSecurity/awesome-agent-skills-security

PROTECTIVE LIST OF AGENT SKILLS

Collected materials on protecting AI‑agent skills: attacks, defenses, frameworks, and benchmarks. The list helps researchers and developers assess tool‑use vulnerabilitie

★ 287
Keesan12/martin-loop

MARTIN LOOP: AUTONOMOUS CODE AGENTS

Launch AI agents to write code without constant supervision. Martin Loop keeps tasks within strict boundaries, checks them and provides accountability from start to finis

★ 286 · TypeScript
Junkz3/wrench-board

WRENCH-BOARD - AI Technician for Micro-soldering

The wrench-board project provides an AI assistant for micro-welding, helping workshops of any skill level perform complex repairs.

★ 266 · Python
cvs-health/langfair

LANGFAR LLM Bias

A Python library for evaluating LLM bias and fairness at the level of specific use cases.

★ 261 · Python
yzhao062/anywhere-agents

ANYWHERE-AGENTS: ONE CONFIG

The project provides a unified configuration for all AI agents: portable, efficient, and safe operation with protection against dangerous commands.

★ 261 · Python
lacs-project/sysknife

AI LINUX ADMINISTRATOR IN RUST

Sysknife is an agent for Linux administration. The model picks the action; the daemon builds the command. All actions require approval and are logged with Ed25519 signatu

★ 261 · Rust
PerryLink/dsh-auto-review

DS AUTO-REVIEW:AI SYSTEM

dsh-auto-review offers a read-only subagent that returns allow/deny with reasons for DeepSeek Harness approval requests. Verdict is fail-closed by default; the chain appr

★ 249 · TypeScript
DobermanCore/Doberman-Core

DOBERMAN-CORE — PROTECTION FOR AI AGENTS

Doberman-Core intercepts every input, output, and tool call during AI-agent operation, blocking unsafe actions. It works as middleware, checking authorization and applyin

★ 243 · Python
humanbound/humanbound

RED-TEAMING AI AGENTS IN PYTHON

Humanbound — adversarial testing engine, SDK, and CLI for AI agents. Runs locally or via Humanbound Platform. 218 stars on GitHub.

★ 239 · Python
KalarisLabs/claude-systems-prompts

ALL PROMPTS CLAUDE IN ONE

All official Claude prompts in one repository: verbatim, dated and source-linked, from Claude Fable 5.2 to Haiku 3. 23 models from Anthropic's release notes, plus analysi

★ 232 · Python
guardana/guardana

GUARDANA AI MODEL PROTECTION

Guardana scans model artifacts, live endpoints, MCP servers, and agent traces for security threats. It generates reproducible evidence for release decisions.

★ 225 · Python
OWASP/www-project-agent-memory-guard

DEFENSE MEMORY AI-AGENTS

Provides runtime defense against memory AI-agent poisoning (OWASP ASI06). Drop-in for LangChain, OpenAI Agents, AutoGen, CrewAI and mem0.

★ 203 · Python
node9-ai/node9-proxy

NODE9-PROXY: SHIELD FOR AI AGENTS

node9-proxy provides deterministic management and auditing of autonomous AI agents, ensuring safe code execution.

★ 201 · TypeScript
permit0-ai/permit0

PERMIT0 — AI Agent Error Protection

Authorize AI-agent actions before execution. Deterministic policies, simple integration from day one.

★ 192 · Rust
Anil-matcha/awesome-gpt-6-astra

GPT-6 ASTRA: PROMPTS INTEGRATIONS

Evidence-backed use cases, prompts, integrations, evaluations, and safety notes for OpenAI GPT-6 Astra.

★ 17 · Python

New projects on this topic in Telegram

Telegram →
AI Safety — 33 open-source GitHub projects | REDDYX AI