AI Safety: open-source projects on GitHub
33 repositories on AI Safety, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
OPEN SOURCE RED TEAMING TOOLKIT FOR AI
Open-source toolkit from Microsoft for finding security and safety vulnerabilities in generative AI systems. Automates scanning, adaptive multi-turn attacks, and manual t
★ 4,966 · PythonAI SAFETY HOT DIGEST
AI Safety HOT collects daily AI safety news, papers & incidents, jailbreaks, prompt injection, alignment and evals. On the platform — website, daily digest, RSS and MCP.
★ 2,125 · TeXUQLM: Detecting LLM Hallucinations
Python package for detecting language model hallucinations based on uncertainty quantification.
★ 1,217 · PythonClinical Evidence Benchmark
Benchmarks assistant responses by extracting the single fact underlying each answer and testing if the action follows that evidence.
★ 1,131 · PythonFactCircuit Fact-Check Platform
FactCircuit is an auditable loop for checking claims and evidence with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
★ 927 · PythonPROTECTION OF AI AGENTS FROM THREATS
safelabs-eval — framework for red‑teaming and evaluation of AI agent protection based on the ASI01–ASI10 taxonomy inspired by OWASP. Written in Python, it supports CrewAI
★ 829 · PythonTRACEKIT: AUDIT AI AGENT ACTIONS
Tracekit provides tamper-evident tracing of AI agent actions with policy gates and cross-checks. See what the agent did and why — critical for safety and compliance.
★ 726 · Pythonagent-safe-pipeline - SAFE AI AGENT IN PRODUCTION
Architecture for AI agents: fixed intent capture, independent Decionis policy check, and safe execution via a one‑time grant.
★ 672 · TypeScriptRepo for LLM Unlearning
A collection of papers, code, and resources on machine 'unlearning' in large language models. Helps researchers remove knowledge from LLMs.
★ 638SECURE SANDBOX FOR AI AGENTS
Box isolates AI agents, using OS isolation and default‑deny Dogwood policies. It injects credentials from a secure store without exposing them to the agent, and restricts
★ 546 · RustCORDUM - Manage AI Agents
Cordum is an open control plane for AI agents: pre‑execution policy, approval gates, and audit trail. Compatible with LangChain, CrewAI, MCP.
★ 485 · GoPRYSAI PLAYBOOK: COMPLETE LLM GUIDE
LLM playbook on practice: 8 locales, unified core, Codex track and adapters for ChatGPT, Claude Code, Gemini, DeepSeek, Grok.
★ 433 · PythonAGENTLENS: VERIFY AGENT BEFORE HIRE
Decentralized audit protocol and marketplace for AI agents with TEE attestation, MDDRM reputation registry, and SGX/DCAP-based sandbox.
★ 407 · TypeScriptCOLLABORATION LAYER FOR AI AGENTS
Rhesis — platform for testing, simulating, and tracing AI agents. Experts annotate behavior, engineers improve agent based on found issues. Closes feedback loop between t
★ 404 · PythonORBIT SELF-HOSTED AI
Orbit — self-hosted platform for private RAG and multimodal AI applications in Python.
★ 364 · PythonHUB FOR LLM JAILBREAKING
JailbreakHub collects prompts and methods to bypass LLM safety. The project has 245★ and attack classification. Useful for vulnerability research.
★ 363 · RoffAEGIS: AI AGENT PROTECTION
Aegis provides runtime policies for AI agents with a cryptographic audit trail, human approval, and a kill‑switch — without modifying code.
★ 361 · TypeScriptAGENT-CONTROL - Centralized Agent Management
A control plan for managing agent behavior in production. Configurable, extensible, and ready for operation.
★ 337 · PythonPROTECTIVE LIST OF AGENT SKILLS
Collected materials on protecting AI‑agent skills: attacks, defenses, frameworks, and benchmarks. The list helps researchers and developers assess tool‑use vulnerabilitie
★ 287MARTIN LOOP: AUTONOMOUS CODE AGENTS
Launch AI agents to write code without constant supervision. Martin Loop keeps tasks within strict boundaries, checks them and provides accountability from start to finis
★ 286 · TypeScriptWRENCH-BOARD - AI Technician for Micro-soldering
The wrench-board project provides an AI assistant for micro-welding, helping workshops of any skill level perform complex repairs.
★ 266 · PythonLANGFAR LLM Bias
A Python library for evaluating LLM bias and fairness at the level of specific use cases.
★ 261 · PythonANYWHERE-AGENTS: ONE CONFIG
The project provides a unified configuration for all AI agents: portable, efficient, and safe operation with protection against dangerous commands.
★ 261 · PythonAI LINUX ADMINISTRATOR IN RUST
Sysknife is an agent for Linux administration. The model picks the action; the daemon builds the command. All actions require approval and are logged with Ed25519 signatu
★ 261 · RustDS AUTO-REVIEW:AI SYSTEM
dsh-auto-review offers a read-only subagent that returns allow/deny with reasons for DeepSeek Harness approval requests. Verdict is fail-closed by default; the chain appr
★ 249 · TypeScriptDOBERMAN-CORE — PROTECTION FOR AI AGENTS
Doberman-Core intercepts every input, output, and tool call during AI-agent operation, blocking unsafe actions. It works as middleware, checking authorization and applyin
★ 243 · PythonRED-TEAMING AI AGENTS IN PYTHON
Humanbound — adversarial testing engine, SDK, and CLI for AI agents. Runs locally or via Humanbound Platform. 218 stars on GitHub.
★ 239 · PythonALL PROMPTS CLAUDE IN ONE
All official Claude prompts in one repository: verbatim, dated and source-linked, from Claude Fable 5.2 to Haiku 3. 23 models from Anthropic's release notes, plus analysi
★ 232 · PythonGUARDANA AI MODEL PROTECTION
Guardana scans model artifacts, live endpoints, MCP servers, and agent traces for security threats. It generates reproducible evidence for release decisions.
★ 225 · PythonDEFENSE MEMORY AI-AGENTS
Provides runtime defense against memory AI-agent poisoning (OWASP ASI06). Drop-in for LangChain, OpenAI Agents, AutoGen, CrewAI and mem0.
★ 203 · PythonNODE9-PROXY: SHIELD FOR AI AGENTS
node9-proxy provides deterministic management and auditing of autonomous AI agents, ensuring safe code execution.
★ 201 · TypeScriptPERMIT0 — AI Agent Error Protection
Authorize AI-agent actions before execution. Deterministic policies, simple integration from day one.
★ 192 · RustGPT-6 ASTRA: PROMPTS INTEGRATIONS
Evidence-backed use cases, prompts, integrations, evaluations, and safety notes for OpenAI GPT-6 Astra.
★ 17 · PythonNew projects on this topic in Telegram
Telegram →