Data Engineering: open-source projects on GitHub
20 repositories on Data Engineering, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
WORKFLOW ORCHESTRATOR FOR DATA AND AI
Open-source orchestration platform for data, AI, and infrastructure. Unifies scheduled and event-driven workflows through declarative YAML with Git versioning and AI Copi
★ 30,287 · JavaCocoindex: Incremental Engine for Agents
An incremental engine for agents with long-term horizon. If you like it, give it a star!
★ 9,785 · PythonPython shell xonsh
A cross‑platform shell that combines the power of Python with a familiar command‑line interface, and supports AI assistants.
★ 9,336 · PythonUNSTRACT — LLM vs Data Strings
Extract structured data from unstructured documents using LLM. Ready for deployment via API and ETL pipelines. ⭐ 6624
★ 6,624 · PythonDAFT — Data Engine for AI
High-performance engine for AI and multimodal tasks that processes images, audio, video, and structured data at any scale. Built on Rust ⭐5533
★ 5,533 · RustHAMILTON - MODULAR DATAFLOW
Apache Hamilton simplifies creating testable, self-documenting data pipelines with lineage tracking.
★ 2,512 · Jupyter NotebookDUCKLE: POWERFUL LOCAL ETL STUDIO
Open-source local ETL/ELT studio with 360+ components, running at DuckDB speed and generating pipelines via on-device AI.
★ 1,573 · RustSAG: WORKBENCH FOR DOCUMENT RAG
Ready-to-use workbench for document search based on SAG. TypeScript, 1136 stars!
★ 1,136 · TypeScriptKTX — Context for AI Agents
Allows AI agents (Claude Code, Codex, etc.) to precisely request data via MCP with skills, memory, and a semantic layer. 🔥
★ 668 · TypeScriptALTIMATE-CODE AI FOR DATA
Open-source agentic engine for dbt, SQL, and cloud storage: 100+ tools, support for 10 warehouses, AI‑powered.
★ 612 · TypeScriptDBT-MCP: Simplify DBT
An MCP server for integrating dbt with any AI agents and workflows. It allows running dbt models through a unified interface.
★ 578 · PythonAGENTS: AI Agent for Data
Agents — a toolkit for creating AI agents that automate data engineering. It supports ETL pipelines, LLM, and workflow management.
★ 463 · PythonContext Layer for AI Agents
Ktx is an executable context layer that lets Claude Code, Codex, and any AI agent precisely query data via MCP using the agent's skills and memory.
★ 427 · TypeScriptKtx — Context Layer
Ktx provides a universal context management layer for AI agents, allowing storing, updating, and sharing state across various modules and services.
★ 395 · TypeScriptCOMPREHENSIVE DATA SCIENCE PORTFOLIO
A collection of projects on AI, data, and DevOps with real-world cases, advanced techniques, and step-by-step tutorials. Suitable for beginners and professionals studying
★ 357 · Jupyter Notebookhflow: DATA QUALITY VERIFICATION FOR AI
hflow — Python SDK for robotics teams. Verifies data quality before AI model training. Integration with Airflow and DuckDB.
★ 310 · PythonCOLLECTION OF AI SKILLS FOR DAGSTER
Project collects 218+ AI skills for automating Dagster workflows. Ready modules for data orchestration, Claude Code integration, and engineering tasks. Open Python market
★ 218 · PythonComprehensive AI Engineering Course
Free 173-lesson course covering data, ML, deep learning, and AI engineering with 166 runnable labs. Covers math, code, systems, and security under CC BY-NC 4.0.
★ 175 · PythonSchema-Aware SQL Access Control
Filters database schemas by user permissions before LLM queries, enforcing row-level security across Postgres, Oracle, MySQL, and SQL Server. Cuts prompt tokens by 75%.
★ 137 · PythonAnalytic Context Layer
ClariLayer preserves your analytic context between AI sessions, automatically loading the needed data and metadata via the MCP protocol.
★ 71 · TypeScriptNew projects on this topic in Telegram
Telegram →