R_REDDYX.XYZ

Data Engineering: open-source projects on GitHub

20 repositories on Data Engineering, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094AI Agents 1319MCP 1601RAG 533Claude Code 1595Agents 400Claude 978Machine Learning 190Data Analysis 25Agent 547Analytics 36Devops 82MCP Server 631Business Intelligence 15
kestra-io/kestra

WORKFLOW ORCHESTRATOR FOR DATA AND AI

Open-source orchestration platform for data, AI, and infrastructure. Unifies scheduled and event-driven workflows through declarative YAML with Git versioning and AI Copi

★ 30,287 · Java
cocoindex-io/cocoindex

Cocoindex: Incremental Engine for Agents

An incremental engine for agents with long-term horizon. If you like it, give it a star!

★ 9,785 · Python
xonsh/xonsh

Python shell xonsh

A cross‑platform shell that combines the power of Python with a familiar command‑line interface, and supports AI assistants.

★ 9,336 · Python
Zipstack/unstract

UNSTRACT — LLM vs Data Strings

Extract structured data from unstructured documents using LLM. Ready for deployment via API and ETL pipelines. ⭐ 6624

★ 6,624 · Python
Eventual-Inc/Daft

DAFT — Data Engine for AI

High-performance engine for AI and multimodal tasks that processes images, audio, video, and structured data at any scale. Built on Rust ⭐5533

★ 5,533 · Rust
apache/hamilton

HAMILTON - MODULAR DATAFLOW

Apache Hamilton simplifies creating testable, self-documenting data pipelines with lineage tracking.

★ 2,512 · Jupyter Notebook
slothflowlabs/duckle

DUCKLE: POWERFUL LOCAL ETL STUDIO

Open-source local ETL/ELT studio with 360+ components, running at DuckDB speed and generating pipelines via on-device AI.

★ 1,573 · Rust
Zleap-AI/SAG

SAG: WORKBENCH FOR DOCUMENT RAG

Ready-to-use workbench for document search based on SAG. TypeScript, 1136 stars!

★ 1,136 · TypeScript
Kaelio/ktx-ai-data-agents-context

KTX — Context for AI Agents

Allows AI agents (Claude Code, Codex, etc.) to precisely request data via MCP with skills, memory, and a semantic layer. 🔥

★ 668 · TypeScript
AltimateAI/altimate-code

ALTIMATE-CODE AI FOR DATA

Open-source agentic engine for dbt, SQL, and cloud storage: 100+ tools, support for 10 warehouses, AI‑powered.

★ 612 · TypeScript
dbt-labs/dbt-mcp

DBT-MCP: Simplify DBT

An MCP server for integrating dbt with any AI agents and workflows. It allows running dbt models through a unified interface.

★ 578 · Python
astronomer/agents

AGENTS: AI Agent for Data

Agents — a toolkit for creating AI agents that automate data engineering. It supports ETL pipelines, LLM, and workflow management.

★ 463 · Python
Kaelio/ktx-ai-data-agents-mcp-context-skills

Context Layer for AI Agents

Ktx is an executable context layer that lets Claude Code, Codex, and any AI agent precisely query data via MCP using the agent's skills and memory.

★ 427 · TypeScript
Kaelio/ktx

Ktx — Context Layer

Ktx provides a universal context management layer for AI agents, allowing storing, updating, and sharing state across various modules and services.

★ 395 · TypeScript
MelihGulum/Comprehensive-Data-Science-AI-Project-Portfolio

COMPREHENSIVE DATA SCIENCE PORTFOLIO

A collection of projects on AI, data, and DevOps with real-world cases, advanced techniques, and step-by-step tutorials. Suitable for beginners and professionals studying

★ 357 · Jupyter Notebook
Hebbian-Robotics/hflow

hflow: DATA QUALITY VERIFICATION FOR AI

hflow — Python SDK for robotics teams. Verifies data quality before AI model training. Integration with Airflow and DuckDB.

★ 310 · Python
dagster-io/skills

COLLECTION OF AI SKILLS FOR DAGSTER

Project collects 218+ AI skills for automating Dagster workflows. Ready modules for data orchestration, Claude Code integration, and engineering tasks. Open Python market

★ 218 · Python
BlueRingsLabs/no-shortcuts

Comprehensive AI Engineering Course

Free 173-lesson course covering data, ML, deep learning, and AI engineering with 166 runnable labs. Covers math, code, systems, and security under CC BY-NC 4.0.

★ 175 · Python
ashishsinha1602/schemagate

Schema-Aware SQL Access Control

Filters database schemas by user permissions before LLM queries, enforcing row-level security across Postgres, Oracle, MySQL, and SQL Server. Cuts prompt tokens by 75%.

★ 137 · Python
clarilayer/clarilayer

Analytic Context Layer

ClariLayer preserves your analytic context between AI sessions, automatically loading the needed data and metadata via the MCP protocol.

★ 71 · TypeScript

New projects on this topic in Telegram

Telegram →
Data Engineering — 20 open-source GitHub projects | REDDYX AI