R_REDDYX.XYZ

Crawler: open-source projects on GitHub

13 repositories on Crawler, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

Web Scraping 53LLM 2094Data Extraction 15Web Crawler 12AI Agents 1319Markdown 98Scraping 15MCP 1601RAG 533Scraper 15MCP Server 631AI Search 20Web Search 16Playwright 60
firecrawl/firecrawl

Firecrawl Web API

Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.

★ 118,560 · TypeScript
unclecode/crawl4ai

OPEN SOURCE CRAWLER FOR LLM

Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.

★ 87,369 · Python
D4Vinci/Scrapling

Adaptive web scraper

An adaptive framework for web scraping that supports both single requests and large-scale crawlers.

★ 49,091 · Python
ScrapeGraphAI/Scrapegraph-ai

Scrapegraph: AI scraper on Python

Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.

★ 25,439 · Python
Evil0ctal/Douyin_TikTok_Download_API

DOUYIN TIKTOK DOWNLOADER API

Self-hosted TikTok/Douyin scraper with async REST API, MCP, CLI and console downloads videos without watermarks and collects posts, profiles, comments and playlists. Incl

★ 21,404 · Python
adbar/trafilatura

TRAFILATURA - FAST WEB PARSE

Trafilatura extracts clean text and metadata from pages, supporting CSV, JSON, HTML and other formats.

★ 6,055 · Python
spider-rs/spider

SPIDER: LIGHTNING-FAST WEB CRAWLER

High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.

★ 2,849 · Rust
fastcrw/crw

FAST ALTERNATIVE FIRECRAWL IN RUST

crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f

★ 1,257 · Rust
codelibs/fess

FESS: AI SEARCH FOR BUSINESS

Fess is an open source search server built on OpenSearch, indexing web, files, databases and clouds, supporting 20+ languages and semantic search with AI/RAG.

★ 1,151 · Java
polyrabbit/hacker-news-digest

HACKER-NEWS-DIGEST - READ FASTER WITH AI

Automatically summarizes fresh Hacker News with ChatGPT so you stay updated without extra reading.

★ 754 · Python
vakra-dev/reader

INFRASTRUCTURE FOR AI AGENTS

Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.

★ 565 · TypeScript
he-yufeng/FindJobs-Agent

🔥 AI Job Agent

LLM tool for skill analysis, AI interviews, resume scoring, and job vacancy structuring with adaptive complexity.

★ 244 · Python
RayenMlayeh/jevcrawl

AI Web Answer Crawler

A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.

★ 89 · Python

New projects on this topic in Telegram

Telegram →
Crawler — 13 open-source GitHub projects | REDDYX AI