R_REDDYX.XYZ

Web Crawler: open-source projects on GitHub

12 repositories on Web Crawler, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

Web Scraping 53AI Agents 1319Crawler 13MCP 1601Data Extraction 15LLM 2094Markdown 98RAG 533MCP Server 631Scraper 15Scraping 15Web Search 16Claude 978AI Search 20
firecrawl/firecrawl

Firecrawl Web API

Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.

★ 118,560 · TypeScript
unclecode/crawl4ai

OPEN SOURCE CRAWLER FOR LLM

Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.

★ 87,369 · Python
ScrapeGraphAI/Scrapegraph-ai

Scrapegraph: AI scraper on Python

Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.

★ 25,439 · Python
lexmount/moli

Headless Browser for AI Agents

Lightweight headless browser built in Rust for AI agents, offering fast execution and high compatibility with modern web standards.

★ 23,747 · Rust
firecrawl/firecrawl-mcp-server

Firecrawl MCP Server

🔥 Official Firecrawl MCP Server – adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

★ 6,384 · JavaScript
KnockOutEZ/wigolo

WIGOLO: Local AI Assistant

Wigolo — local search and data collection for AI agents without API keys or the cloud. Free, public beta, 621⭐.

★ 5,544 · TypeScript
spider-rs/spider

SPIDER: LIGHTNING-FAST WEB CRAWLER

High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.

★ 2,849 · Rust
fastcrw/crw

FAST ALTERNATIVE FIRECRAWL IN RUST

crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f

★ 1,257 · Rust
vakra-dev/reader

INFRASTRUCTURE FOR AI AGENTS

Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.

★ 565 · TypeScript
lewisdonovan/google-news-scraper

SCRAPER GOOGLE NEWS IN JavaScript

Lightweight scraper for extracting articles from Google News. Supports pagination, filtering by date and keywords, output in JSON. Suitable for research and forecasting

★ 385 · JavaScript
YOLOW92/pagesift

Rust Web Crawler for AI

High-performance Rust crawler extracting clean Markdown and structured data for AI agents and RAG pipelines. Supports PDF/RSS parsing, browser rendering, and Firecrawl-co

★ 272 · Rust
RayenMlayeh/jevcrawl

AI Web Answer Crawler

A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.

★ 89 · Python

New projects on this topic in Telegram

Telegram →
Web Crawler — 12 open-source GitHub projects | REDDYX AI