Web Crawler: open-source projects on GitHub
12 repositories on Web Crawler, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Firecrawl Web API
Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.
★ 118,560 · TypeScriptOPEN SOURCE CRAWLER FOR LLM
Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.
★ 87,369 · PythonScrapegraph: AI scraper on Python
Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.
★ 25,439 · PythonHeadless Browser for AI Agents
Lightweight headless browser built in Rust for AI agents, offering fast execution and high compatibility with modern web standards.
★ 23,747 · RustFirecrawl MCP Server
🔥 Official Firecrawl MCP Server – adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
★ 6,384 · JavaScriptWIGOLO: Local AI Assistant
Wigolo — local search and data collection for AI agents without API keys or the cloud. Free, public beta, 621⭐.
★ 5,544 · TypeScriptSPIDER: LIGHTNING-FAST WEB CRAWLER
High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.
★ 2,849 · RustFAST ALTERNATIVE FIRECRAWL IN RUST
crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f
★ 1,257 · RustINFRASTRUCTURE FOR AI AGENTS
Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.
★ 565 · TypeScriptSCRAPER GOOGLE NEWS IN JavaScript
Lightweight scraper for extracting articles from Google News. Supports pagination, filtering by date and keywords, output in JSON. Suitable for research and forecasting
★ 385 · JavaScriptRust Web Crawler for AI
High-performance Rust crawler extracting clean Markdown and structured data for AI agents and RAG pipelines. Supports PDF/RSS parsing, browser rendering, and Firecrawl-co
★ 272 · RustAI Web Answer Crawler
A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.
★ 89 · PythonNew projects on this topic in Telegram
Telegram →