Crawler: open-source projects on GitHub
13 repositories on Crawler, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Firecrawl Web API
Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.
★ 118,560 · TypeScriptOPEN SOURCE CRAWLER FOR LLM
Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.
★ 87,369 · PythonAdaptive web scraper
An adaptive framework for web scraping that supports both single requests and large-scale crawlers.
★ 49,091 · PythonScrapegraph: AI scraper on Python
Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.
★ 25,439 · PythonDOUYIN TIKTOK DOWNLOADER API
Self-hosted TikTok/Douyin scraper with async REST API, MCP, CLI and console downloads videos without watermarks and collects posts, profiles, comments and playlists. Incl
★ 21,404 · PythonTRAFILATURA - FAST WEB PARSE
Trafilatura extracts clean text and metadata from pages, supporting CSV, JSON, HTML and other formats.
★ 6,055 · PythonSPIDER: LIGHTNING-FAST WEB CRAWLER
High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.
★ 2,849 · RustFAST ALTERNATIVE FIRECRAWL IN RUST
crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f
★ 1,257 · RustFESS: AI SEARCH FOR BUSINESS
Fess is an open source search server built on OpenSearch, indexing web, files, databases and clouds, supporting 20+ languages and semantic search with AI/RAG.
★ 1,151 · JavaHACKER-NEWS-DIGEST - READ FASTER WITH AI
Automatically summarizes fresh Hacker News with ChatGPT so you stay updated without extra reading.
★ 754 · PythonINFRASTRUCTURE FOR AI AGENTS
Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.
★ 565 · TypeScript🔥 AI Job Agent
LLM tool for skill analysis, AI interviews, resume scoring, and job vacancy structuring with adaptive complexity.
★ 244 · PythonAI Web Answer Crawler
A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.
★ 89 · PythonNew projects on this topic in Telegram
Telegram →