Web Scraping: open-source projects on GitHub
53 repositories on Web Scraping, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Firecrawl Web API
Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.
★ 118,560 · TypeScriptOPEN SOURCE CRAWLER FOR LLM
Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.
★ 87,369 · PythonAI Productivity On-Device
A universal AI productivity tool that runs locally and guarantees complete privacy without complex setup.
★ 59,922 · JavaScriptAdaptive web scraper
An adaptive framework for web scraping that supports both single requests and large-scale crawlers.
★ 49,091 · PythonSTAGEHAND SDK FOR BROWSER WEB AGENTS
Stagehand provides TypeScript SDK for creating browser agents that can interact, search, extract, and fetch data from any website reliably across the web. It simplifies a
★ 26,054 · TypeScriptScrapegraph: AI scraper on Python
Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.
★ 25,439 · PythonHeadless Browser for AI Agents
Lightweight headless browser built in Rust for AI agents, offering fast execution and high compatibility with modern web standards.
★ 23,747 · RustDOUYIN TIKTOK DOWNLOADER API
Self-hosted TikTok/Douyin scraper with async REST API, MCP, CLI and console downloads videos without watermarks and collects posts, profiles, comments and playlists. Incl
★ 21,404 · PythonAI website cloner
A TypeScript template for cloning any website with one command using AI agents. 14,490 stars.
★ 14,490 · TypeScriptAI AGENT FOR BROWSER AUTOMATION
Open-source AI agent for automating tasks in your browser. Works via side panel, uses your logged-in profile to click, type, and navigate tabs.
★ 14,332 · TypeScriptClaude Skill Creator
Transforms documentation, GitHub repositories, and PDFs into Claude AI skills, automatically detecting conflicts.
★ 13,443 · PythonSELENIUMBASE: STEALTH BROWSER AUTOMATION
SeleniumBase combines browser automation, E2E testing, and stealth mode. Bypasses CAPTCHA and Cloudflare via CDP Mode. Includes MCP server for AI agent integration.
★ 13,255 · PythonAPI-MEGA-LIST - COMPLETE API LIST
A collection of the most useful APIs for quick project start: from automation to full-stack applications.
★ 8,392 · JavaScriptFirecrawl MCP Server
🔥 Official Firecrawl MCP Server – adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
★ 6,384 · JavaScriptTRAFILATURA - FAST WEB PARSE
Trafilatura extracts clean text and metadata from pages, supporting CSV, JSON, HTML and other formats.
★ 6,055 · PythonWIGOLO: Local AI Assistant
Wigolo — local search and data collection for AI agents without API keys or the cloud. Free, public beta, 621⭐.
★ 5,544 · TypeScriptCamoFox: Stealth Browser
Stealth headless browser for AI agents, bypassing Cloudflare, bot protection, and anti‑scraping. Fully compatible with Puppeteer and Playwright.
★ 4,734 · JavaScriptAGENT FOR SCORING LINKS IN CHAT
Noria is an agentic extractor for messengers. Receives messages and URLs, scrapes web pages, evaluates content via custom matrices (jobs, scholarships), and sends a struc
★ 4,630 · JavaScriptSTEALTH BROWSER ANTI‑BOT BYPASS
The project enables browser automation by bypassing anti‑bot systems using AI‑generated network hooks and pixel‑perfect UI cloning via chat.
★ 3,364 · PythonSPIDER: LIGHTNING-FAST WEB CRAWLER
High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.
★ 2,849 · RustLEADS FROM GOOGLE MAPS NO API
Script extracts leads from Google Maps phones, emails, websites and socials into clean CSV. Runs locally via Claude, no API-key, on your machine.
★ 2,453 · PythonBRIGHTDATA-MCP: SINGLE SOLUTION FOR WEB
Model Context Protocol server for accessing public web data without complex setup.
★ 2,434 · JavaScriptRPA: Visual Automation with AI
Open-source RPA platform with computer vision, OCR, and LLM assistant. Import/export tests from Selenium IDE.
★ 2,108 · JavaScriptOPENSERP — AI SEARCH API SELF-HOSTED
Self-hosted SERP API that renders Google, Bing, Yandex and other search engine results with page extraction.
★ 1,584 · GoBROWSER AUTOMATION FOR AI AGENTS
CLI tool for browser automation with built-in anti‑bot bypass, human task handoff, and parallel execution across multiple sessions.
★ 1,429 · PythonCHATGPT-SCRAPER - INTELLIGENT SCRAPER
Get structured responses from ChatGPT scraping API in one line. Enable live search, HTML context, and automate tasks.
★ 1,411FAST ALTERNATIVE FIRECRAWL IN RUST
crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f
★ 1,257 · RustHEADLESS BROWSER ON SPIDERMONKEY FOR AI
Lightweight headless browser on SpiderMonkey engine (Firefox 160) and Necko network stack to run site JavaScript from Python. Designed for AI agents: scraping, bypassing
★ 1,245 · C++AGENT API CATALOG
awesome-agent-apis collects over 660 generative media models hosted on muapi, plus community‑submitted API tools for SEO, enrichment, social media, and web scraping. Each
★ 1,070 · PythonUNIVERSAL CONTENT READER
x-reader — MCP server that reads content from more than 10 platforms: social networks, blogs, repositories and documents. Written in Python, it supports text and metadata
★ 990 · PythonAGENT-SKILLS - SMART SKILLS FOR AGENTS
Official agent skills from Oxylabs that simplify integration and expand automation capabilities.
★ 878INVISIBLE PLAYWRIGHT MCP FOR AI
Playwright MCP server invisible to anti-bots and captchas. AI agent runs in stealth Firefox, bypasses protection, and automates browser undetected.
★ 865 · PythonREVERSE-API-ENGINEER - Website API Agent
Python AI agent automatically analyzes websites and turns them into full-fledged APIs — no more manual parser writing.
★ 758 · PythonOC: WEB IN TERMINAL
Convert any website into a compact CLI for AI agents, saving tokens when working with web pages.
★ 696 · JavaScriptFIERY CLI FOR WEB-DATA
Firecrawl CLI provides web search, scraping, crawling and browser interaction skills directly from the terminal or any AI agent. It is the most complete web data API for
★ 686 · TypeScriptINFRASTRUCTURE FOR AI AGENTS
Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.
★ 565 · TypeScriptAGENT SELF-SEARCH NET
Self-contained search skill for AI agents: SearXNG, anti-detect browser, and hallucination protection. Free, no limits.
★ 546 · JavaScriptPULLMD: CONVERT CONTENT TO MARKDOWN
Self-hosted service converts URLs and files (pages, documents, images, audio, YouTube) to Markdown. Supports PWA, REST, MCP, Claude Code skill, Reddit‑aware, and refresha
★ 524 · JavaScriptMCP SCRAPER: 573 API
EveryInfra MCP server and plugins for AI agents. 573 scraper APIs for TikTok, Instagram, Amazon, YouTube, Xiaohongshu, Douyin and 100+ platforms. From $0.56 per 1K calls.
★ 497 · PythonFLYTO-CORE: AI AGENT ENGINE
Open-source engine for executing AI agents with 412 modules, native MCP support, queues, versioning, and usage measurement.
★ 483 · Pythoncamofox-browser 🔥
Anti-detection browser server for AI agents. REST API Camoufox with OpenClaw plugins, 257 stars on GitHub.
★ 445 · TypeScriptAI-SEARCH-PY - Full Intelligent Interpreter
Convert simple English queries into structured data with automatic Markdown extraction and JavaScript rendering for dynamic sites.
★ 423CAMOUFOX CLI: ANONYMOUS BROWSER
CLI utility for camouflaging a browser based on Camoufox, removing Playwright traces and enabling stealth automation for AI agents.
★ 363 · PythonREPORT ON AI VISIBILITY
geo-aeo-tracker: open-source AI visibility and AEO tracker. Track how ChatGPT, Perplexity, Gemini and Google AI Mode mention and cite your brand. MCP server + CLI, self-h
★ 328 · TypeScriptSPOTIFY SCRAPER: DATA WITHOUT API
Extract public Spotify data — tracks, albums, artists, playlists, podcasts, and lyrics — without the official API. Sync + async, typed models, single dependency.
★ 327 · PythonSLM EXTRACTOR FOR MAIN HTML CONTENT
MinerU-HTML uses a small language model to extract main content from HTML pages. Outputs clean HTML without clutter. Ideal for RAG, deep research agents, and training dat
★ 302 · PythonSEARCHARVESTER: AI AGENT SEARCH AND FETCH
Self‑hosted search engine based on SearXNG (100+ sources) with FastAPI and trafilatura. Tavily‑compatible /search and /extract endpoints with size limits and pagination,
★ 283 · PythonRust Web Crawler for AI
High-performance Rust crawler extracting clean Markdown and structured data for AI agents and RAG pipelines. Supports PDF/RSS parsing, browser rendering, and Firecrawl-co
★ 272 · RustSTEALTH CHROMIUM BROWSER CLEARCOTE
clearcote-browser — open-source stealth Chromium with engine-level fingerprint spoofing. It is fully de-Googled, Playwright-compatible as a drop-in replacement and built
★ 263 · PythonCLI-Anything-WEB: CLI From Any Site
Claude Code plugin that generates production-ready Python CLI for any web app. Already 20+ ready-made tools.
★ 242 · PythonAI Web Answer Crawler
A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.
★ 89 · PythonDamru Stealthy Android Browser
Damru is a Python framework for stealth Android browser automation, using C++ overrides, root property spoofing, and GPU binary patching to bypass Cloudflare, DataDome, a
★ 31 · PythonDeep Web Researcher
A local agent for deep web research that can bypass bot protection and gather data from any site. It generates reports with accurate links and works fully offline.
★ 31 · PythonNew projects on this topic in Telegram
Telegram →