R_REDDYX.XYZ

Web Scraping: open-source projects on GitHub

53 repositories on Web Scraping, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

AI Agents 1319MCP 1601LLM 2094MCP Server 631Browser Automation 72Playwright 60Automation 272Data Extraction 15Web Crawler 12AI Agent 690Crawler 13Scraping 15RAG 533Claude Code 1595
firecrawl/firecrawl

Firecrawl Web API

Firecrawl provides a simple API for searching, scraping, and interacting with web pages, optimized for use in AI applications.

★ 118,560 · TypeScript
unclecode/crawl4ai

OPEN SOURCE CRAWLER FOR LLM

Crawl4ai turns any site into clean Markdown ready for RAG and agents. 84k GitHub stars, run locally or via cloud.

★ 87,369 · Python
Mintplex-Labs/anything-llm

AI Productivity On-Device

A universal AI productivity tool that runs locally and guarantees complete privacy without complex setup.

★ 59,922 · JavaScript
D4Vinci/Scrapling

Adaptive web scraper

An adaptive framework for web scraping that supports both single requests and large-scale crawlers.

★ 49,091 · Python
browserbase/stagehand

STAGEHAND SDK FOR BROWSER WEB AGENTS

Stagehand provides TypeScript SDK for creating browser agents that can interact, search, extract, and fetch data from any website reliably across the web. It simplifies a

★ 26,054 · TypeScript
ScrapeGraphAI/Scrapegraph-ai

Scrapegraph: AI scraper on Python

Library for web scraping using artificial intelligence. Automatically extracts structured data from any web page.

★ 25,439 · Python
lexmount/moli

Headless Browser for AI Agents

Lightweight headless browser built in Rust for AI agents, offering fast execution and high compatibility with modern web standards.

★ 23,747 · Rust
Evil0ctal/Douyin_TikTok_Download_API

DOUYIN TIKTOK DOWNLOADER API

Self-hosted TikTok/Douyin scraper with async REST API, MCP, CLI and console downloads videos without watermarks and collects posts, profiles, comments and playlists. Incl

★ 21,404 · Python
JCodesMore/ai-website-cloner-template

AI website cloner

A TypeScript template for cloning any website with one command using AI agents. 14,490 stars.

★ 14,490 · TypeScript
nanobrowser/nanobrowser

AI AGENT FOR BROWSER AUTOMATION

Open-source AI agent for automating tasks in your browser. Works via side panel, uses your logged-in profile to click, type, and navigate tabs.

★ 14,332 · TypeScript
yusufkaraaslan/Skill_Seekers

Claude Skill Creator

Transforms documentation, GitHub repositories, and PDFs into Claude AI skills, automatically detecting conflicts.

★ 13,443 · Python
seleniumbase/SeleniumBase

SELENIUMBASE: STEALTH BROWSER AUTOMATION

SeleniumBase combines browser automation, E2E testing, and stealth mode. Bypasses CAPTCHA and Cloudflare via CDP Mode. Includes MCP server for AI agent integration.

★ 13,255 · Python
cporter202/API-mega-list

API-MEGA-LIST - COMPLETE API LIST

A collection of the most useful APIs for quick project start: from automation to full-stack applications.

★ 8,392 · JavaScript
firecrawl/firecrawl-mcp-server

Firecrawl MCP Server

🔥 Official Firecrawl MCP Server – adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

★ 6,384 · JavaScript
adbar/trafilatura

TRAFILATURA - FAST WEB PARSE

Trafilatura extracts clean text and metadata from pages, supporting CSV, JSON, HTML and other formats.

★ 6,055 · Python
KnockOutEZ/wigolo

WIGOLO: Local AI Assistant

Wigolo — local search and data collection for AI agents without API keys or the cloud. Free, public beta, 621⭐.

★ 5,544 · TypeScript
jo-inc/camofox-browser

CamoFox: Stealth Browser

Stealth headless browser for AI agents, bypassing Cloudflare, bot protection, and anti‑scraping. Fully compatible with Puppeteer and Playwright.

★ 4,734 · JavaScript
AhmadHassan-BTed/Noria

AGENT FOR SCORING LINKS IN CHAT

Noria is an agentic extractor for messengers. Receives messages and URLs, scrapes web pages, evaluates content via custom matrices (jobs, scholarships), and sends a struc

★ 4,630 · JavaScript
vibheksoni/stealth-browser-mcp

STEALTH BROWSER ANTI‑BOT BYPASS

The project enables browser automation by bypassing anti‑bot systems using AI‑generated network hooks and pixel‑perfect UI cloning via chat.

★ 3,364 · Python
spider-rs/spider

SPIDER: LIGHTNING-FAST WEB CRAWLER

High-speed web data streaming with minimal latency. Ideal for real-time parsing and data collection.

★ 2,849 · Rust
Mahanaicoach/google-maps-scraper-kit

LEADS FROM GOOGLE MAPS NO API

Script extracts leads from Google Maps phones, emails, websites and socials into clean CSV. Runs locally via Claude, no API-key, on your machine.

★ 2,453 · Python
brightdata/brightdata-mcp

BRIGHTDATA-MCP: SINGLE SOLUTION FOR WEB

Model Context Protocol server for accessing public web data without complex setup.

★ 2,434 · JavaScript
A9T9/RPA

RPA: Visual Automation with AI

Open-source RPA platform with computer vision, OCR, and LLM assistant. Import/export tests from Selenium IDE.

★ 2,108 · JavaScript
karust/openserp

OPENSERP — AI SEARCH API SELF-HOSTED

Self-hosted SERP API that renders Google, Bing, Yandex and other search engine results with page extraction.

★ 1,584 · Go
browser-act/skills

BROWSER AUTOMATION FOR AI AGENTS

CLI tool for browser automation with built-in anti‑bot bypass, human task handoff, and parallel execution across multiple sessions.

★ 1,429 · Python
ScrapingBee/chatgpt-scraper-api

CHATGPT-SCRAPER - INTELLIGENT SCRAPER

Get structured responses from ChatGPT scraping API in one line. Enable live search, HTML context, and automate tasks.

★ 1,411
fastcrw/crw

FAST ALTERNATIVE FIRECRAWL IN RUST

crw — fast Firecrawl/Tavily alternative in Rust: web scraper, crawler and search API with MCP server for AI agents. Compatible API (/scrape, /crawl, /search), runs 2.3× f

★ 1,257 · Rust
cutetailbud/ff-ruyijs

HEADLESS BROWSER ON SPIDERMONKEY FOR AI

Lightweight headless browser on SpiderMonkey engine (Firefox 160) and Necko network stack to run site JavaScript from Python. Designed for AI agents: scraping, bypassing

★ 1,245 · C++
Anil-matcha/awesome-agent-apis

AGENT API CATALOG

awesome-agent-apis collects over 660 generative media models hosted on muapi, plus community‑submitted API tools for SEO, enrichment, social media, and web scraping. Each

★ 1,070 · Python
runesleo/x-reader

UNIVERSAL CONTENT READER

x-reader — MCP server that reads content from more than 10 platforms: social networks, blogs, repositories and documents. Written in Python, it supports text and metadata

★ 990 · Python
oxylabs/agent-skills

AGENT-SKILLS - SMART SKILLS FOR AGENTS

Official agent skills from Oxylabs that simplify integration and expand automation capabilities.

★ 878
oss-invisible/playwright_mcp

INVISIBLE PLAYWRIGHT MCP FOR AI

Playwright MCP server invisible to anti-bots and captchas. AI agent runs in stealth Firefox, bypasses protection, and automates browser undetected.

★ 865 · Python
kalil0321/reverse-api-engineer

REVERSE-API-ENGINEER - Website API Agent

Python AI agent automatically analyzes websites and turns them into full-fledged APIs — no more manual parser writing.

★ 758 · Python
only-cli/oc

OC: WEB IN TERMINAL

Convert any website into a compact CLI for AI agents, saving tokens when working with web pages.

★ 696 · JavaScript
firecrawl/cli

FIERY CLI FOR WEB-DATA

Firecrawl CLI provides web search, scraping, crawling and browser interaction skills directly from the terminal or any AI agent. It is the most complete web data API for

★ 686 · TypeScript
vakra-dev/reader

INFRASTRUCTURE FOR AI AGENTS

Open web toolkit for AI: scraping, crawling, clean Markdown, and browser sessions ready for your agents.

★ 565 · TypeScript
Johell1NS/browser-search

AGENT SELF-SEARCH NET

Self-contained search skill for AI agents: SearXNG, anti-detect browser, and hallucination protection. Free, no limits.

★ 546 · JavaScript
AeternaLabsHQ/pullmd

PULLMD: CONVERT CONTENT TO MARKDOWN

Self-hosted service converts URLs and files (pages, documents, images, audio, YouTube) to Markdown. Supports PWA, REST, MCP, Claude Code skill, Reddit‑aware, and refresha

★ 524 · JavaScript
everyinfra/everyinfra-agent-plugins

MCP SCRAPER: 573 API

EveryInfra MCP server and plugins for AI agents. 573 scraper APIs for TikTok, Instagram, Amazon, YouTube, Xiaohongshu, Douyin and 100+ platforms. From $0.56 per 1K calls.

★ 497 · Python
flytohub/flyto-core

FLYTO-CORE: AI AGENT ENGINE

Open-source engine for executing AI agents with 412 modules, native MCP support, queues, versioning, and usage measurement.

★ 483 · Python
redf0x1/camofox-browser

camofox-browser 🔥

Anti-detection browser server for AI agents. REST API Camoufox with OpenClaw plugins, 257 stars on GitHub.

★ 445 · TypeScript
oxylabs/AI-Search-py

AI-SEARCH-PY - Full Intelligent Interpreter

Convert simple English queries into structured data with automatic Markdown extraction and JavaScript rendering for dynamic sites.

★ 423
Bin-Huang/camoufox-cli

CAMOUFOX CLI: ANONYMOUS BROWSER

CLI utility for camouflaging a browser based on Camoufox, removing Playwright traces and enabling stealth automation for AI agents.

★ 363 · Python
danishashko/geo-aeo-tracker

REPORT ON AI VISIBILITY

geo-aeo-tracker: open-source AI visibility and AEO tracker. Track how ChatGPT, Perplexity, Gemini and Google AI Mode mention and cite your brand. MCP server + CLI, self-h

★ 328 · TypeScript
AliAkhtari78/SpotifyScraper

SPOTIFY SCRAPER: DATA WITHOUT API

Extract public Spotify data — tracks, albums, artists, playlists, podcasts, and lyrics — without the official API. Sync + async, typed models, single dependency.

★ 327 · Python
opendatalab/MinerU-HTML

SLM EXTRACTOR FOR MAIN HTML CONTENT

MinerU-HTML uses a small language model to extract main content from HTML pages. Outputs clean HTML without clutter. Ideal for RAG, deep research agents, and training dat

★ 302 · Python
vakovalskii/searcharvester

SEARCHARVESTER: AI AGENT SEARCH AND FETCH

Self‑hosted search engine based on SearXNG (100+ sources) with FastAPI and trafilatura. Tavily‑compatible /search and /extract endpoints with size limits and pagination,

★ 283 · Python
YOLOW92/pagesift

Rust Web Crawler for AI

High-performance Rust crawler extracting clean Markdown and structured data for AI agents and RAG pipelines. Supports PDF/RSS parsing, browser rendering, and Firecrawl-co

★ 272 · Rust
clearcotelabs/clearcote-browser

STEALTH CHROMIUM BROWSER CLEARCOTE

clearcote-browser — open-source stealth Chromium with engine-level fingerprint spoofing. It is fully de-Googled, Playwright-compatible as a drop-in replacement and built

★ 263 · Python
ItamarZand88/CLI-Anything-WEB

CLI-Anything-WEB: CLI From Any Site

Claude Code plugin that generates production-ready Python CLI for any web app. Already 20+ ready-made tools.

★ 242 · Python
RayenMlayeh/jevcrawl

AI Web Answer Crawler

A MCP server and CLI that crawls websites to locate answer pages for AI agents, achieving twice the hit rate of Crawl4AI at one‑tenth the cost of Claude crawling.

★ 89 · Python
akwin1234/damru

Damru Stealthy Android Browser

Damru is a Python framework for stealth Android browser automation, using C++ overrides, root property spoofing, and GPU binary patching to bypass Cloudflare, DataDome, a

★ 31 · Python
Mrbaeksang/deepcloak

Deep Web Researcher

A local agent for deep web research that can bypass bot protection and gather data from any site. It generates reports with accurate links and works fully offline.

★ 31 · Python

New projects on this topic in Telegram

Telegram →
Web Scraping — 53 open-source GitHub projects | REDDYX AI