AIRust★ 272⑂ 37 forksApache-2.0
YOLOW92/pagesift
Rust Web Crawler for AI
High-performance Rust crawler extracting clean Markdown and structured data for AI agents and RAG pipelines. Supports PDF/RSS parsing, browser rendering, and Firecrawl-compatible APIs.
// KEY FEATURES
- Crawls adaptively with browser rendering for JavaScript-heavy sites
- Outputs clean Markdown and structured JSON for RAG ingestion
- Provides Firecrawl-compatible API with Python and Node.js SDKs
#ai-agents#browser-automation#data-extraction#firecrawl#html-to-markdown#markdown#mcp#pdf-extraction#rag#rust#web-crawler#web-scraping
Open on GitHub →New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram