← Explore
TOPIC

#web-crawler

Open source repositories tagged with #web-crawler, ranked by health score.

apify
apify/crawlee
TypeScript
89
health

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

24.6k
xberg-io
xberg-io/crawlberg
Rust
89
health

High-performance web crawling engine with bindings for 11 languages

134
microlinkhq
microlinkhq/browserless
JavaScript
88
health

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

1.8k