D4Vinci/ScraplingPublic

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

AI summary: Adaptive Python web scraping framework supporting everything from single requests to full-scale concurrent crawls.

Stars
104.7K
+18 today
Forks
10.9K
Watchers
352
Open issues
4
Open PRs
17
Contributors
~54
Commits
1.8K
Branches
2
Views (30d)
5

PythonBSD-3-ClauseCreated Oct 13, 2024Last push 3d agoLatest release v0.5.1+1.8K stars this week+15.5K this month

Quick answers

What is Scrapling?
Adaptive Python web scraping framework supporting everything from single requests to full-scale concurrent crawls.
What does Scrapling do?
Scrapling is a highly adaptive web scraping library that combines dynamic DOM parsing with robust fetching and crawling architectures. Its parsing engine learns from website structures, allowing it to automatically relocate elements when page designs change unexpectedly. The built-in fetchers are designed to bypass strict anti-bot systems like Cloudflare Turnstile out of the box, utilizing stealthy headless browser techniques. Additionally, it provides a comprehensive spider framework that scales to handle concurrent, multi-session crawls with automated proxy rotation and pause-resume functionality. It unifies the simplicity of single-page requests with the power of massive distributed crawling in one tool.
Who is Scrapling for?
Data engineers, web scrapers, and AI developers who need a reliable, bot-resistant tool for extracting web data. It requires Python.
How popular is Scrapling on GitHub?
D4Vinci/Scrapling has 104,663 stars and 10,929 forks on GitHub, and gained 1,750 stars in the last 7 days.
What license does Scrapling use?
D4Vinci/Scrapling is released under the BSD-3-Clause license.

Star history

since Jul 28, 2026
050K100KJul 2026Sep 2026Oct 2026Nov 2026
104.7K stars as of Nov 21, 2026. Measured daily since Jul 28, 2026; GitHub no longer exposes earlier star timestamps.

Update history

4 recorded
  • Nov 6, 2026New release v0.5.1 (previously v0.5).
  • Oct 26, 2026New release v0.5 (previously v0.4.15).
  • Oct 26, 2026Description changed from "🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join her" to "🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, j".
  • Oct 17, 2026Description changed from "🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy" to "🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join her".

Contribution activity

commits per day, last 52 weeks
NovDecJanFebMarAprMayJunJulAugSepOctNovMonWedFri2025-11-23: 2 commits2025-11-24: 5 commits2025-11-25: 7 commits2025-11-26: 5 commits2025-11-27: 0 commits2025-11-28: 0 commits2025-11-29: 0 commits2025-11-30: 0 commits2025-12-01: 0 commits2025-12-02: 1 commit2025-12-03: 3 commits2025-12-04: 0 commits2025-12-05: 0 commits2025-12-06: 0 commits2025-12-07: 0 commits2025-12-08: 1 commit2025-12-09: 1 commit2025-12-10: 0 commits2025-12-11: 0 commits2025-12-12: 0 commits2025-12-13: 0 commits2025-12-14: 0 commits2025-12-15: 0 commits2025-12-16: 0 commits2025-12-17: 3 commits2025-12-18: 8 commits2025-12-19: 0 commits2025-12-20: 0 commits2025-12-21: 0 commits2025-12-22: 0 commits2025-12-23: 0 commits2025-12-24: 3 commits2025-12-25: 0 commits2025-12-26: 7 commits2025-12-27: 9 commits2025-12-28: 7 commits2025-12-29: 5 commits2025-12-30: 2 commits2025-12-31: 8 commits2026-01-01: 9 commits2026-01-02: 0 commits2026-01-03: 4 commits2026-01-04: 0 commits2026-01-05: 0 commits2026-01-06: 0 commits2026-01-07: 0 commits2026-01-08: 6 commits2026-01-09: 0 commits2026-01-10: 2 commits2026-01-11: 12 commits2026-01-12: 6 commits2026-01-13: 1 commit2026-01-14: 0 commits2026-01-15: 0 commits2026-01-16: 0 commits2026-01-17: 2 commits2026-01-18: 2 commits2026-01-19: 11 commits2026-01-20: 6 commits2026-01-21: 0 commits2026-01-22: 0 commits2026-01-23: 1 commit2026-01-24: 0 commits2026-01-25: 0 commits2026-01-26: 0 commits2026-01-27: 0 commits2026-01-28: 0 commits2026-01-29: 2 commits2026-01-30: 0 commits2026-01-31: 0 commits2026-02-01: 0 commits2026-02-02: 3 commits2026-02-03: 0 commits2026-02-04: 0 commits2026-02-05: 2 commits2026-02-06: 4 commits2026-02-07: 6 commits2026-02-08: 12 commits2026-02-09: 7 commits2026-02-10: 19 commits2026-02-11: 4 commits2026-02-12: 5 commits2026-02-13: 3 commits2026-02-14: 10 commits2026-02-15: 14 commits2026-02-16: 0 commits2026-02-17: 0 commits2026-02-18: 4 commits2026-02-19: 0 commits2026-02-20: 0 commits2026-02-21: 0 commits2026-02-22: 0 commits2026-02-23: 8 commits2026-02-24: 2 commits2026-02-25: 5 commits2026-02-26: 1 commit2026-02-27: 12 commits2026-02-28: 5 commits2026-03-01: 0 commits2026-03-02: 4 commits2026-03-03: 7 commits2026-03-04: 0 commits2026-03-05: 2 commits2026-03-06: 4 commits2026-03-07: 5 commits2026-03-08: 12 commits2026-03-09: 8 commits2026-03-10: 0 commits2026-03-11: 7 commits2026-03-12: 0 commits2026-03-13: 0 commits2026-03-14: 1 commit2026-03-15: 7 commits2026-03-16: 3 commits2026-03-17: 8 commits2026-03-18: 0 commits2026-03-19: 2 commits2026-03-20: 0 commits2026-03-21: 0 commits2026-03-22: 1 commit2026-03-23: 2 commits2026-03-24: 1 commit2026-03-25: 4 commits2026-03-26: 1 commit2026-03-27: 2 commits2026-03-28: 2 commits2026-03-29: 6 commits2026-03-30: 9 commits2026-03-31: 0 commits2026-04-01: 2 commits2026-04-02: 3 commits2026-04-03: 8 commits2026-04-04: 5 commits2026-04-05: 16 commits2026-04-06: 0 commits2026-04-07: 4 commits2026-04-08: 2 commits2026-04-09: 0 commits2026-04-10: 0 commits2026-04-11: 0 commits2026-04-12: 10 commits2026-04-13: 4 commits2026-04-14: 1 commit2026-04-15: 2 commits2026-04-16: 2 commits2026-04-17: 4 commits2026-04-18: 0 commits2026-04-19: 0 commits2026-04-20: 0 commits2026-04-21: 0 commits2026-04-22: 4 commits2026-04-23: 0 commits2026-04-24: 2 commits2026-04-25: 2 commits2026-04-26: 2 commits2026-04-27: 0 commits2026-04-28: 1 commit2026-04-29: 0 commits2026-04-30: 3 commits2026-05-01: 0 commits2026-05-02: 1 commit2026-05-03: 0 commits2026-05-04: 0 commits2026-05-05: 1 commit2026-05-06: 1 commit2026-05-07: 0 commits2026-05-08: 0 commits2026-05-09: 0 commits2026-05-10: 3 commits2026-05-11: 9 commits2026-05-12: 0 commits2026-05-13: 0 commits2026-05-14: 0 commits2026-05-15: 1 commit2026-05-16: 0 commits2026-05-17: 0 commits2026-05-18: 1 commit2026-05-19: 0 commits2026-05-20: 0 commits2026-05-21: 0 commits2026-05-22: 0 commits2026-05-23: 0 commits2026-05-24: 0 commits2026-05-25: 0 commits2026-05-26: 0 commits2026-05-27: 1 commit2026-05-28: 1 commit2026-05-29: 0 commits2026-05-30: 3 commits2026-05-31: 0 commits2026-06-01: 3 commits2026-06-02: 1 commit2026-06-03: 1 commit2026-06-04: 2 commits2026-06-05: 0 commits2026-06-06: 0 commits2026-06-07: 7 commits2026-06-08: 0 commits2026-06-09: 0 commits2026-06-10: 0 commits2026-06-11: 0 commits2026-06-12: 0 commits2026-06-13: 0 commits2026-06-14: 3 commits2026-06-15: 0 commits2026-06-16: 0 commits2026-06-17: 0 commits2026-06-18: 2 commits2026-06-19: 0 commits2026-06-20: 0 commits2026-06-21: 0 commits2026-06-22: 1 commit2026-06-23: 3 commits2026-06-24: 0 commits2026-06-25: 0 commits2026-06-26: 2 commits2026-06-27: 0 commits2026-06-28: 0 commits2026-06-29: 0 commits2026-06-30: 0 commits2026-07-01: 0 commits2026-07-02: 2 commits2026-07-03: 0 commits2026-07-04: 5 commits2026-07-05: 0 commits2026-07-06: 2 commits2026-07-07: 1 commit2026-07-08: 2 commits2026-07-09: 0 commits2026-07-10: 0 commits2026-07-11: 3 commits2026-07-12: 5 commits2026-07-13: 1 commit2026-07-14: 0 commits2026-07-15: 3 commits2026-07-16: 0 commits2026-07-17: 0 commits2026-07-18: 0 commits2026-07-19: 0 commits2026-07-20: 1 commit2026-07-21: 1 commit2026-07-22: 0 commits2026-07-23: 1 commit2026-07-24: 2 commits2026-07-25: 1 commit2026-07-26: 9 commits2026-07-27: 3 commits2026-07-28: 0 commits2026-07-29: 0 commits2026-07-30: 2 commits2026-07-31: 0 commits2026-08-01: 0 commits2026-08-02: 0 commits2026-08-03: 0 commits2026-08-04: 0 commits2026-08-05: 0 commits2026-08-06: 1 commit2026-08-07: 0 commits2026-08-08: 0 commits2026-08-09: 14 commits2026-08-10: 0 commits2026-08-11: 5 commits2026-08-12: 1 commit2026-08-13: 0 commits2026-08-14: 0 commits2026-08-15: 0 commits2026-08-16: 0 commits2026-08-17: 0 commits2026-08-18: 0 commits2026-08-19: 8 commits2026-08-20: 1 commit2026-08-21: 5 commits2026-08-22: 2 commits2026-08-23: 8 commits2026-08-24: 1 commit2026-08-25: 1 commit2026-08-26: 0 commits2026-08-27: 0 commits2026-08-28: 0 commits2026-08-29: 0 commits2026-08-30: 0 commits2026-08-31: 0 commits2026-09-01: 1 commit2026-09-02: 1 commit2026-09-03: 0 commits2026-09-04: 1 commit2026-09-05: 0 commits2026-09-06: 0 commits2026-09-07: 0 commits2026-09-08: 0 commits2026-09-09: 0 commits2026-09-10: 0 commits2026-09-11: 2 commits2026-09-12: 3 commits2026-09-13: 3 commits2026-09-14: 2 commits2026-09-15: 0 commits2026-09-16: 0 commits2026-09-17: 1 commit2026-09-18: 2 commits2026-09-19: 3 commits2026-09-20: 0 commits2026-09-21: 0 commits2026-09-22: 7 commits2026-09-23: 6 commits2026-09-24: 13 commits2026-09-25: 4 commits2026-09-26: 10 commits2026-09-27: 1 commit2026-09-28: 0 commits2026-09-29: 4 commits2026-09-30: 0 commits2026-10-01: 1 commit2026-10-02: 0 commits2026-10-03: 1 commit2026-10-04: 10 commits2026-10-05: 0 commits2026-10-06: 4 commits2026-10-07: 11 commits2026-10-08: 7 commits2026-10-09: 0 commits2026-10-10: 1 commit2026-10-11: 3 commits2026-10-12: 0 commits2026-10-13: 3 commits2026-10-14: 3 commits2026-10-15: 11 commits2026-10-16: 17 commits2026-10-17: 7 commits2026-10-18: 4 commits2026-10-19: 3 commits2026-10-20: 0 commits2026-10-21: 1 commit2026-10-22: 1 commit2026-10-23: 0 commits2026-10-24: 0 commits2026-10-25: 0 commits2026-10-26: 1 commit2026-10-27: 1 commit2026-10-28: 1 commit2026-10-29: 1 commit2026-10-30: 0 commits2026-10-31: 6 commits2026-11-01: 1 commit2026-11-02: 4 commits2026-11-03: 9 commits2026-11-04: 11 commits2026-11-05: 1 commit2026-11-06: 0 commits2026-11-07: 0 commits2026-11-08: 1 commit2026-11-09: 1 commit2026-11-10: 0 commits2026-11-11: 2 commits2026-11-12: 0 commits2026-11-13: 0 commits2026-11-14: 0 commits2026-11-15: 0 commits2026-11-16: 0 commits2026-11-17: 1 commit2026-11-18: 0 commits2026-11-19: 0 commits2026-11-20: 0 commits2026-11-21: 0 commits
773 commits in the last yearLessMore

Signals and awards

derived from tracked data
  • Landmark project

    104,663 stars

  • Very active

    773 commits in 52 weeks

  • Well documented

    High community health score

  • Permissive license

    BSD-3-Clause

  • Continuous integration

    Automated checks passing

  • Repeat trending

    29 trending appearances

  • Top 10% tracked

    Rank 83 of 1243

What Scrapling does

Scrapling is a highly adaptive web scraping library that combines dynamic DOM parsing with robust fetching and crawling architectures. Its parsing engine learns from website structures, allowing it to automatically relocate elements when page designs change unexpectedly. The built-in fetchers are designed to bypass strict anti-bot systems like Cloudflare Turnstile out of the box, utilizing stealthy headless browser techniques. Additionally, it provides a comprehensive spider framework that scales to handle concurrent, multi-session crawls with automated proxy rotation and pause-resume functionality. It unifies the simplicity of single-page requests with the power of massive distributed crawling in one tool.

Data engineers, web scrapers, and AI developers who need a reliable, bot-resistant tool for extracting web data. It requires Python.

  • Adaptive Parsing: Automatically finds relocated HTML elements when a target website updates its structure.
  • Stealth Fetching: Bypasses modern anti-bot protections like Cloudflare using pre-configured stealth capabilities.
  • Spider Framework: Provides a structured class system for orchestrating concurrent, large-scale web crawls.
  • Proxy Rotation Support: Integrates easily with external proxy providers to manage IP bans and regional restrictions.
  • AI Agent Integration: Includes an MCP server and skills specifically designed for AI agents to perform complex scraping tasks.

Where teams use it

Resilient Data Extraction

Scrape dynamic e-commerce or news websites without breaking scripts every time the CSS changes.

Anti-Bot Evasion

Collect data from highly protected web properties that deploy Cloudflare or Kasada shields.

Large-Scale Crawling

Orchestrate spiders to concurrently map and extract information from thousands of pages across a domain.

AI-Powered Scraping

Equip LLM agents with the ability to fetch and parse live web data using the integrated MCP server.

README

main branch

Scrapling Poster
Effortless Web Scraping for the Modern Web

D4Vinci%2FScrapling | Trendshift Star History Rank
README بالعربية README en Español README em Português (Brasil) README en Français README auf Deutsch 简体中文版自述文件 日本語のREADME Русская версия README 한국어 README
Tests PyPI version Docker Pulls PyPI package downloads Static Badge OpenClaw Skill
Discord X (formerly Twitter) Follow
Supported Python versions

Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP

Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.

Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports.

Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.

What you can do with Scrapling

  • Fetch & interact: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
  • Parse & extract: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
  • Crawl & export: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
  • Scrape with AI or from your terminal: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.

Start with a single page:

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)

Or scale up with a spider:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")

This prints 10 and saves the quotes and authors to quotes.json.

At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies.

Platinum Sponsors

NodeMaven - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. SCRAPLING2X - 2X traffic for Black Friday
Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for Akamai, DataDome, Kasada, and Incapsula. Simple API calls, no browser automation required.
Evomi : residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass.
Scraper API for hassle-free results. MCP and N8N integrations are available.
TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets.
Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off.
Read a full review of Scrapling on The Web Scraping Club (Nov 2025), the #1 newsletter dedicated to Web Scraping.
Byteful supplies ethical residential, mobile, ISP, and datacenter proxies, plus scraping and AI automation tools. Use code SCRAPLING10: 10% off residential proxies.
ProxyLane: 90% fewer CAPTCHAs on your Scrapling runs. 28M real P2P residential devices, 98.7% clean IPs, 195 countries. City/ISP/ZIP targeting, sticky/rotating sessions, traffic never expires. Try 350 MB for $1.95. Code SCRAPEANDTAKE: 25% off.
LundProxy: Stop paying per GB. Unlimited residential traffic, 100M+ IPs, 190+ countries. Pay by bandwidth. Request a free 50 Mbps residential proxies trial.
Resolvr: Scraping at scale means CAPTCHAs at scale. Resolvr solves reCAPTCHA and Cloudflare challenges on real devices through one API. Free beta access is available now.
hCaptcha in your Scrapling runs? NoneCap returns valid hCaptcha tokens, Enterprise sitekeys included, through one API. 1,300 free credits on signup.
$1/GB at DataImpulse. Ethical first-party proxies. 90M+ IPs, 195+ countries. Residential, datacenter, mobile. Pay-as-you-go = no subscription. Traffic never expires.

Do you want to show your ad here? Click here

Sponsors


Do you want to show your ad here? Click here and choose the tier that suits you!


Key Features

Spiders - A Full Crawling Framework

  • 🕷️ Scrapy-like Spider API: Define spiders with start_urls, async parse callbacks, and Request/Response objects.
  • ⚡ Concurrent Crawling: Configurable concurrency limits, per-domain throttling, and download delays.
  • 🔄 Multi-Session Support: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
  • 💾 Pause & Resume: Save pending requests and seen URLs with crawldir. Press Ctrl+C to stop gracefully and restart with the same directory.
  • 📡 Streaming Mode: Stream scraped items as they arrive via async for item in spider.stream() with real-time stats - ideal for UI, pipelines, and long-running crawls.
  • 🛡️ Blocked Request Detection: Automatic detection and retry of blocked requests with customizable logic.
  • 🚦 AutoThrottle: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what Retry-After asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
  • 🤖 Robots.txt Compliance: Optional robots_txt_obey flag that respects Disallow, Crawl-delay, and Request-rate directives with per-domain caching.
  • 🧪 Development Mode: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your parse() logic without re-hitting the target servers.
  • 🧩 Ready-made Spider Templates: Skip the boilerplate with CrawlSpider for rule-based link following, SitemapSpider for sitemap/robots.txt-driven crawls, XMLFeedSpider/CSVFeedSpider for iterating XML/RSS and CSV feeds, and ShopifySpider to collect products from public Shopify stores, one item per variant.
  • 🔗 Link Extraction: A standalone LinkExtractor primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
  • 📦 Built-in Export: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with result.items.to_json(), to_jsonl(), to_csv(), and to_xml().

Advanced Websites Fetching with Session Support

  • HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
  • Dynamic Loading: Fetch dynamic websites with full browser automation through the StealthyFetcher class supporting Playwright's Chromium and Google's Chrome.
  • Anti-bot Bypass: Advanced stealth capabilities with StealthyFetcher and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
  • Session Management: Persistent session support with FetcherSession and StealthySession classes for cookie and state management across requests.
  • Proxy Rotation: Built-in ProxyRotator with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
  • Domain & Ad Blocking: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
  • DNS Leak Prevention: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
  • Remote Browsers: Instead of launching a browser locally, connect to one that's already running through CDP with cdp_url, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with executable_path.
  • Background API Capture: Enable record_requests=True on a browser session, then search its saved responses with session.network.search(limit=None). Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies.
  • Async Support: Complete async support across all fetchers and dedicated async session classes.

Parsing and Data Extraction

  • 🔄 Smart Element Tracking: Relocate elements after website changes using intelligent similarity algorithms.
  • 🎯 Smart Flexible Selection: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
  • 🔍 Find Similar Elements: Automatically locate elements similar to found elements.

AI Features

  • 🤖 MCP Server: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA is faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. (demo video)
  • 🧠 Agent Skill: A ready-to-install Agent Skill that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
  • 📚 RAG-ready Markdown: Turn any page into clean, sanitized, LLM-ready Markdown with one line (page.markdown()), or crawl a whole website into a Markdown corpus with the ContentSpider template, all without an LLM in the loop. (docs)

High-Performance & battle-tested Architecture

  • 🚀 Lightning Fast: Optimized performance outperforming most Python scraping libraries.
  • 🔋 Memory Efficient: Optimized data structures and lazy loading for a minimal memory footprint.
  • ⚡ Fast JSON Serialization: 10x faster than the standard library.
  • 🏗️ Battle tested: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.

Developer/Web Scraper Friendly Experience

  • 🎯 Interactive Web Scraping Shell: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
  • 🚀 Use it directly from the Terminal: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
  • 🛠️ Rich Navigation API: Advanced DOM traversal with parent, sibling, and child navigation methods.
  • 🧬 Enhanced Text Processing: Built-in regex, cleaning methods, and optimized string operations.
  • 📝 Auto Selector Generation: Generate robust CSS/XPath selectors for any element.
  • 🔌 Familiar API: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
  • 🤝 Drop-in Scrapy Integration: Already invested in Scrapy? Decorate any callback with scrapling_response to parse the responses you already fetch with Scrapling's parser, no rewrite needed.
  • 📘 Complete Type Coverage: Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with PyRight and MyPy with each change.
  • 🔋 Ready Docker image: With each release, a Docker image containing all browsers is automatically built and pushed.

Getting Started

Let's give you a quick glimpse of what Scrapling can do without deep diving.

Basic Usage

HTTP requests with session support

from scrapling.fetchers import Fetcher, FetcherSession

with FetcherSession(impersonate='chrome') as session:  # Use latest version of Chrome's TLS fingerprint
    page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

# Or use one-off requests
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

Advanced stealth mode

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('#padded_content a').getall()

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()

Full browser automation

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, disable_resources=False, network_idle=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
    data = page.xpath('//span[@class="text"]/text()').getall()  # XPath selector if you prefer it

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()

Spiders

Build full crawlers with concurrent requests, multiple session types, and pause/resume:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10
    
    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }
            
        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Use multiple session types in a single spider:

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]
    
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
    
    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            # Route protected pages through the stealth session
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)  # explicit callback

Pause and resume long crawls with checkpoints by running the spider like this:

QuotesSpider(crawldir="./crawl_data").start()

With crawldir set, press Ctrl+C to save pending requests and seen URLs. Restart with the same crawldir to resume. Save scraped items separately.

Or skip writing the crawling logic altogether with the ready-made templates, like pulling an entire Shopify store's catalog:

from scrapling.spiders import ShopifySpider

class MyStore(ShopifySpider):
    target_website = "example.com"

result = MyStore().start()  # Every product in the store, one item per variant

The template reads products exposed through public collection endpoints, one item per variant.

Advanced Parsing & Navigation

from scrapling.fetchers import Fetcher

# Rich element selection and navigation
page = Fetcher.get('https://quotes.toscrape.com/')

# Get quotes with multiple selection methods
quotes = page.css('.quote')  # CSS selector
quotes = page.xpath('//div[@class="quote"]')  # XPath
quotes = page.find_all('div', {'class': 'quote'})  # BeautifulSoup-style
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # and so on...
# Find element by text content
quotes = page.find_by_text('Albert Einstein', first_match=False)

# Advanced navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
quote_texts = page.css('.quote').css('.text::text').getall()  # Chained selectors
first_quote = page.css('.quote')[0]
author = first_quote.css('.author::text').get()
next_quote = first_quote.next
parent_container = first_quote.parent

# Element relationships and similarity
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements

You can use the parser right away if you don't want to fetch websites like below:

from scrapling.parser import Selector

page = Selector("<html>...</html>")

And it works precisely the same way!

Async Session Management Examples

import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession

async def main():
    async with FetcherSession(http3=True) as http_session:  # `FetcherSession` is context-aware and can work in both sync/async patterns
        page1 = await http_session.get('https://quotes.toscrape.com/')
        page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
        print(len(page1.css('.quote')), len(page2.css('.quote')))

    async with AsyncStealthySession(max_pages=2) as browser_session:
        urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
        pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
        print([len(page.css('.quote')) for page in pages])

asyncio.run(main())

CLI & Interactive Shell

Scrapling includes a powerful command-line interface:

asciicast

Launch the interactive Web Scraping shell

pip install "scrapling[shell]"
scrapling shell

Extract pages to a file directly without programming. Use --css-selector to pick the content, or --ai-targeted to select and sanitize the body content. Use .txt for text, .md for Markdown, or .html for HTML.

scrapling extract get 'https://quotes.toscrape.com/' quotes.md
scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare

Note

There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation here

Performance Benchmarks

Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.

Text Extraction Speed Test (5000 nested elements)

# Library Time (ms) vs Scrapling
1 Scrapling 1.99 1.0x
2 Parsel/Scrapy 2.06 1.035
3 Raw Lxml 2.56 1.286
4 PyQuery 23.98 ~12x
5 Selectolax 197.02 ~99x
6 MechanicalSoup 1545.15 ~776.5x
7 BS4 with Lxml 1562.1 ~785.0x
8 BS4 with html5lib 3412.73 ~1714.9x

Element Similarity & Text Search Performance

Scrapling's adaptive element finding capabilities significantly outperform alternatives:

Library Time (ms) vs Scrapling
Scrapling 2.3 1.0x
AutoScraper 12.58 5.47x

All benchmarks represent averages of 100+ runs. See benchmarks.py for methodology.

Installation

Scrapling requires Python 3.10 or higher:

pip install scrapling

Important

This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from scrapling.fetchers or scrapling.spiders, like in the examples above, will raise ModuleNotFoundError with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.

Optional Dependencies

  1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:

    pip install "scrapling[fetchers]"
    
    scrapling install           # normal install
    scrapling install  --force  # force reinstall

    This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.

    Or you can install them from the code instead of running a command like this:

    from scrapling.cli import install
    
    install([], standalone_mode=False)          # normal install
    install(["--force"], standalone_mode=False) # force reinstall
  2. Extra features:

    • Install the MCP server feature:
      pip install "scrapling[ai]"
    • Install dependencies for (building RAG systems):
      pip install "scrapling[rag]"
    • Install shell features (Web Scraping shell and the extract command):
      pip install "scrapling[shell]"
    • Install everything:
      pip install "scrapling[all]"

    Run scrapling install when you need browser fetchers. HTTP requests do not need a browser installation.

Docker

You can also install a Docker image with all extras and browsers with the following command from DockerHub:

docker pull pyd4vinci/scrapling

Or download it from the GitHub registry:

docker pull ghcr.io/d4vinci/scrapling:latest

This image is automatically built and pushed using GitHub Actions and the repository's main branch.

Contributing

We welcome contributions! Please read our contributing guidelines before getting started.

Disclaimer

Caution

This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.

🎓 Citations

If you have used our library for research purposes please quote us with the following reference:

  @misc{scrapling,
    author = {Karim Shoair},
    title = {Scrapling},
    year = {2024},
    url = {https://github.com/D4Vinci/Scrapling},
    note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
  }

License

This work is licensed under the BSD-3-Clause License.

Acknowledgments

This project includes code adapted from:

  • Parsel (BSD License)-Used for translator submodule

Designed & crafted with ❤️ by Karim Shoair.

View on GitHub

Recent activity

commits and pull requests

Releases and announcements

55 total
  1. Release v0.5.1v0.5.1Nov 4, 2026

    **Small version bump, big browser upgrade! This version lets AI agents follow motion and control drag and drop, adds dedicated remote browser sessions, and brings fixes across parsing, spiders, proxies, and the shell 🚀** **With this update, we extend the limits of current AI models** 🔥 > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** > [!WARNING] > **This release introduces breaking changes to remote browser connections. Connecting to remote browsers now requires a new browser session class. Check the [remote-session migration guide](https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html#move-cdp-connections-to-remote-sessions) before updating.** ## 🚀 New Stuff and quality of life changes - **AI agents can now follow motion and take finer control of the mouse** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)): - **Watch the page in motion**: record a short video and share it as a sequence of frames with timestamps to help the agent understand animations, moving controls, and changes that a single screenshot can miss. Pair this with the existing ability to chain actions!

  2. Release v0.5v0.5Oct 17, 2026

    **Scrapling v0.5 upgrades almost everything: fetching, parsing, spiders, browser automation, AI tools, and docs in ten languages 🚀** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** > [!WARNING] > **This release introduces a lot of breaking changes that don't fit here. Check the [Migrating to v0.5 guide](https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html) before updating.** ## 🚀 New Stuff and quality of life changes - **A much more capable MCP server for AI agents that can solve almost any CAPTCHA** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)): - **Complete browser workflows**: fill forms, click, hover, scroll, use the keyboard, handle dialogs, and wait for results in one ordered sequence. Supports different field types and slower interactions, with actions stopping at the first error. - **Read and search the current page without reloading it**: extract HTML, Markdown, or text from a whole page or one element. Searchable page snapshots show element roles, names, and positions, with focused views that retain the surrounding context. - **Run custom JavaScript and take focused

  3. Release v0.4.15v0.4.15Aug 23, 2026

    **One of the biggest releases this year: a reworked MCP server, RAG-ready Markdown in one line, an improved Cloudflare solver, and browser tabs that stay open for automation 🚀** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** > [!WARNING] > **This release introduces breaking changes to the MCP server. Check the [breaking changes](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html#breaking-changes) section before updating.** ## 🚀 New Stuff and quality of life changes - **Browser tabs now stay open and get reused across requests** (Check the [docs](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html#session-management)): - All browser sessions keep their tabs after a request, and the next request reuses a free tab instead of opening a new one. - Every request re-applies its own settings (timeouts, headers, resource blocking) to the tab it gets, so nothing leaks between requests. - Tabs that hit an error are closed and replaced, and the new `close_pages()` method closes every open tab. - The page you fetched stays loaded, so a `page_setup` function on the next request runs on it before

  4. Release v0.4.14v0.4.14Aug 10, 2026

    **A quick maintenance release to fix installation with `uv` 🔧** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** ## 🐛 Bug Fixes - **Fixed `uv` refusing to install v0.4.13 by default and silently falling back to an older version**. The previous release required a prerelease version of `curl_cffi`, which `uv` doesn't allow unless explicitly enabled. All dependencies now resolve to stable releases. (Fixes [#407](https://github.com/D4Vinci/Scrapling/issues/407)) _🙏 Special thanks to the community for all the continuous testing and feedback_ --- # Big shoutout to our Platinum Sponsors <div style="text-align: center;"> <a href="https://go.nodemaven.com/scraplingjuly" target="_blank" title="Proxies with the Highest IP Scores"> <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/NodeMaven.jpg" width="240" height="100"> </a> <a href="https://proxidize.com/?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=d4vinci" target="_blank" title="Clean Proxies with No Nonsense."> <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/

  5. Release v0.4.13v0.4.13Aug 9, 2026

    **A new update bringing feed spiders and a smarter MCP server 🎉** > [!NOTE] > - **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** > - This will most likely be the last update before the major updates in v0.5 ## 🚀 New Stuff and quality of life changes - **New feed spider templates**. `XMLFeedSpider` iterates over the nodes of any XML feed (RSS, Atom, product feeds, etc.), and `CSVFeedSpider` iterates over CSV rows as dictionaries. Both decompress gzipped feeds automatically. (Check the [docs](https://scrapling.readthedocs.io/en/latest/spiders/generic-templates.html)) ```python from scrapling.spiders import XMLFeedSpider class RSSSpider(XMLFeedSpider): name = "rss" start_urls = ["https://example.com/feed.xml"] async def parse_node(self, response, node): yield {"title": node.findtext("title"), "link": node.findtext("link")} result = RSSSpider().start() ``` - **Upgraded the MCP server to MCP SDK v2 and made it smarter**. The server now ships instructions that teach your AI agent how to use the tools efficiently; every tool declares annotations so clients like Claude Code can auto-approve the re

Code frequency

additions and deletions
+83.7K-83.7KWeek of 2025-11-23: +1,200 linesWeek of 2025-11-23: -1,865 linesWeek of 2025-11-30: +11 linesWeek of 2025-11-30: -16 linesWeek of 2025-12-07: +15 linesWeek of 2025-12-07: -0 linesWeek of 2025-12-14: +97 linesWeek of 2025-12-14: -65 linesWeek of 2025-12-21: +1,474 linesWeek of 2025-12-21: -1,825 linesWeek of 2025-12-28: +633 linesWeek of 2025-12-28: -707 linesWeek of 2026-01-04: +110 linesWeek of 2026-01-04: -32 linesWeek of 2026-01-11: +1,187 linesWeek of 2026-01-11: -235 linesWeek of 2026-01-18: +3,038 linesWeek of 2026-01-18: -743 linesWeek of 2026-01-25: +3 linesWeek of 2026-01-25: -2 linesWeek of 2026-02-01: +1,678 linesWeek of 2026-02-01: -693 linesWeek of 2026-02-08: +9,907 linesWeek of 2026-02-08: -8,403 linesWeek of 2026-02-15: +1,676 linesWeek of 2026-02-15: -167 linesWeek of 2026-02-22: +662 linesWeek of 2026-02-22: -679 linesWeek of 2026-03-01: +1,206 linesWeek of 2026-03-01: -212 linesWeek of 2026-03-08: +5,382 linesWeek of 2026-03-08: -440 linesWeek of 2026-03-15: +892 linesWeek of 2026-03-15: -260 linesWeek of 2026-03-22: +825 linesWeek of 2026-03-22: -697 linesWeek of 2026-03-29: +2,142 linesWeek of 2026-03-29: -377 linesWeek of 2026-04-05: +808 linesWeek of 2026-04-05: -427 linesWeek of 2026-04-12: +4,797 linesWeek of 2026-04-12: -319 linesWeek of 2026-04-19: +591 linesWeek of 2026-04-19: -397 linesWeek of 2026-04-26: +185 linesWeek of 2026-04-26: -109 linesWeek of 2026-05-03: +103 linesWeek of 2026-05-03: -0 linesWeek of 2026-05-10: +1,923 linesWeek of 2026-05-10: -183 linesWeek of 2026-05-17: +0 linesWeek of 2026-05-17: -0 linesWeek of 2026-05-24: +125 linesWeek of 2026-05-24: -57 linesWeek of 2026-05-31: +70 linesWeek of 2026-05-31: -118 linesWeek of 2026-06-07: +199 linesWeek of 2026-06-07: -153 linesWeek of 2026-06-14: +59 linesWeek of 2026-06-14: -48 linesWeek of 2026-06-21: +480 linesWeek of 2026-06-21: -67 linesWeek of 2026-06-28: +618 linesWeek of 2026-06-28: -22 linesWeek of 2026-07-05: +147 linesWeek of 2026-07-05: -366 linesWeek of 2026-07-12: +609 linesWeek of 2026-07-12: -131 linesWeek of 2026-07-19: +237 linesWeek of 2026-07-19: -177 linesWeek of 2026-07-26: +1,611 linesWeek of 2026-07-26: -113 linesWeek of 2026-08-02: +0 linesWeek of 2026-08-02: -10 linesWeek of 2026-08-09: +1,446 linesWeek of 2026-08-09: -353 linesWeek of 2026-08-16: +1,870 linesWeek of 2026-08-16: -760 linesWeek of 2026-08-23: +2,714 linesWeek of 2026-08-23: -495 linesWeek of 2026-08-30: +206 linesWeek of 2026-08-30: -103 linesWeek of 2026-09-06: +68 linesWeek of 2026-09-06: -25 linesWeek of 2026-09-13: +1,367 linesWeek of 2026-09-13: -75 linesWeek of 2026-09-20: +9,158 linesWeek of 2026-09-20: -3,128 linesWeek of 2026-09-27: +1,117 linesWeek of 2026-09-27: -1,107 linesWeek of 2026-10-04: +11,187 linesWeek of 2026-10-04: -6,576 linesWeek of 2026-10-11: +83,717 linesWeek of 2026-10-11: -5,351 linesWeek of 2026-10-18: +4,296 linesWeek of 2026-10-18: -152 linesWeek of 2026-10-25: +1,459 linesWeek of 2026-10-25: -322 linesWeek of 2026-11-01: +2,595 linesWeek of 2026-11-01: -590 linesWeek of 2026-11-08: +38 linesWeek of 2026-11-08: -10 linesWeek of 2026-11-15: +0 linesWeek of 2026-11-15: -103 linesNov 23, 2025Nov 15, 2026
+165.9K lines added, -39.3K removed over the last year.

Commits per week

last 52 weeks
600Week of 2025-11-23: 19 commitsWeek of 2025-11-30: 4 commitsWeek of 2025-12-07: 2 commitsWeek of 2025-12-14: 11 commitsWeek of 2025-12-21: 19 commitsWeek of 2025-12-28: 35 commitsWeek of 2026-01-04: 8 commitsWeek of 2026-01-11: 21 commitsWeek of 2026-01-18: 20 commitsWeek of 2026-01-25: 2 commitsWeek of 2026-02-01: 15 commitsWeek of 2026-02-08: 60 commitsWeek of 2026-02-15: 18 commitsWeek of 2026-02-22: 33 commitsWeek of 2026-03-01: 22 commitsWeek of 2026-03-08: 28 commitsWeek of 2026-03-15: 20 commitsWeek of 2026-03-22: 13 commitsWeek of 2026-03-29: 33 commitsWeek of 2026-04-05: 22 commitsWeek of 2026-04-12: 23 commitsWeek of 2026-04-19: 8 commitsWeek of 2026-04-26: 7 commitsWeek of 2026-05-03: 2 commitsWeek of 2026-05-10: 13 commitsWeek of 2026-05-17: 1 commitsWeek of 2026-05-24: 5 commitsWeek of 2026-05-31: 7 commitsWeek of 2026-06-07: 7 commitsWeek of 2026-06-14: 5 commitsWeek of 2026-06-21: 6 commitsWeek of 2026-06-28: 7 commitsWeek of 2026-07-05: 8 commitsWeek of 2026-07-12: 9 commitsWeek of 2026-07-19: 6 commitsWeek of 2026-07-26: 14 commitsWeek of 2026-08-02: 1 commitsWeek of 2026-08-09: 20 commitsWeek of 2026-08-16: 16 commitsWeek of 2026-08-23: 10 commitsWeek of 2026-08-30: 3 commitsWeek of 2026-09-06: 5 commitsWeek of 2026-09-13: 11 commitsWeek of 2026-09-20: 40 commitsWeek of 2026-09-27: 7 commitsWeek of 2026-10-04: 33 commitsWeek of 2026-10-11: 44 commitsWeek of 2026-10-18: 9 commitsWeek of 2026-10-25: 10 commitsWeek of 2026-11-01: 26 commitsWeek of 2026-11-08: 4 commitsWeek of 2026-11-15: 1 commitsNov 23, 2025Nov 15, 2026
773 commits in the last 52 weeks.

When work happens

weekday and hour
SunMonTueWedThuFriSat036912151821Sun 0:00 — 13 commitsSun 1:00 — 12 commitsSun 2:00 — 19 commitsSun 3:00 — 23 commitsSun 4:00 — 26 commitsSun 5:00 — 11 commitsSun 6:00 — 6 commitsSun 7:00 — 3 commitsSun 8:00 — 1 commitsSun 9:00 — 1 commitsSun 10:00 — 0 commitsSun 11:00 — 1 commitsSun 12:00 — 3 commitsSun 13:00 — 12 commitsSun 14:00 — 7 commitsSun 15:00 — 16 commitsSun 16:00 — 33 commitsSun 17:00 — 8 commitsSun 18:00 — 29 commitsSun 19:00 — 15 commitsSun 20:00 — 25 commitsSun 21:00 — 18 commitsSun 22:00 — 27 commitsSun 23:00 — 28 commitsMon 0:00 — 38 commitsMon 1:00 — 16 commitsMon 2:00 — 25 commitsMon 3:00 — 35 commitsMon 4:00 — 5 commitsMon 5:00 — 8 commitsMon 6:00 — 14 commitsMon 7:00 — 0 commitsMon 8:00 — 0 commitsMon 9:00 — 2 commitsMon 10:00 — 0 commitsMon 11:00 — 5 commitsMon 12:00 — 14 commitsMon 13:00 — 10 commitsMon 14:00 — 8 commitsMon 15:00 — 10 commitsMon 16:00 — 19 commitsMon 17:00 — 9 commitsMon 18:00 — 5 commitsMon 19:00 — 2 commitsMon 20:00 — 5 commitsMon 21:00 — 7 commitsMon 22:00 — 3 commitsMon 23:00 — 2 commitsTue 0:00 — 10 commitsTue 1:00 — 12 commitsTue 2:00 — 20 commitsTue 3:00 — 10 commitsTue 4:00 — 5 commitsTue 5:00 — 5 commitsTue 6:00 — 7 commitsTue 7:00 — 0 commitsTue 8:00 — 0 commitsTue 9:00 — 0 commitsTue 10:00 — 1 commitsTue 11:00 — 9 commitsTue 12:00 — 3 commitsTue 13:00 — 8 commitsTue 14:00 — 7 commitsTue 15:00 — 11 commitsTue 16:00 — 11 commitsTue 17:00 — 7 commitsTue 18:00 — 15 commitsTue 19:00 — 5 commitsTue 20:00 — 10 commitsTue 21:00 — 25 commitsTue 22:00 — 25 commitsTue 23:00 — 23 commitsWed 0:00 — 18 commitsWed 1:00 — 16 commitsWed 2:00 — 22 commitsWed 3:00 — 22 commitsWed 4:00 — 9 commitsWed 5:00 — 10 commitsWed 6:00 — 3 commitsWed 7:00 — 2 commitsWed 8:00 — 0 commitsWed 9:00 — 3 commitsWed 10:00 — 0 commitsWed 11:00 — 3 commitsWed 12:00 — 14 commitsWed 13:00 — 5 commitsWed 14:00 — 8 commitsWed 15:00 — 11 commitsWed 16:00 — 11 commitsWed 17:00 — 4 commitsWed 18:00 — 6 commitsWed 19:00 — 13 commitsWed 20:00 — 13 commitsWed 21:00 — 16 commitsWed 22:00 — 4 commitsWed 23:00 — 10 commitsThu 0:00 — 16 commitsThu 1:00 — 29 commitsThu 2:00 — 18 commitsThu 3:00 — 10 commitsThu 4:00 — 11 commitsThu 5:00 — 1 commitsThu 6:00 — 0 commitsThu 7:00 — 1 commitsThu 8:00 — 0 commitsThu 9:00 — 1 commitsThu 10:00 — 1 commitsThu 11:00 — 2 commitsThu 12:00 — 3 commitsThu 13:00 — 8 commitsThu 14:00 — 5 commitsThu 15:00 — 3 commitsThu 16:00 — 11 commitsThu 17:00 — 9 commitsThu 18:00 — 0 commitsThu 19:00 — 7 commitsThu 20:00 — 11 commitsThu 21:00 — 6 commitsThu 22:00 — 2 commitsThu 23:00 — 14 commitsFri 0:00 — 9 commitsFri 1:00 — 15 commitsFri 2:00 — 13 commitsFri 3:00 — 18 commitsFri 4:00 — 8 commitsFri 5:00 — 3 commitsFri 6:00 — 4 commitsFri 7:00 — 1 commitsFri 8:00 — 1 commitsFri 9:00 — 2 commitsFri 10:00 — 2 commitsFri 11:00 — 3 commitsFri 12:00 — 0 commitsFri 13:00 — 2 commitsFri 14:00 — 8 commitsFri 15:00 — 5 commitsFri 16:00 — 7 commitsFri 17:00 — 9 commitsFri 18:00 — 4 commitsFri 19:00 — 10 commitsFri 20:00 — 6 commitsFri 21:00 — 6 commitsFri 22:00 — 10 commitsFri 23:00 — 4 commitsSat 0:00 — 10 commitsSat 1:00 — 10 commitsSat 2:00 — 9 commitsSat 3:00 — 15 commitsSat 4:00 — 6 commitsSat 5:00 — 1 commitsSat 6:00 — 1 commitsSat 7:00 — 0 commitsSat 8:00 — 0 commitsSat 9:00 — 0 commitsSat 10:00 — 0 commitsSat 11:00 — 3 commitsSat 12:00 — 1 commitsSat 13:00 — 5 commitsSat 14:00 — 13 commitsSat 15:00 — 11 commitsSat 16:00 — 29 commitsSat 17:00 — 21 commitsSat 18:00 — 15 commitsSat 19:00 — 15 commitsSat 20:00 — 19 commitsSat 21:00 — 20 commitsSat 22:00 — 20 commitsSat 23:00 — 7 commits
Commit volume by weekday and hour (UTC). Larger dots mean more commits.

Who is committing

last 52 weeks
Maintainer commits840 (89%)
Community commits109 (11%)

949 commits in total over the last year.

DateListRankStars gained
Jun 4, 2026daily#18+23
Jun 3, 2026daily#22+26
Jun 2, 2026daily#12+37
Jun 1, 2026daily#8+52
May 31, 2026daily#23+60
May 30, 2026daily#17+75
May 23, 2026daily#18+101
May 21, 2026daily#14+62
May 8, 2026daily#23+50
May 7, 2026daily#10+87
May 6, 2026daily#20+64
May 5, 2026daily#13+73
May 4, 2026daily#11+132
May 3, 2026daily#7+274
May 2, 2026daily#5+335
  • public-apis/public-apis

    A collective list of free APIs

    500.5K stars · Python

  • morluto/rea

    Reverse engineer anything with agents, from app behavior down to native binaries.

    471.6K stars · TypeScript

  • openclaw/openclaw

    The AI that really does things. Any OS. Any Platform. The lobster way. 🦞

    394.9K stars · TypeScript

  • donnemartin/system-design-primer

    Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

    378.7K stars · Python

  • obra/superpowers

    An agentic skills framework & software development methodology that works.

    322.9K stars · Shell

  • affaan-m/ECC

    The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

    307.9K stars · JavaScript