Build a Web Scraper
Write a resilient web scraper with rate limiting, error handling, and structured data extraction.
The Prompt
Build a web scraper. Include: polite rate limiting, HTML parsing with selectors, pagination, error handling with retries, structured JSON output, anti-detection measures, page caching to avoid re-scraping. Language: [PYTHON (Playwright/BeautifulSoup) / NODE (Puppeteer/Cheerio)] Target data: [DESCRIBE WHAT TO SCRAPE] Dynamic JS content: [YES / NO] Output: [JSON / CSV]
Example Output
Python Playwright scraper with 2 req/second limit, rotating user agents, exponential backoff on 429s. Outputs structured JSON with timestamp and source URL. Disk cache to avoid re-scraping during dev.
FAQ
Which AI model is best for Build a Web Scraper?
Claude Sonnet 4 — writes careful scrapers with rate limiting.
How do I use the Build a Web Scraper prompt?
Copy the prompt, replace the [BRACKETED] placeholders with your specific information, and paste into your preferred AI assistant (ChatGPT, Claude, Gemini, etc.). Python Playwright scraper with 2 req/second limit, rotating user agents, exponential backoff on 429s. Outputs structured JSON with timestamp and source URL. Disk cache to avoid re-scraping during dev.
Model Recommendation
Claude Sonnet 4 — writes careful scrapers with rate limiting.