Browser-Automated Catalogue Scraper
A Playwright-driven scraper that collects a large retailer's full product catalogue directly from its rendered storefront, for a site with no public product API to call instead — monitored end to end, with an instant email if a run ever fails.
Architecture
Problem
Not every retailer exposes a public product API. This site's catalogue is only reachable through its rendered storefront pages, spread across thousands of paginated category listings, which rules out simply calling a JSON endpoint the way an API-based scrape would.
Solution
A headless browser driven by Playwright loads each category's paginated listing pages directly, and the rendered HTML is parsed to pull structured product data — price, RRP, stock status, brand and category — straight from the listing cards, without needing to open every individual product page.
Automation
The scrape runs automatically once a day via APScheduler, alongside the other scheduled jobs in this pipeline, so the catalogue snapshot stays current without a manual step.
Reliability
The scraper follows the site's own pagination boundary rather than a hardcoded page count, stopping cleanly once a category is exhausted, and respects the site's robots.txt rules throughout.
Monitoring & alerts
Every run reports its outcome — success, failure, and duration — to AWS CloudWatch alongside the pipeline's other scrapers, so fail rates and last-run times are visible at a glance. If a run actually errors out, it sends an instant email via AWS SNS rather than waiting to be noticed.
Infrastructure
Output is written to the same AWS S3 bucket used by the project's other pipelines, provisioned as code with AWS CDK.