Shopify Product Data Scraper
A Scrapy-based scraper that collects a Shopify store's full product catalogue from its public products API and turns it into clean, structured outputs on a daily schedule — monitored end to end, with an instant email if a run ever fails.
Architecture
Problem
Shopify stores expose their full catalogue through a public products.json endpoint, but the data is deeply nested, paginated, and mixes product-level and variant-level detail — not something you can use directly for analysis or monitoring without reshaping it first.
Solution
A Scrapy spider pages through the store's products.json API until the catalogue is exhausted. A pipeline then produces three complementary outputs from the same crawl: the raw catalogue as scraped, a lightweight product index (id, type, title, key dates), and a flattened table of every variant with its pricing and stock detail.
Automation
The scrape runs automatically once a day via APScheduler, alongside the other scheduled jobs in this pipeline, so the catalogue snapshot stays current without a manual step.
Reliability
The spider respects the site's robots.txt rules and stops as soon as the API reports no further products, rather than relying on a hardcoded page count.
Monitoring & alerts
Every run reports its outcome — success, failure, and duration — to AWS CloudWatch alongside the pipeline's other scrapers, so fail rates and last-run times are visible at a glance. If a run actually errors out, it sends an instant email via AWS SNS rather than waiting to be noticed.
Infrastructure
Output is written to the same AWS S3 bucket used by the project's other pipelines, provisioned as code with AWS CDK.