Property Data Collection Pipeline
An automated pipeline that collects property listing data, structures it, and stores it reliably — with monitoring built in, so an instant email goes out the moment a scraper actually fails.
Architecture
Problem
Property listings change constantly and are spread across multiple websites. Collecting this data by hand doesn't scale, and once it's collected it needs to be stored somewhere reliable and refreshed automatically, without someone needing to babysit it.
Solution
A Python-based scraper collects property listing data from source websites and models it using structured dataclasses, giving every listing a consistent shape before it's stored. Raw results are stored in AWS S3.
Automation
Collection jobs are scheduled and run automatically using APScheduler, rather than being triggered manually — new data is collected on a set schedule without any manual step.
Monitoring & alerts
Every run reports its outcome — success, failure, and duration — to AWS CloudWatch, so fail rates and last-run times are visible at a glance rather than hidden in server logs. When a scraper actually errors out, it sends an instant email via AWS SNS — so a failure gets noticed within minutes, not days later when someone happens to check.
Infrastructure
The AWS infrastructure this project depends on is defined and provisioned as code using AWS CDK, so the environment is reproducible rather than manually configured through a console.