Property Market Analysis
A batch pipeline that turns raw listings from the collection pipeline's S3 bucket into a clean, canonical dataset, with day-over-day tracking of new listings, removed listings, and price changes.
Architecture
Problem
Raw scraped listings sitting in S3 as JSON aren't directly comparable — different sources use different field names, prices are formatted strings, and there's no shared shape to analyse across them.
Solution
A daily batch job normalises each source's raw JSON into one canonical schema — price, bedrooms, property type, postcode area, estate agent — and writes it as immutable, partitioned Parquet using DuckDB. It's incremental: only newly-scraped days are ever reprocessed.
Historical analysis
Because the pipeline keeps every day's snapshot, it can diff consecutive scrapes to detect new listings, listings that disappear from the market, and asking-price changes — not just a static catalogue.
Precomputed, not queried live
The batch job precomputes small JSON datasets — price distributions, agent/region breakdowns, market-activity time series — ready for any downstream consumer to read directly, without scanning raw scrape data or running an always-on database.
Extensibility
The loader keys off the same dated S3 layout the collection pipeline already writes to, so it automatically picks up new scrapes as they land each day without any changes to the analysis code.