The Unglamorous 80%

Nobody starts a real estate data project excited about address normalization. Teams get excited about the scraping, beating anti-bot systems and pulling thousands of listings a night, and about the analysis at the far end. Then reality arrives. The same house appears three times under three different addresses. Half the lot sizes are in acres and half in square feet. The "price" column contains $1,250/mo, 1250000, and Contact for pricing. A quarter of the "active" listings sold five weeks ago.

In practice, cleaning and normalization eat the majority of the engineering effort in any serious property data pipeline. The 80/20 framing gets thrown around a lot, and for once our experience actually matches it. It's also where the value gets decided. A dashboard built on duplicated, stale, inconsistently formatted records isn't slightly wrong; it's confidently wrong, which is worse, because the first stakeholder who spot-checks a bad number stops trusting all of them.

What follows is the field guide to that 80%: the specific failure modes, what to do about each, and a QA checklist you can run against any pipeline, your own or a vendor's.

Addresses First, Everything Else Second

The address is the natural key of real estate data, and raw scraped addresses are chaos. The same property arrives as:

• 123 Main St W Apt 4B, Springfield, IL 62704
• 123 West Main Street #4B, Springfield, Illinois
• 123 W MAIN ST UNIT 4B SPRINGFIELD IL

String matching sees three properties. Until addresses are standardized, everything downstream fails: dedup, cross-platform joins, county-record enrichment, all of it.

The gold standard is USPS CASS-style standardization. Parse the address into components (number, pre-directional, street name, suffix, unit type, unit number, city, state, ZIP), then normalize each against USPS conventions: Street becomes ST, West becomes W, Apartment/Apt/# become UNIT. Certified CASS engines validate against the USPS database itself; the usaddress Python library or APIs like Smarty get you most of the way without certification, and for most analytics work that's plenty.

Three rules we treat as non-negotiable. Store both the raw scraped address and the normalized form, because normalization has bugs and you will want to reprocess. Never compare raw strings; normalize first, always. And treat the unit number as first-class. Dropping unit numbers merges every condo in a building into one record, and this exact bug is the first thing we check when a client's inventory count looks suspiciously low. It's practically a rite of passage.

One more wrinkle: vanity and alias addresses. Corner buildings with two valid street addresses, marketing addresses that differ from the legal one. Where county parcel data is available, the assessor's parcel number is a stronger key than any address string, and it's worth the enrichment work to get it.

Deduplicating Across Platforms

Scrape Zillow, Redfin, and Realtor.com and most properties show up two or three times, with fields that don't quite agree. Dedup has two halves: deciding which records are the same property, and deciding which values to keep.

For matching, in order of reliability: parcel number when a source exposes it (near-certain), normalized address plus unit (the workhorse, resolving the large majority of cases once your address layer is good), and finally geocode proximity plus attribute similarity for the residue. Same coordinates within ~25 meters, same beds and baths, square footage within ~5 percent: probably the same property with an unmatchable address string. When confidence is marginal, flag it for review instead of auto-merging. A wrong merge is worse than a duplicate.

For survivorship, when merged records disagree (Zillow says 1,850 sqft, Redfin says 1,830): set per-field source priority based on observed reliability rather than crowning one platform the blanket winner, and prefer the most recently observed value for volatile fields like price and status. Keep the losing values. Store the merged golden record and the source records behind it, so disagreements stay auditable and you can revise the rules later without re-scraping history. And don't silently resolve big disagreements; if two sources differ on square footage by more than 10 percent, one of them is wrong in a way that matters, and that record belongs in a review queue.

Price and Unit Chaos

Price fields are where format inconsistency gets expensive, because prices feed every metric.

Sale versus rent ambiguity comes first: $2,400 is a rent, $240,000 is a price, and a scraper that mixes listing types will happily store both in one column. Segregate by listing type at ingestion and range-check the results. A "sale price" under $10,000 or a "rent" over $50,000 a month deserves a flag, not a database row.

Then the formatting zoo: $549,000, 549000, 549K, From $549,000 on new construction, Contact for price. Parse to an integer, keep the raw string, and store unparseable values as NULL. Never as zero. Zeros poison medians silently, and a median poisoned by a few hundred zero-priced records looks plausible enough that nobody questions it for weeks.

Square footage needs commas stripped and ranges handled (1,800–2,200 sqft on new builds: store min and max rather than silently averaging), plus the placeholder values platforms use nulled out: 0, 1, and 99999 all show up in the wild. Lot size is the classic trap: some platforms report acres, some square feet, some either depending on lot size, so 0.25 and 10,890 are the same lot. Values under ~50 are acres, values over ~500 are square feet; convert to one unit and flag the ambiguous middle. Baths arrive as 2.5 or as 2 full, 1 half; pick one representation and map everything into it.

The general principle across all of it: parse defensively, keep raw values, make "unknown" explicit. Every silent coercion becomes an untraceable analytics bug three months later.

Stale Listings

A scraped dataset starts decaying the moment it's collected. Listings sell, get withdrawn, expire, relist, and platforms remove or update them on their own schedules. Without active freshness management your "active inventory" number inflates week over week, and everything built on it drifts along with it.

Track first-seen and last-seen timestamps per listing per source. A listing that stops appearing in your crawls is the primary delisting signal, but distinguish "confirmed gone" (the page returns a sold or removed state) from "not observed" (your crawl may have simply missed it), and only mark listings stale after two or three consecutive missed observations. Cleaner still: platforms usually flip a listing to pending or sold before removing it, and catching the status flip beats inferring from absence.

Watch for relist laundering too. The same property delisted and relisted a week later at a new price is one market story, not two, and linking those listing spells by property identity (that address work again) is what makes true days-on-market computable. Cumulative DOM across relists is usually the honest number, and it's rarely the number the platform shows.

Whatever your rules miss, age out hard. A listing sitting unchanged for 120+ days in a market with 30-day median DOM is suspect; recheck it rather than letting it pad inventory counts. Freshness is also a cadence question, since weekly scraping means every property's status is up to a week wrong. Cadence trade-offs are covered in our market dashboard guide.

Geocoding

Most real estate analysis is spatial — comps within a radius, submarket rollups, school-district cuts — so every record needs coordinates, and scraped coordinates deserve skepticism. Platforms sometimes serve ZIP-centroid or street-interpolated coordinates, which will quietly place a property half a mile from its parcel and corrupt every radius query that touches it.

Geocode from the normalized address, record the precision level (rooftop / parcel / interpolated / centroid) alongside the coordinates, and require rooftop-or-parcel precision before a record participates in radius-based comp logic. Cross-check scraped coordinates against your own geocode; disagreement beyond ~100 meters means one of them is wrong. And cache geocodes by normalized address. Re-geocoding the same address nightly is a pure waste of API budget, and we've seen bills that prove people do it.

The QA Checklist

Run these on every pipeline cycle, and alert on threshold breaches rather than eyeballing charts:

• Volume sanity — records within ±20% of the trailing average per source and geography; a drop means a broken scraper, a spike means a parsing bug or site change
• Field completeness — % non-null for price, beds, baths, sqft, address per source, tracked over time; completeness cliffs signal selector rot
• Range checks — no $0 prices, no 0-sqft homes, no 40-bath houses, no coordinates outside the target bounding box
• Dedup rate stability — cross-platform merge rate in its normal band; a falling match rate usually means address normalization regressed
• Duplicate residue — pull 20 random merged records, manually verify the sources refer to the same property
• Staleness distribution — share of "active" listings unobserved for >7 days stays below threshold
• Format drift — count of unparseable price/sqft/lot values per run; a jump means a platform changed a display format
• Cross-source agreement — median disagreement on price and sqft between platforms for matched properties; widening means a survivorship or parsing problem
• Referential integrity — every listing event points to a valid property record, no orphans after merges

Teams that automate this catch pipeline breaks in hours. Teams that don't find out when a client asks why inventory doubled in a week.

How We Handle It

Data QA is most of what clients are actually buying from a managed service; collection without cleaning just relocates the 80% onto your team. Our real estate deliveries include the full layer described above: CASS-style address standardization with raw values preserved and parcel-number keys where county data allows, cross-platform dedup with per-field survivorship and auditable source records, defensive parsing with explicit nulls instead of silent zeros, freshness management with relist linkage and true DOM, and the QA checklist running as an automated gate before anything ships. A broken source gets caught on our side, not in your dashboard.

Delivery is CSV, JSON, API, or direct database writes, on your cadence. For the pipeline mechanics upstream of all this, start with our real estate scraping guide; the same-property matching problem also shows up at tracker scale in our house price tracking article.

Clean Data Is the Product

Anyone can collect property listings. The datasets that support real decisions are the ones where somebody did the unglamorous work: normalized every address, merged every duplicate, parsed every price defensively, aged out every stale record.

If you'd rather spend your team's time on the analysis than the janitorial layer, talk to our team. Tell us what property data you need and we'll send a cleaned, deduplicated sample within days, QA checklist included.