Why Build Your Own

Every serious real estate operation eventually hits the same wall: the market reports you can buy are too slow, too aggregated, or cover the wrong geography. Zillow Research publishes excellent metro-level data, monthly, weeks after the fact. MLS stats go to members, on the MLS's schedule, with the MLS's definitions. By the time a cooling market shows up in a published median, the agents on the ground have known for two months.

A dashboard built on scraped listing data fixes the lag. It shows your markets, at your granularity (zip, neighborhood, school district), refreshed daily if you want. Brokerages use these to arm agents with hyperlocal stats for listing presentations. Funds use them to time acquisitions. Proptech companies productize them. The architecture is the same in every case, and it's simpler than most teams expect: scrape, normalize, store, visualize.

This is the full build, in order.

Step 1: Pick the Metrics Before You Scrape Anything

The most common dashboard failure isn't technical. It's collecting data first and deciding what it means later. Start from the questions your users actually ask, work backwards to metrics, then to raw fields.

The core set for a residential market dashboard:

• Median list price, by zip and property type, weekly. Median, not mean; one $8M listing shouldn't move your neighborhood trend line.
• Median price per square foot, which controls for the size mix of what happens to be listed.
• Active inventory count. Paired with sales velocity this gives months-of-supply, the classic balance metric.
• New listings per week. A jump in supply with flat demand precedes price softening.
• Days on market, median, computed from first-seen dates in your own scrape history.
• Price cut share: the percentage of active listings that have taken at least one reduction, and the median cut size.
• Sale-to-list ratio, where sold prices are visible.
• Median asking rent over median asking price, for gross yield. Investors live on this one, and it means scraping rentals as well as sales.

If you make me rank them: price cut share is the single most sensitive direction indicator we've seen, and rising DOM is the earliest. Both move well before medians do. And notice that the highest-value metrics are all derived from change over time, not from any snapshot. That one observation drives the entire architecture. You're building a system that looks at the same listings repeatedly and records what changed.

Step 2: Sources

For US residential, the practical stack is the core trio — Zillow, Redfin, Realtor.com — for sale listings, price history, and AVM estimates, with Redfin exposing some of the cleanest sold-price data of the three. For the rent side of yield calculations, Zillow Rentals and Apartments.com plus regional players; our guide to scraping rental listings has the platform-by-platform detail. County assessor and recorder data sits underneath as slow-moving ground truth for validating the scraped layer.

One platform is enough for a prototype. For production, scrape at least two and cross-reference. Every platform has coverage holes, and matching the same property across Zillow and Redfin is what makes your inventory counts trustworthy. The broader source landscape is in our real estate scraping guide.

Step 3: The Pipeline

Four stages, each independently replaceable: scrape, normalize, Postgres, BI tool.

Scraping layer

Daily runs against your target geographies, collecting active listings with price, status, address, beds/baths/sqft, and listing history where exposed. All of the operational pain in the whole system lives here. The major platforms run PerimeterX-class fingerprinting, rate limiting, and CAPTCHA challenges, and scrapers break whenever a site ships a redesign. It's also the one layer that's easy to outsource, because everything downstream is standard data engineering your team fully controls.

Normalization layer

Before anything hits the database: standardize addresses, unify price formats, map each platform's status vocabulary onto one of yours (active / pending / sold / off-market), and merge cross-platform records into a single property identity. Skimp here and every chart downstream is quietly wrong. Inventory double-counted, DOM computed off duplicate records, trend lines contaminated by format noise. This layer deserves its own article and has one: cleaning and normalizing property data.

Storage: Postgres, append-only

Postgres. Free, boring, and every BI tool speaks to it. The one design decision that matters is storing observations over time rather than current state:

CREATE TABLE properties (
  property_id   BIGSERIAL PRIMARY KEY,
  address_norm  TEXT NOT NULL,        -- normalized address
  zip           TEXT NOT NULL,
  property_type TEXT,                 -- sfh / condo / townhouse / multi
  beds          SMALLINT,
  baths         NUMERIC(3,1),
  sqft          INTEGER,
  lat           DOUBLE PRECISION,
  lng           DOUBLE PRECISION,
  UNIQUE (address_norm)
);

CREATE TABLE listing_events (
  event_id     BIGSERIAL PRIMARY KEY,
  property_id  BIGINT REFERENCES properties(property_id),
  source       TEXT NOT NULL,         -- zillow / redfin / realtor
  observed_at  TIMESTAMPTZ NOT NULL,
  status       TEXT NOT NULL,         -- active / pending / sold / delisted
  price        INTEGER,
  event_type   TEXT NOT NULL          -- new_listing / price_change / status_change / heartbeat
);

CREATE INDEX ON listing_events (property_id, observed_at);
CREATE INDEX ON listing_events (observed_at, event_type);

Two tables carry the whole system: one row per physical property, and an append-only event stream. Every metric on the dashboard is a query over listing_events. DOM is the gap between a property's new_listing and its terminal event; price-cut share is a count of negative price_change events over active inventory. Materialize the weekly zip-level aggregates into a summary table so dashboards aren't scanning raw events on every page load. That's it. Resist the urge to add more tables until a real query forces you to.

Visualization layer

Tool	Cost	Best For	Watch Out For
Metabase	Free self-hosted	Fast setup, SQL-optional users, embedding	Limited chart customization at the edges
Looker Studio	Free	Sharing with non-technical stakeholders, zero hosting	Needs a connector to Postgres; slower on big tables
Grafana	Free self-hosted	Time-series-heavy views, alerting on thresholds	UI reads "ops tool" to business users

For a brokerage or fund, just use Metabase. Point it at Postgres, build zip-level trend lines and a filterable listing table in an afternoon, share links internally. Grafana earns its place when you want alerting, like pinging the acquisitions channel when price-cut share in a target zip crosses 20%. Looker Studio wins when the audience is external stakeholders who will never install anything.

Step 4: Refresh Cadence

Match cost to the speed of the underlying signal. The workhorse is a daily active-listing scrape; daily observation gets DOM accurate to about a day and catches nearly all price changes. Sold and pending sweeps can run every two or three days since status transitions are less frequent. Rebuild the summary tables nightly after the scrape lands, so dashboards always show yesterday-complete data. Add a weekly full re-crawl of target geographies to catch listings the incremental crawl missed and to confirm delistings. And if you have an acquisition workflow that acts on same-day price cuts, run an intraday scrape on a shortlist of watched zips, 2 to 4 times daily.

Honestly, more than daily is a waste for most teams. Doubling scrape frequency roughly doubles collection cost and only pays off in hours-not-days use cases. Most brokerages and funds land on daily; proptech products with alerting add the intraday hot list.

Pitfalls We Keep Seeing

The same failure patterns show up in almost every homegrown dashboard we've been asked to look at.

Mean instead of median, everywhere. One luxury listing entering a small zip drags the mean up double digits and manufactures a trend. Medians for prices, always, with counts displayed alongside so users can see when a segment is too thin to trust. Related: no minimum sample thresholds. A zip with six active listings produces violently noisy weeklies; suppress anything computed from fewer than about 20 observations, or roll thin zips up to city level.

Mixing property types in one trend line. Condos and single-family homes have different price levels and dynamics, so a shift in listing mix masquerades as a market move. Segment first, and lean on price per square foot for cross-type comparison.

Trusting the platform's displayed DOM. Platforms show it inconsistently and reset it on relists. Compute DOM from your own first-seen timestamps and link relisted properties, or your DOM trend will quietly flatter the market.

And the one that actually kills projects: silent scraper decay. The most dangerous dashboard is one whose feed broke ten days ago and still renders beautifully. We learned to put data-freshness and record-volume indicators on the dashboard itself, where every viewer can see them, after watching exactly this failure play out. Nobody checks the pipeline logs. Everybody notices a red "data 10 days old" banner.

Who Runs These

Brokerages, mostly, arming agents for pricing conversations. "In this zip, a quarter of active listings have cut price and median DOM is up nine days since March" wins an argument with an overconfident seller better than any anecdote. Funds and family offices use the dashboard as a screening layer: when a zip's leading indicators soften, acquisition criteria loosen there first. iBuyers and high-volume flippers feed the same data into pricing models, where staleness directly misprices offers. For proptech companies the dashboard is the product, and the differentiator is rarely the charts; it's the freshness and cleanliness of the pipeline underneath. Lenders and appraisers round out the list, watching collateral markets between appraisal cycles.

Where Builds Fail, and Where We Fit

Teams that build these dashboards almost never fail at the database or the charts. That layer is solved and well documented. They fail at the top of the funnel: scrapers break every few weeks as platforms tighten anti-bot systems, maintenance quietly becomes a half-time engineering job, duplicates corrupt the inventory counts, and the project gets abandoned the first time the data goes visibly stale.

The pragmatic split is to own the schema, the metrics, and the dashboards, and outsource the scraping and normalization. That seam is exactly where ScrapeAny plugs in. We deliver cleaned, deduplicated, change-tracked listing data daily for your geographies, as CSV, JSON, API, or direct writes into your Postgres. Your team builds on a feed that doesn't break when Zillow redesigns, and the anti-bot arms race is our problem.

Ship the Dashboard

A market dashboard is one of the highest-leverage data projects in real estate: two tables, a free BI tool, and a reliable feed produce intelligence your competitors are buying months late in PDF form. The schema above is genuinely enough to start with.

Want the data layer handled? Talk to our team. Tell us your metros and metrics and we'll have a sample feed flowing into your database within days.