Estimate house prices

What you will be able to do

Take a second real regression problem from the business question to a tested model and a recommendation. You estimate the sale price of a house for a lender. The prices are skewed, every error is a percentage, and the market moves under the model.

Case studyPortfolio piece

Dataset

House sales in King County, USA, 2014–2015 (OpenML 42092)

Licence

CC0 1.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Log target
• Location features
• Outliers
• Ridge and boosting
• Error by price band

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook has every code cell of this page, three tasks with checks, and the instructions. Its setup cells install the packages and download the data, so it needs no local setup. It runs in about two minutes. Read the story here, and run the code there.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. The lessons it leans on most:

• MAPE and the mean error, and why a percentage suits prices, from Regression metrics.
• Cross-validation, and a split by time, from Train, validation and test sets.
• The one-standard-error rule, from Tune hyperparameters.
• Pipelines with one-hot encoding, from Preprocess with a pipeline, and gradient boosting, from Ensembles.
• Slices with counts, from Slice analysis, and the model card, from Document use boundaries.
• The first case study, Forecast bike-share demand, is useful but not required.

What you need on your computer (only for the local route; the notebook needs none of it)

• A Python environment with pandas 3.0.6, scikit-learn 1.9.1 and matplotlib, such as the .venv of your project from Define a baseline: Set up your project. Download the notebook from the box above into any folder, and open it as in Define a baseline: Open a notebook in VS Code. The notebook downloads the data itself into a new folder house-prices/. To start again, delete that folder.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The notebook was tested in Jupyter. Windows is not tested.

The situation

Nadia Haddad leads the valuation team of a regional mortgage lender in the Seattle area. A mortgage is a loan to buy a house, and the house is the guarantee. If the borrower cannot pay, the lender sells the house to get the money back. So before the lender approves a loan, it must know what the house is worth. A licensed appraiser visits the house and writes a valuation. That takes several days and costs a fee, and the lender’s policy requires it for every loan.

Nadia wants a first estimate on the day an application arrives, from the description of the house alone. It does not replace the appraiser. It tells the team early when the agreed price looks far from the value. Then the appraiser looks at those cases first, and the loan officer can warn the buyer.

Errors in both directions cost something, but not the same. If the estimate is too high, the house looks like a better guarantee than it is. Suppose the lender lends 80% of the estimate, and the estimate is 10% too high. Then the loan is really 88% of the value, and a sale after a default may not cover it. If the estimate is too low, a good application looks risky, and the team wastes time and annoys a customer. Nadia’s target: most estimates within ±10% of the sale price, and fewer than 1 in 5 more than 10% too high.

Samir Patel, the team’s analyst, builds the model. Chris, from the loans team, has quick ideas.

What you will build

You will build a model that estimates the sale price of a house from its size, quality and place. You will also make a rule that says when to use the estimate. You will compare it with a valuer’s quick method, check it with cross-validation and a later validation period, and write down where it fails. At the end you have a notebook, an experiment log with eight experiments, a saved model, a model card and a recommendation for Nadia.

The case study uses skills from the whole course. You check the data (Module 2), choose the metrics and the split (Module 3), and build pipelines (Module 4). You use boosting and an experiment log (Module 5), and the one-standard-error rule (Module 6). Then you check slices (Module 7) and write a model card (Module 8). It adds one new tool, which it explains when it comes: a model that learns the logarithm of the price.

The data

The data is a public record of house sales in King County, Washington, the county of Seattle, from May 2014 to May 2015. It was first published on Kaggle by a user called harlfoxem, and is now dataset 42092 on OpenML. Both pages give the licence CC0 1.0: the data is in the public domain, and anyone may reuse it without conditions. We still give the credit: House Sales in King County, USA (harlfoxem, Kaggle 2016; OpenML 42092). Nadia, Samir and the lender are invented. The sales are real.

Nadia’s team does not know the data yet, and the uploader gave no description of the columns. Their meanings come from the county’s property records, and the dataset card in the labs repository records them. Nextia hosts the file as a compressed CSV, with the date written as 2014-10-13. Nothing else is changed.

Load it and look at three rows

Every code block on this page is a cell of the notebook. The setup cells have already loaded pandas as pd and numpy as np, and downloaded the file into the working folder.

houses = pd.read_csv("king_county_house_sales.csv.gz", parse_dates=["date"])
print(houses.shape)
print("sales from", houses["date"].min().date(), "to", houses["date"].max().date())
print(houses[["date", "price", "bedrooms", "bathrooms", "sqft_living", "grade", "zipcode"]].head(3).to_string())

The output:

Output

(21613, 21)
sales from 2014-05-02 to 2015-05-27
        date     price  bedrooms  bathrooms  sqft_living  grade  zipcode
0 2014-10-13  221900.0         3       1.00         1180      7    98178
1 2014-12-09  538000.0         3       2.25         2570      7    98125
2 2015-02-25  180000.0         2       1.00          770      6    98028

What this tells us. One row is one sale of one house: 21,613 sales in about 13 months. price is the target, in US dollars. The first house, sold in October 2014, has 3 bedrooms, 1 bathroom and 1,180 square feet of living area, and sold for 221,900 dollars. (A square foot is about 0.093 m², so 1,180 sq ft is about 110 m².)

The columns we use

These are the columns that describe a house, in plain words:

Column	Meaning
bedrooms, bathrooms, floors	counts; 0.5 bathroom is a toilet without a shower; 1.5 floors is a partial upper floor
sqft_living, sqft_above, sqft_basement	living area, the part above ground, and the basement (0 = no basement)
sqft_lot	the area of the land
waterfront, view	1 if the house is on the water; the quality of the view from 0 (no rated view) to 4
condition	the state of the house, from 1 (poor) to 5 (very good)
grade	the county’s building grade, from 1 to 13: the quality of construction and design (7 is average)
yr_built, yr_renovated	year built; year of the last renovation (0 = never)
zipcode, lat, long	postal code (70 codes), latitude and longitude
sqft_living15, sqft_lot15	the living and land area of the 15 nearest houses
id, date	the house and the sale date: used only to split the data, never as inputs

First checks

print("missing values:", int(houses.isna().sum().sum()), "| duplicate rows:", int(houses.duplicated().sum()))
print("houses sold more than once:", int(houses["id"].duplicated().sum()), "extra sales")
print("bedrooms:", houses["bedrooms"].value_counts().sort_index().to_dict())
print("bathrooms 0:", int((houses["bathrooms"] == 0).sum()),
      "| yr_renovated 0:", int((houses["yr_renovated"] == 0).sum()),
      "| sqft_basement 0:", int((houses["sqft_basement"] == 0).sum()))

The output:

Output

missing values: 0 | duplicate rows: 0
houses sold more than once: 177 extra sales
bedrooms: {0: 13, 1: 199, 2: 2760, 3: 9824, 4: 6882, 5: 1601, 6: 272, 7: 38, 8: 13, 9: 6, 10: 3, 11: 1, 33: 1}
bathrooms 0: 10 | yr_renovated 0: 20699 | sqft_basement 0: 13126

What this tells us. No cell is empty and no row is a copy of another. But some numbers are codes: yr_renovated is 0 for the 20,699 houses that were never renovated, and sqft_basement is 0 for houses without a basement. A model must not read “renovated in year 0”. Trees handle this well, because they can split between 0 and 1900; we keep it in mind for the linear model. Thirteen houses have 0 bedrooms (a studio, or a missing value) and one has 33. And 177 sales are second or third sales of a house that is already in the data.

Look at the house with 33 bedrooms next to the other large houses:

print(houses.loc[houses["bedrooms"] >= 10, ["bedrooms", "bathrooms", "sqft_living", "price"]].to_string())

The output:

Output

       bedrooms  bathrooms  sqft_living      price
8757         11       3.00         3000   520000.0
13314        10       5.25         4590  1148000.0
15161        10       2.00         3610   650000.0
15870        33       1.75         1620   640000.0
19254        10       3.00         2920   660000.0

What this tells us. The other houses with 10 or 11 bedrooms have 2,900 to 4,600 sq ft. A house with 33 bedrooms in 1,620 sq ft would have rooms of about 50 sq ft. This is a typing error for 3. Samir fixes it in his copy of the data and writes the fix in the dataset card, so that anyone can repeat it:

houses.loc[houses["bedrooms"] == 33, "bedrooms"] = 3  # 1,620 sq ft: a typing error for 3
print("largest number of bedrooms now:", int(houses["bedrooms"].max()))

The output:

Output

largest number of bedrooms now: 11

What this tells us. The fix worked: no house has more than 11 bedrooms now, which fits the other large houses. The fix is in the notebook and in the dataset card, so the hosted file stays as the source published it.

Repeat sales

Next, the houses that were sold more than once. How much did their price change?

repeat = houses[houses["id"].duplicated(keep=False)].sort_values(["id", "date"])
first, last = repeat.groupby("id")["price"].first(), repeat.groupby("id")["price"].last()
print("houses:", repeat["id"].nunique(), "| median change from first to last sale:",
      f"{100 * ((last / first).median() - 1):.1f}%")

The output:

Output

houses: 176 | median change from first to last sale: 54.4%

What this tells us. 176 houses were sold twice or more within a year, and the median price went up by 54% between the first and the last sale. Most of them were bought, renovated and sold again. But the description in the data is the same for both sales, so the model cannot see the work. These sales also tell us something about the split: two sales of the same house are not two independent examples. If one sale is in train and the other is in valid, the check is not about a new house any more. So all sales of a house must stay in the same part of the split.

Frame the problem

In the terms of Translate a need:

• Decision: which applications the appraiser looks at first, and when the loan officer warns a buyer that the price looks high.
• Prediction unit: one sale of one house.
• Target: price, the sale price. A number, so this is regression.
• Prediction time: the day of the application, before the appraisal. The description of the house is known, from the county’s records and the listing.
• Not available: the sale price itself, and anything that the appraiser finds in the house (the state of the kitchen, damp, a new roof).
• Leaks to watch: the 15-neighbour areas were probably measured in 2015. For a sale in 2014, they may describe the street a few months later. The effect is small, and Samir keeps them, with a note.
• People and fairness: no column describes the buyer or the seller. But the place is a strong input, and in the United States the place is linked to income and race. A model that under-values some neighbourhoods would repeat a known harm in lending. So Samir checks the errors by area, and the limits come back to it.

Explore what the decision needs

We do not explore every column. Nadia’s decision raises three questions. What do the prices look like, and how should we measure a miss? Does the market move during the year? And how much does the place matter?

The price

print(houses["price"].describe().round(0).to_dict())
print("share above 1 million:", round(float((houses["price"] > 1_000_000).mean()), 3))

The output:

Output

{'count': 21613.0, 'mean': 540088.0, 'std': 367127.0, 'min': 75000.0, '25%': 321950.0, '50%': 450000.0, '75%': 645000.0, 'max': 7700000.0}
share above 1 million: 0.068

The price goes from 75,000 to 7,700,000 dollars. The median is 450,000, but the mean is 540,088: a few very expensive houses pull the mean up. Look at the shape of the prices.

Sale prices of a fixed random sample of 2,000 sales, in bins of 100,000 dollars. Most sales are between 200,000 and 700,000 dollars; a long tail reaches to the right. The 4 sales of the sample above 3 million are not shown.

Text description of the chart

A histogram of the sale prices of 2,000 sales chosen at random, from 0 to 3 million dollars, in bins of 100,000 dollars. Only 1 sale is below 100,000. The bars rise quickly to a peak of 427 sales between 300,000 and 400,000. Then they fall slowly: 354 between 400,000 and 500,000, and 254 between 500,000 and 600,000. There are 112 between 700,000 and 800,000, and 29 between 1 million and 1.1 million. A long, thin tail continues to the right. The 4 sales above 3 million are left out of the chart.

What this tells us. The price is skewed: many ordinary houses and a long tail of expensive ones. That changes how we should measure and how we should learn. A miss of 50,000 dollars is 20% of a 250,000-dollar house, and only 2.5% of a 2-million-dollar house. Nadia’s target is already a percentage, “within ±10%”, and that is the right way to see these errors.

Prices think in percent: the logarithm

When you talk about prices, you already think in percent. A shop says “20% off”, not “100 dollars off”. 100 dollars off a pair of socks and 100 dollars off a car are not the same thing. House prices behave the same way. A good view adds a share of the price, not a fixed amount: it adds more to a large house than to a small one.

The logarithm turns percentages into equal steps. On a log scale, every step of “× 10” has the same size: 100,000 → 1,000,000 → 10,000,000 are three equal steps. So “10% more” is the same distance for a cheap house and for an expensive one. The next chart shows the same prices as the log, base 10, of the price.

The same 2,000 prices on a log scale. The shape is almost symmetric: a model that learns the log of the price treats a 10% error the same for every house.

Text description of the chart

A histogram of the same 2,000 prices on a log scale, from 4.9 (about 79,000 dollars) to 6.7 (about 5 million dollars), in bins of 0.05. The bars form an almost symmetric hill. Its top is the bin from 5.6 to 5.65, about 400,000 to 450,000 dollars, with 208 sales. The long right tail of the first chart has become a short tail on both sides.

So the models in this case study learn the log of the price, and turn the answer back into dollars at the end. A model that learns the log treats a miss of 10% as the same size for every house, which is exactly Nadia’s view.

The market moves

months = houses.groupby(houses["date"].dt.to_period("M"))["price"].agg(sales="size", median_price="median")
print(months.T.to_string())

The output:

Output

date           2014-05   2014-06   2014-07   2014-08   2014-09   2014-10   2014-11   2014-12   2015-01   2015-02   2015-03   2015-04   2015-05
sales           1768.0    2180.0    2211.0    1940.0    1774.0    1878.0    1411.0    1471.0     978.0    1250.0    1875.0    2231.0     646.0
median_price  465000.0  465000.0  465000.0  442100.0  450000.0  446900.0  435000.0  432500.0  438500.0  425545.0  450000.0  476500.0  455000.0

• Train
• Valid
• Test

Median sale price per month, May 2014 to May 2015, coloured by the part of the split. The axis starts at 400,000 dollars, so the changes look larger than they are: from the lowest month to the highest, prices move by about 12%.

Text description of the chart

A line chart of 13 months with the median price from 400,000 to 500,000 dollars. From May to July 2014 the median is 465,000. It falls slowly through the autumn and winter to 432,500 in December 2014 (train), 438,500 in January 2015 and 425,545 in February 2015 (valid). In spring it rises: 450,000 in March, 476,500 in April, the highest month, and 455,000 in May 2015 (test). The number of sales also falls in winter, to 978 in January 2015, and rises again in spring, to 2,231 in April.

What this tells us. House prices follow the seasons: fewer sales and lower prices in winter, more and higher in spring. A model that learns from the year up to February will tend to be too low in the spring. We will see this again in the final test.

Three areas

The place matters most in house prices. To check the errors by place later, Samir puts each sale in one of three areas. The first is the zip codes that start with 981: Seattle and its close neighbours. The other two are the rest of the county, north and south of latitude 47.5.

def area(part):
    """Three areas: zip codes 981xx (Seattle and close neighbours), then north/east and south."""
    seattle = part["zipcode"] // 100 == 981
    return np.select([seattle, part["lat"] >= 47.5], ["Seattle", "North and east"], "South")

houses["area"] = area(houses)
per_sqft = (houses["price"] / houses["sqft_living"]).groupby(houses["area"]).median().round(0)
print(houses["area"].value_counts().to_dict())
print("median price per sq ft:", per_sqft.to_dict())

The output:

Output

{'Seattle': 8977, 'North and east': 7209, 'South': 5427}
median price per sq ft: {'North and east': 264.0, 'Seattle': 291.0, 'South': 165.0}

What this tells us. A square foot in Seattle sells for a median of 291 dollars, and in the south of the county for 165. The same house is worth much more in one place than in another. The areas are only for checking errors; the models use the latitude, longitude or zip code instead.

Split and baseline

A split by time, with each house in one part

Nadia’s model will estimate next month’s sales from the sales of the past. So the split follows time, as in the first case study. Train is the sales of 2014, valid is January and February 2015, and test is March to May 2015. But with one change: all sales of a house go to the part of its first sale, so that no house is in two parts. This is a group split: the group is the house.

TARGET = "price"
FEATURES = ["bedrooms", "bathrooms", "sqft_living", "sqft_lot", "floors", "waterfront", "view", "condition",
            "grade", "sqft_above", "sqft_basement", "yr_built", "yr_renovated", "lat", "long",
            "sqft_living15", "sqft_lot15", "zipcode"]
first_sale = houses.groupby("id")["date"].transform("min")  # every sale of a house goes where its first sale goes
train = houses[first_sale < "2015-01-01"]
valid = houses[(first_sale >= "2015-01-01") & (first_sale < "2015-03-01")]
test = houses[first_sale >= "2015-03-01"]
print("train", len(train), "| valid", len(valid), "| test", len(test))
print("sales moved to an earlier part:", int((houses["date"] != first_sale).sum()))

The output:

Output

train 14768 | valid 2191 | test 4654
sales moved to an earlier part: 177

What this tells us. Train has 14,768 sales, valid 2,191 and test 4,654. The 177 repeat sales moved to the part of the house’s first sale. That breaks the time order a little: a few sales from 2015 are now in train. They are less than 1% of train, and the other choice, a house in two parts, is worse.

How we measure errors

The errors are percentages of the real price. The next function reports four numbers that Nadia understands:

• the median error in percent: positive means the estimate is too low. It shows the direction, like the mean error in Regression metrics, and a median is not pulled by a few wild sales;
• the MAPE, the mean absolute percentage error. Prices are never close to zero, so MAPE works well here;
• the share within 10%: Nadia’s target;
• the share over by 10% or more: the dangerous direction for a lender.

def report(real, estimate):
    """Errors as percentages of the real price (positive = the estimate is too low)."""
    pct = 100 * (real - estimate) / real
    return {"median error %": round(float(np.median(pct)), 1),
            "MAPE %": round(float(np.mean(np.abs(pct))), 1),
            "within 10%": round(float(np.mean(np.abs(pct) <= 10)), 3),
            "over by 10%+": round(float(np.mean(pct < -10)), 3)}

Three baselines

A valuer who has no model makes a quick estimate on paper. Samir copies three such methods, each learned from train only. The first is the median price: the same estimate for every house. The second is the median price per square foot of the whole county, times the house’s living area. The third is the median price per square foot in the same zip code, times the living area: size and place together.

Predict. Which baseline gets the most estimates within 10%? About what share?

median_estimate = np.full(len(valid), train[TARGET].median())
per_sqft = (train[TARGET] / train["sqft_living"]).median()
zip_per_sqft = (train[TARGET] / train["sqft_living"]).groupby(train["zipcode"]).median()
baselines = {"median price": median_estimate,
             "price per sq ft": per_sqft * valid["sqft_living"],
             "zip price per sq ft": valid["zipcode"].map(zip_per_sqft) * valid["sqft_living"]}
baseline_results = {name: report(valid[TARGET], estimate) for name, estimate in baselines.items()}
print(pd.DataFrame(baseline_results).T.to_string())

The output:

Output

                     median error %  MAPE %  within 10%  over by 10%+
median price                   -4.7    45.3       0.154         0.461
price per sq ft                -2.8    35.4       0.188         0.446
zip price per sq ft             2.9    19.2       0.372         0.285

What this tells us. The median price is within 10% for only 15% of houses. Size alone does a little better. Size and place together, the zip-code baseline, get 37% of houses within 10%, with a MAPE of 19.2%. That is the method a valuer can use on the back of an envelope, and every model must beat it. It is also too high by 10% or more for 28.5% of houses: too risky for Nadia.

The experiment log

The log has this course’s columns, with the metrics of this problem: id, date, change, cv_mape and cv_std (the mean and spread of 5 folds on train), valid_mape, valid_within_10, decision and reason. As in Machine learning is iterative, every attempt goes in.

import datetime
from pathlib import Path

LOG_FILE = Path("experiment_log.csv")
LOG_COLUMNS = ["id", "date", "change", "cv_mape", "cv_std", "valid_mape", "valid_within_10", "decision", "reason"]

def log_experiment(id, change, cv, result, decision, reason):
    """Add one row (cv = (mean, std) or None), or replace the row with the same id."""
    log = pd.read_csv(LOG_FILE) if LOG_FILE.exists() else pd.DataFrame(columns=LOG_COLUMNS)
    row = {"id": id, "date": datetime.date.today().isoformat(), "change": change,
           "cv_mape": cv[0] if cv else None, "cv_std": cv[1] if cv else None,
           "valid_mape": result["MAPE %"], "valid_within_10": result["within 10%"],
           "decision": decision, "reason": reason}
    log = pd.concat([log[log["id"] != id], pd.DataFrame([row])], ignore_index=True)
    log.sort_values("id").to_csv(LOG_FILE, index=False)

def show_log():
    log = pd.read_csv(LOG_FILE)
    print(log[["id", "change", "cv_mape", "cv_std", "valid_mape", "valid_within_10", "decision"]].to_string(index=False))

LOG_FILE.unlink(missing_ok=True)  # start a new log

The baselines have nothing to tune, so they have no CV score:

log_experiment("E01", "Baseline: the median price of train", None, baseline_results["median price"],
               "baseline", "The floor.")
log_experiment("E02", "Baseline: median price per sq ft x living area", None, baseline_results["price per sq ft"],
               "baseline", "Size alone.")
log_experiment("E03", "Baseline: median price per sq ft in the zip code x living area", None,
               baseline_results["zip price per sq ft"], "baseline", "Size and place: what a quick desk estimate does.")
show_log()

The output:

Output

 id                                                         change  cv_mape  cv_std  valid_mape  valid_within_10 decision
E01                            Baseline: the median price of train      NaN     NaN        45.3            0.154 baseline
E02                 Baseline: median price per sq ft x living area      NaN     NaN        35.4            0.188 baseline
E03 Baseline: median price per sq ft in the zip code x living area      NaN     NaN        19.2            0.372 baseline

What this tells us. The log holds the three baselines. Every model from here on must beat E03: a valid MAPE below 19.2% and more than 37.2% of houses within 10%.

Models

House sales are not as tied to time as hourly rentals. So Samir compares models as you did in Tune hyperparameters: 5-fold cross-validation on train, then one check on valid. GroupKFold keeps all sales of one house in the same fold, for the same reason as the split.

from sklearn.model_selection import GroupKFold, cross_val_score

def cv_mape(model):
    """5-fold CV on train, the sales of one house in one fold; mean and std of the MAPE, in %."""
    scores = -cross_val_score(model, train, train[TARGET], groups=train["id"], cv=GroupKFold(n_splits=5),
                              scoring="neg_mean_absolute_percentage_error")
    return round(float(100 * scores.mean()), 2), round(float(100 * scores.std()), 2)

Ridge regression, in dollars and in logs

The first model is ridge regression. The zip code becomes 70 one-hot columns (it is a name for a place, not a quantity: 98178 is not “more” than 98125). Latitude and longitude are left out, because a straight line in latitude cannot follow the shape of a city. The other numbers are scaled.

from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

NUMBERS = [c for c in FEATURES if c not in ("zipcode", "lat", "long")]
prepare = ColumnTransformer([("zipcode", OneHotEncoder(handle_unknown="ignore"), ["zipcode"]),
                             ("numbers", StandardScaler(), NUMBERS)])
ridge = Pipeline([("prepare", prepare), ("model", Ridge(alpha=1.0))])
ridge_cv = cv_mape(ridge)
ridge_result = report(valid[TARGET], ridge.fit(train, train[TARGET]).predict(valid))
print("CV", ridge_cv, "| valid", ridge_result)

The output:

Output

CV (19.53, 0.37) | valid {'median error %': 2.4, 'MAPE %': 21.3, 'within 10%': 0.337, 'over by 10%+': 0.298}

What this tells us. The ridge model is worse than the zip-code baseline on valid: MAPE 21.3% against 19.2%. Its CV MAPE of 19.5% tells the same story. On the dollar scale, a model with 70 zip codes and every size column cannot beat a valuer’s quick method. Now give the same pipeline the log of the price. scikit-learn’s TransformedTargetRegressor does it for you: it takes the log of the target before fit(), and turns each estimate back into dollars after predict().

Predict. Will the MAPE fall a little or a lot?

ridge_log = TransformedTargetRegressor(regressor=ridge, func=np.log, inverse_func=np.exp)
ridge_log_cv = cv_mape(ridge_log)
ridge_log_result = report(valid[TARGET], ridge_log.fit(train, train[TARGET]).predict(valid))
print("CV", ridge_log_cv, "| valid", ridge_log_result)

The output:

Output

CV (13.55, 0.05) | valid {'median error %': 2.3, 'MAPE %': 14.9, 'within 10%': 0.486, 'over by 10%+': 0.234}

What this tells us. The same model, with the same inputs, drops from 21.3% to 14.9% on valid, and from 19.5% to 13.6% in CV. Nothing changed but the scale of the target. On the dollar scale, the model spends its effort on the expensive houses, where the dollar errors are largest. On the log scale, every house counts the same, in percent. The CV spread is tiny (0.05), so the five folds agree.

log_experiment("E04", "Ridge, zip code one-hot, scaled numbers", ridge_cv, ridge_result, "reject",
               "Worse than the zip baseline on valid: its errors grow with the price.")
log_experiment("E05", "E04 on the log of the price", ridge_log_cv, ridge_log_result, "keep",
               "MAPE down by about 6 points: errors scale with the price.")
show_log()

The output:

Output

 id                                                         change  cv_mape  cv_std  valid_mape  valid_within_10 decision
E01                            Baseline: the median price of train      NaN     NaN        45.3            0.154 baseline
E02                 Baseline: median price per sq ft x living area      NaN     NaN        35.4            0.188 baseline
E03 Baseline: median price per sq ft in the zip code x living area      NaN     NaN        19.2            0.372 baseline
E04                        Ridge, zip code one-hot, scaled numbers    19.53    0.37        21.3            0.337   reject
E05                                    E04 on the log of the price    13.55    0.05        14.9            0.486     keep

What this tells us. E05 is the first model that beats the baseline, so the log marks it keep. E04 stays in the log with its reason: it is the evidence for learning the log of the price.

Gradient boosting with the map

A linear model adds one effect per zip code. But within a zip code, a street on the water and a street next to a highway can differ a lot. Trees can cut the map into small pieces: “latitude above 47.6 and longitude below −122.2” is a neighbourhood. So the boosting model uses lat and long instead of the zip code, and it also learns the log of the price.

from sklearn.ensemble import HistGradientBoostingRegressor

TREE_FEATURES = [c for c in FEATURES if c != "zipcode"]  # trees use lat and long for the place

def boosting(**settings):
    select = ColumnTransformer([("features", "passthrough", TREE_FEATURES)])
    model = Pipeline([("select", select), ("model", HistGradientBoostingRegressor(random_state=0, **settings))])
    return TransformedTargetRegressor(regressor=model, func=np.log, inverse_func=np.exp)

boosting_cv = cv_mape(boosting())
boosting_result = report(valid[TARGET], boosting().fit(train, train[TARGET]).predict(valid))
print("CV", boosting_cv, "| valid", boosting_result)

The output:

Output

CV (11.71, 0.19) | valid {'median error %': 2.7, 'MAPE %': 13.2, 'within 10%': 0.554, 'over by 10%+': 0.194}

What this tells us. Boosting gets 55% of valid houses within 10%, against 49% for ridge on the log price and 37% for the zip-code baseline. Its CV MAPE, 11.71%, is 1.8 points better than ridge.

Tune one setting, with the one-standard-error rule

The obvious next step is a slower boosting: a learning rate of 0.05 and 600 trees. Is the gain real, or noise? Here a lower MAPE is better, so the one-standard-error rule works the other way round. Take the best mean and add its standard error: the std of the 5 folds divided by √5. Every setting at or below that limit ties with the best.

slower_cv = cv_mape(boosting(learning_rate=0.05, max_iter=600))
print("E06", boosting_cv, "| E07", slower_cv)
standard_error = slower_cv[1] / np.sqrt(5)  # the std of the 5 fold scores / the square root of 5
print("limit = best mean + one standard error =", round(slower_cv[0] + standard_error, 2))

The output:

Output

E06 (11.71, 0.19) | E07 (11.57, 0.24)
limit = best mean + one standard error = 11.68

What this tells us. E07 is the best, with 11.57%. Its limit is 11.68%. E06, at 11.71%, is just above the limit, so it does not tie with E07. The rule says: keep E07. The gain is small, and E07 takes about twice as long to train. But it is a real gain, and the model is still small and fast. Samir checks E07 once on valid, and logs both:

slower_result = report(valid[TARGET], boosting(learning_rate=0.05, max_iter=600).fit(train, train[TARGET]).predict(valid))
print("E07 on valid:", slower_result)
log_experiment("E06", "Gradient boosting on the log price, lat and long, defaults", boosting_cv, boosting_result,
               "reject", "1.8 points better than E05 in CV, but just outside the limit of E07.")
log_experiment("E07", "E06 with learning_rate 0.05 and 600 trees", slower_cv, slower_result, "keep",
               "Best CV; E06 is above the one-standard-error limit. Valid agrees.")
show_log()

The output:

Output

E07 on valid: {'median error %': 2.7, 'MAPE %': 12.9, 'within 10%': 0.556, 'over by 10%+': 0.194}
 id                                                         change  cv_mape  cv_std  valid_mape  valid_within_10 decision
E01                            Baseline: the median price of train      NaN     NaN        45.3            0.154 baseline
E02                 Baseline: median price per sq ft x living area      NaN     NaN        35.4            0.188 baseline
E03 Baseline: median price per sq ft in the zip code x living area      NaN     NaN        19.2            0.372 baseline
E04                        Ridge, zip code one-hot, scaled numbers    19.53    0.37        21.3            0.337   reject
E05                                    E04 on the log of the price    13.55    0.05        14.9            0.486     keep
E06     Gradient boosting on the log price, lat and long, defaults    11.71    0.19        13.2            0.554   reject
E07                      E06 with learning_rate 0.05 and 600 trees    11.57    0.24        12.9            0.556     keep

What this tells us. Valid agrees: 12.9% against 13.2%. E07 is the model.

Your turn: a model without the place

Chris asks: “Do we really need the map? The house itself should be enough.” In the notebook, you remove lat and long from the boosting model (the default settings of E06, for speed), log it as E08, and compare. Predict first: how many points will the MAPE go up?

Show the answer

TREE_FEATURES = [c for c in FEATURES if c not in ("zipcode", "lat", "long")]
no_place_cv = cv_mape(boosting())
no_place_result = report(valid[TARGET], boosting().fit(train, train[TARGET]).predict(valid))
print("CV", no_place_cv, "| valid", no_place_result)
log_experiment("E08", "E06 without lat and long", no_place_cv, no_place_result, "reject",
               "Much worse: the place is a large part of the price.")
TREE_FEATURES = [c for c in FEATURES if c != "zipcode"]  # back to the E06 features

The output:

Output

CV (22.13, 0.39) | valid {'median error %': -0.0, 'MAPE %': 23.9, 'within 10%': 0.274, 'over by 10%+': 0.37}

What this tells us. Without the place, the CV MAPE nearly doubles, from 11.71% to 22.13%, and only 27% of valid houses are within 10%. That is worse than the zip-code baseline, which knows the place but almost nothing about the house. In house prices, the place carries as much information as the house itself.

The chart summarises the log on valid.

Valid MAPE of each experiment. Grey: baselines. Blue: models. Orange: the chosen model, E07.

Text description of the chart

A column chart of eight experiments with the valid MAPE in percent, from 0 to 50. Baselines: E01, the median price, 45.3; E02, price per square foot, 35.4; E03, price per square foot in the zip code, 19.2. Models: E04, ridge in dollars, 21.3; E05, ridge on the log price, 14.9; E06, boosting on the log price, 13.2; E07, the slower boosting, 12.9, the lowest bar; E08, boosting without latitude and longitude, 23.9.

What this tells us. Two steps did most of the work: learning the log of the price (E04 to E05) and giving the trees the map (E05 to E06). The slower boosting added a little. Without the place, boosting falls behind even the zip-code baseline.

Common mistake

The random split that mixes the months

Symptom: Chris builds the same boosting model with a random split of all sales, 80% and 20%. His MAPE is 11.9%, better than Samir’s 13.2% for the same model. He says the time split “throws away good data”.

from sklearn.model_selection import train_test_split

shuffled_train, shuffled_test = train_test_split(houses, test_size=0.2, random_state=0)
model = boosting().fit(shuffled_train, shuffled_train[TARGET])
print("random split:", report(shuffled_test[TARGET], model.predict(shuffled_test)))
twins = shuffled_test["id"].isin(shuffled_train["id"])
print("test sales of a house that is also in train:", int(twins.sum()))

The output:

Output

random split: {'median error %': 0.5, 'MAPE %': 11.9, 'within 10%': 0.57, 'over by 10%+': 0.221}
test sales of a house that is also in train: 53

Cause: a random split mixes the months. The model learns from spring sales and is tested on other spring sales, so it knows the spring market. Nadia’s model never knows next month’s market. The random split also puts sales of the same house on both sides: 53 of them.

Fix: split as the model will be used. Here that means by time, with each house in one part. Accept the larger error: it is the one Nadia will see. If you need more data for training, refit on train and valid before the final test, as Samir does.

Consequence in AI work: a valuation model tested on a random split looks better than it is. It also hides the effect of a changing market. The first month of real use shows a bias that nobody expected.

Metric and operating point

Samir chose Nadia’s measures before any model: the share of estimates within 10%, and the share that is too high by 10% or more. Now they decide where the team uses the estimate. First, the chosen model, E07, fitted on train and checked on valid:

chosen = boosting(learning_rate=0.05, max_iter=600).fit(train, train[TARGET])
valid_estimate = chosen.predict(valid)
print(report(valid[TARGET], valid_estimate))

The output:

Output

{'median error %': 2.7, 'MAPE %': 12.9, 'within 10%': 0.556, 'over by 10%+': 0.194}

What this tells us. On valid, 55.6% of estimates are within 10%, and 19.4% are too high by 10% or more. So Nadia’s target is met on average, but only just. The chart compares the estimate with the real price for a fixed sample of 400 valid sales.

Estimate against real price for 400 valid sales chosen at random (random_state=0). Points on the line are perfect estimates. Four sales above 2 million dollars are not shown.

Text description of the chart

A scatter plot of 396 valid sales, with the real sale price on the horizontal axis and the estimate on the vertical axis, both from 0 to 2 million dollars. A diagonal line marks a perfect estimate. Most points lie in a band around the line between 200,000 and 1 million dollars, and the band gets wider as the price grows. Seven points are far above the line, with an estimate more than 1.5 times the price, and five are far below it. Of the 396 points, 230 are within 10% of the real price. The four sales above 2 million dollars are left out.

What this tells us. The estimates follow the real prices, and the band around the line gets wider as the price grows. A miss of the same percentage is more dollars for an expensive house, and the expensive houses are also the hardest to estimate. A few sales sit far from the line, and the largest misses below show why.

For which houses can Nadia trust the estimate? As in Slice analysis, we look at groups on valid, with the count next to each number. The first slice is the estimated price, not the real price. Nobody knows the real price on the day of the estimate, and a rule can only use what is known.

BANDS = [0, 400_000, 700_000, 1_000_000, np.inf]
BAND_NAMES = ["under 400k", "400k-700k", "700k-1M", "over 1M"]

def slice_table(part, estimate, by):
    pct = 100 * (part[TARGET] - estimate) / part[TARGET]
    errors = part.assign(pct=pct, abs_pct=pct.abs(), within=pct.abs() <= 10)
    return errors.groupby(by, observed=True).agg(sales=("pct", "size"), median_error=("pct", "median"),
                                                 MAPE=("abs_pct", "mean"), within_10=("within", "mean")).round({"median_error": 1, "MAPE": 1, "within_10": 3})

estimate_band = pd.cut(valid_estimate, BANDS, labels=BAND_NAMES)
print(slice_table(valid, valid_estimate, estimate_band).to_string())

The output:

Output

            sales  median_error  MAPE  within_10
under 400k   1015           3.0  13.8      0.519
400k-700k     813           2.7  11.8      0.599
700k-1M       239          -0.1  10.9      0.653
over 1M       124           7.6  17.1      0.395

Share of valid estimates within 10% of the sale price, by the estimated price. Estimates above 1 million dollars are the weakest band.

Text description of the chart

A column chart of four bands of estimated price, with the share of estimates within 10% of the sale price, from 0 to 100%. Under 400,000 dollars: 51.9%, from 1,015 sales. 400,000 to 700,000: 59.9%, from 813 sales. 700,000 to 1 million: 65.3%, from 239 sales. Over 1 million: 39.5%, from 124 sales, the lowest bar.

What this tells us. Estimates above 1 million dollars are much worse: only 39.5% within 10%, a MAPE of 17.1%. Expensive houses are rare (124 of 2,191), and each one is different: a view, a large garden, an architect. The cheapest band is also weaker than the middle. The middle bands meet Nadia’s target best.

Where to use the estimate

Nadia does not need the estimate to work for every house. She needs to know where it works. The estimated price is known at the application, so a rule can use it. Below a limit, the team uses the estimate as the first number. Above it, an appraiser starts at once. Samir tries a limit of 1 million dollars on valid:

auto = valid_estimate < 1_000_000
print("share of valid sales with an estimate under 1M:", round(float(auto.mean()), 3))
print("those sales:", report(valid[TARGET][auto], valid_estimate[auto]))
print("the others: ", report(valid[TARGET][~auto], valid_estimate[~auto]))

The output:

Output

share of valid sales with an estimate under 1M: 0.943
those sales: {'median error %': 2.6, 'MAPE %': 12.7, 'within 10%': 0.566, 'over by 10%+': 0.193}
the others:  {'median error %': 7.6, 'MAPE %': 17.1, 'within 10%': 0.395, 'over by 10%+': 0.202}

What this tells us. 94.3% of applications have an estimate under 1 million. For them, 56.6% of estimates are within 10%, and 19.3% are too high by 10% or more. Both meet Nadia’s target, but only just. The other 5.7% go straight to an appraiser. This limit is the operating point of this problem. Like a threshold in Thresholds and calibration, Samir chooses it on valid from what each error costs.

In the notebook, you try a lower limit of 700,000 dollars. Predict first: will the share within 10% go up much?

Show the answer

under_700k = valid_estimate < 700_000
policy_700k = report(valid[TARGET][under_700k], valid_estimate[under_700k])
print(round(float(under_700k.mean()), 3), policy_700k)

The output:

Output

0.834 {'median error %': 2.9, 'MAPE %': 12.9, 'within 10%': 0.555, 'over by 10%+': 0.191}

What this tells us. With 700,000 as the limit, only 83.4% of applications get an estimate, and the share within 10% stays at about 56%. The lower limit sends more houses to the appraiser for no gain. The band from 700,000 to 1 million is one of the best. Samir keeps the limit of 1 million.

Errors and slices

The price bands chose the operating point. Now look for other places where the model fails, on valid, with the same slice_table(). In the notebook, you make the same table by area. Predict first: which area is hardest?

Show the answer

area_slices = slice_table(valid, valid_estimate, valid["area"])
print(area_slices.to_string())

The output:

Output

                sales  median_error  MAPE  within_10
area
North and east    688           2.3  11.3      0.622
Seattle           873           3.2  15.0      0.474
South             630           2.4  12.0      0.598

What this tells us. Seattle is the hardest area: 47.4% within 10%, against about 60% elsewhere, with a MAPE of 15.0%. Its houses are older (median year built 1951, against 1984 in both other areas) and smaller (median 1,640 sq ft). An old house’s value depends on its state inside, which the data cannot see. That is a likely reason, not a proven one. The bias is similar in all three areas: the estimates are too low by 2% to 3%. So on this data, the model does not value one area lower than the others. That is a first, partial fairness check: the data has no information about the people, and areas this large can hide differences between neighbourhoods.

The largest misses tell what the model cannot see:

check = valid.assign(estimate=valid_estimate.round(-3), error_pct=(100 * (valid[TARGET] - valid_estimate) / valid[TARGET]).round(0))
columns = ["date", "price", "estimate", "error_pct", "sqft_living", "grade", "condition", "yr_built", "area"]
print(check.sort_values("error_pct")[columns].head(3).to_string(index=False))
print(check.sort_values("error_pct")[columns].tail(3).to_string(index=False))

The output:

Output

      date    price  estimate  error_pct  sqft_living  grade  condition  yr_built           area
2015-02-09 130000.0  416000.0     -220.0         1110      7          4      1960 North and east
2015-02-19 900000.0 2556000.0     -184.0         7120     12          3      2007 North and east
2015-02-18 169317.0  479000.0     -183.0          790      7          3      1908        Seattle
      date    price  estimate  error_pct  sqft_living  grade  condition  yr_built    area
2015-02-04 550000.0  252000.0       54.0         1270      5          4      1908   South
2015-02-16 555750.0  247000.0       56.0         1060      6          2      1913 Seattle
2015-01-30 325000.0  135000.0       58.0          410      4          2      1953 Seattle

What this tells us. The largest over-estimate is a house that sold for 130,000 dollars, while the model says 416,000. Nothing in its description explains that: it is probably not a normal market sale, for example a sale inside a family or a forced sale. The data does not mark such sales. The largest under-estimates are old, small houses that sold for more than double the estimate, probably for their land. An appraiser sees both cases at once; the model cannot.

The final test, once

Samir writes the plan before he runs it. He refits E07 on train and valid together and estimates the test sales once. Then he reports the result, whatever it is.

train_and_valid = pd.concat([train, valid])
final = boosting(learning_rate=0.05, max_iter=600).fit(train_and_valid, train_and_valid[TARGET])
test_estimate = final.predict(test)
final_result = report(test[TARGET], test_estimate)
print("final model on test:", final_result)

The output:

Output

final model on test: {'median error %': 6.4, 'MAPE %': 12.8, 'within 10%': 0.502, 'over by 10%+': 0.126}

And the valuer’s method, the zip-code baseline, learned the same way:

zip_per_sqft = (train_and_valid[TARGET] / train_and_valid["sqft_living"]).groupby(train_and_valid["zipcode"]).median()
print("zip price per sq ft:", report(test[TARGET], test["zipcode"].map(zip_per_sqft) * test["sqft_living"]))

The output:

Output

zip price per sq ft: {'median error %': 5.8, 'MAPE %': 18.5, 'within 10%': 0.349, 'over by 10%+': 0.233}

What this tells us. On the test months, the MAPE is 12.8%, almost the same as on valid, and half of the estimates (50.2%) are within 10%. Only 12.6% are too high by 10% or more. The model beats the valuer’s method on every number. But look at the median error: +6.4%. The estimates are now too low for the typical house.

The slices of the test months are a report, not a reason to change the model:

auto = test_estimate < 1_000_000
print("estimate under 1M:", round(float(auto.mean()), 3), report(test[TARGET][auto], test_estimate[auto]))
print(slice_table(test, test_estimate, test["area"]).to_string())
print(slice_table(test, test_estimate, test["date"].dt.month).to_string())

The output:

Output

estimate under 1M: 0.942 {'median error %': 6.3, 'MAPE %': 12.7, 'within 10%': 0.509, 'over by 10%+': 0.122}
                sales  median_error  MAPE  within_10
area
North and east   1520           5.6  11.3      0.561
Seattle          1936           8.3  15.1      0.418
South            1198           4.8  11.2      0.563
      sales  median_error  MAPE  within_10
date
3      1835           5.1  12.4      0.534
4      2189           7.0  12.9      0.492
5       630           8.3  13.9      0.448

What this tells us. The bias grows from +5.1% in March to +8.3% in May. Spring prices rose after the last sale that the model saw, and the model cannot know that. Under the 1-million limit, 50.9% of estimates are within 10%: the target is met, but only just. Seattle is again the hardest area, with 41.8% within 10%. The model is not wrong about houses; it is behind the market.

What does an error of 12.8% mean for Nadia? The median test house sold for 465,000 dollars, so a typical miss is about 60,000 dollars. That is too large to set a loan amount, which is why the appraiser stays. But it is good enough for its job. On the first day, it finds the applications where the agreed price is far above the estimate.

Last, save the model and look at the whole log:

import joblib

joblib.dump(final, "house_price_model.joblib")
show_log()

The output:

Output

 id                                                         change  cv_mape  cv_std  valid_mape  valid_within_10 decision
E01                            Baseline: the median price of train      NaN     NaN        45.3            0.154 baseline
E02                 Baseline: median price per sq ft x living area      NaN     NaN        35.4            0.188 baseline
E03 Baseline: median price per sq ft in the zip code x living area      NaN     NaN        19.2            0.372 baseline
E04                        Ridge, zip code one-hot, scaled numbers    19.53    0.37        21.3            0.337   reject
E05                                    E04 on the log of the price    13.55    0.05        14.9            0.486     keep
E06     Gradient boosting on the log price, lat and long, defaults    11.71    0.19        13.2            0.554   reject
E07                      E06 with learning_rate 0.05 and 600 trees    11.57    0.24        12.9            0.556     keep
E08                                       E06 without lat and long    22.13    0.39        23.9            0.274   reject

What this tells us. The log has eight rows and the reason for every decision. With the saved model, it lets another analyst repeat the work, or retrain next month in the same way.

Recommendations

Samir writes a short note for Nadia, in her words:

1. Use the estimate as a first number for houses estimated under 1 million dollars (about 94% of applications). On the latest months, half of the estimates were within 10% of the sale price. About 1 in 8 was more than 10% too high.
2. Send houses estimated above 1 million straight to an appraiser. For them, the estimate is within 10% only 4 times in 10.
3. When the agreed price is more than 10% above the estimate, let the appraiser look first. This is the case the estimate is built for. Measure how often the appraisal agrees, and review the rule after three months.
4. Expect the estimate to lag the market. In spring 2015 it was about 6% too low. Retrain it every month with the latest sales, and track the median error each month.
5. Never set a loan amount from the estimate. The appraisal stays the valuation.

And what not to use it for:

• any decision about the applicant, such as approving or refusing a loan;
• houses outside King County, or prices today: the data is from 2014 and 2015;
• unusual properties, such as very large lots or waterfront houses above 1 million;
• any use where the place could stand in for the people who live there, without a review by the lender’s compliance team.

Limits and next steps

• The market moves. The model was 6% too low in the spring. A next step is to add a recent price index as an input. Another is to adjust the estimates each month by the change in the median price.
• The model cannot see inside. Condition and grade are rough. Renovations after the county’s records, damp or a new kitchen are invisible. Repeat sales show it: +54% after a renovation that the data did not record.
• Non-market sales are not marked. Sales inside a family or forced sales look like normal sales, and they are among the worst errors.
• Fairness is checked only by area. The errors are similar in the three areas, but the data has no information about the people, and large areas can hide differences between neighbourhoods. A real lender must test for valuation bias with better data, and must review the use of location with its compliance team.
• One county, one year. Nothing here says that the model works elsewhere or later.

The model card, as in Document use boundaries, in short:

Section	Content
Intended use	A first estimate of the sale price on the day of a mortgage application. It sets the order of appraisals and warns about prices far above the estimate.
Users	The valuation team and loan officers.
Not for	Setting loan amounts, approving or refusing a loan, houses estimated above 1 million dollars, other counties, today’s prices.
Data	King County house sales, May 2014 to May 2015, CC0. Train 2014, valid January and February 2015, test March to May 2015, each house in one part. One error fixed (33 bedrooms → 3).
Model	Gradient boosting on the log of the price, learning rate 0.05, up to 600 trees (early stopping kept 332), 17 inputs with latitude and longitude.
Evaluation	CV MAPE 11.57% (std 0.24). Valid: MAPE 12.9%, 55.6% within 10%. Test (once): MAPE 12.8%, 50.2% within 10%, 12.6% over by 10% or more, median error +6.4%. Zip-code baseline on test: MAPE 18.5%.
Slices	Above 1 million: 39.5% within 10% on valid. Seattle: 47.4% on valid, 41.8% on test. Bias similar across areas.
Operating point	Estimates under 1 million dollars are used (94.3% of valid applications); above, an appraiser at once.
Limits	Lags the market; cannot see inside the house; non-market sales; fairness checked only by area.
Owner and review	Samir Patel; monthly retraining and a monthly check of the median error and of the share within 10%. Stop using the estimate if fewer than 45% of the last month’s estimates are within 10%.

Make it a portfolio piece

This case study shows an employer that you can handle a skewed target, a group split and a moving market. It also shows that you know what a model must not decide. Make it yours with one change. For example, use another city’s open sales data, add a monthly price index, or check the errors by zip code. Then publish a small repository.

What to put in the repository:

house-price-estimate/
  README.md                problem, data and licence, approach, results, limits
  MODEL_CARD.md            the table above, in your words
  notebook.ipynb           the notebook, run from top to bottom, with outputs
  experiment_log.csv       every attempt, kept and rejected
  requirements.txt         pandas==3.0.6, scikit-learn==1.9.1, matplotlib

The README is the page that most people read. A README template:

# A first estimate of house prices for a mortgage lender

## Problem
A lender wants a first price estimate on the day of an application, before the appraisal.
Target: within 10% of the sale price for most houses; few estimates more than 10% too high.

## Data and licence
King County house sales, May 2014 - May 2015 (OpenML 42092), CC0. 21,613 sales.
Fixed one data error (33 bedrooms -> 3). Repeat sales of a house kept in one part.

## Approach
Split by time, grouped by house. Three valuer-style baselines. Ridge and gradient boosting
on the log of the price, compared with grouped 5-fold CV and the one-standard-error rule. 8 experiments, logged.

## Results (test set, used once)
Gradient boosting: MAPE 12.8%, 50% within 10%, 13% over by 10% or more.
Zip-code price per sq ft (baseline): MAPE 18.5%, 35% within 10%.
The model was 6% too low in spring 2015: it lags a rising market.

## Limits
Cannot see inside the house; non-market sales not marked; one county, one year;
fairness checked only by area. Not for loan decisions.

How to describe it in an interview, in two sentences: “I built a first house-price estimate for a mortgage lender, tested by time with each house in one part: half of the estimates fell within 10%, against a third for a valuer’s quick method. The test showed the model 6% behind a rising market, so I recommended monthly retraining and a price limit above which an appraiser decides.”

Go further:

1. Add a monthly price index as an input: the median price per square foot of the last 30 days. Check whether the spring bias falls.
2. Estimate a range instead of one number: two models with loss="quantile" at 0.1 and 0.9, and check how often the sale price falls inside.
3. Check the errors by zip code, with the count of sales next to each one. Look for zip codes where the estimates are too low again and again.
4. Explain one estimate with permutation importance and with the houses nearest on the map, as in Explain predictions.
5. Repeat the study on another public sales dataset, and compare what transfers and what does not.

Knowledge check

Check your understanding

The ridge model's valid MAPE fell from 21.3% to 14.9% when it learned the log of the price instead of the price. The inputs did not change. Explain why.

Show the answer

On the dollar scale, the model minimises squared errors in dollars, so the expensive houses dominate. A miss of 200,000 on a 2-million-dollar house counts far more than a miss of 50,000 on a 250,000-dollar house. Yet in percent, the second miss is the larger one. The effects of size, quality and place on a price also multiply rather than add. On the log scale, a 10% miss has the same size for every house, and multiplying effects become additions, which a linear model can learn. So the model now fits what MAPE measures.

Check your understanding

Chris wants to use the real sale price band (under 400k, over 1M, and so on) to decide which applications get the automatic estimate, because the slices by real price look clearer. What is wrong with this, and what do you use instead?

Show the answer

The real sale price is the target. It is not known on the day of the application, so a rule that uses it cannot run. Use only what is known at the prediction time: the estimated price. The slices and the 1-million limit on this page use the estimated price for that reason. A slice by the real price is fine for understanding errors after the fact, but not for a rule.

Check your understanding

On the test months, the MAPE stayed at about 13%, but the median error moved from +2.7% on valid to +6.4%. What happened, and what would you do before the next month?

Show the answer

The model learned prices up to February 2015, and prices rose in spring. So it now estimates too low for the typical house: it lags the market. The size of the errors did not change much, but their direction did. Before the next month, retrain with all sales up to the latest month, and track the median error every month. Consider an input that follows the market, such as the recent median price per square foot. Do not tune on the test months; use the new months as they arrive.

Recap

• A skewed target with percentage costs calls for percentage metrics: MAPE, the share within 10% and the share too high. A model that learns the log of the price fits them: ridge fell from 21.3% to 14.9% MAPE.
• The data needed care before any model: a 33-bedroom typing error, codes that look like numbers, and 177 repeat sales to keep in one part.
• A valuer’s quick method, the zip-code price per square foot, was the baseline to beat (37% within 10% on valid). Boosting with latitude and longitude reached 55.6%; without the place, the error nearly doubled.
• The one-standard-error rule chose the slower boosting by a small but real margin. The estimated price set the operating point: use the estimate under 1 million dollars, send the rest to an appraiser.
• The test, used once, gave MAPE 12.8% and 50% within 10%, but a bias of +6.4%. The model lags a rising market, so it needs monthly retraining and a monthly check.

Next, the first classification case study: a bank’s credit-risk team, a cost matrix, and a threshold chosen by expected cost.

Next: Predict credit-card default