285 lines
9.7 KiB
Plaintext
285 lines
9.7 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"# Optional Lab: Cost Function \n",
|
||
|
|
"<figure>\n",
|
||
|
|
" <center> <img src=\"./images/C1_W1_L3_S2_Lecture_b.png\" style=\"width:1000px;height:200px;\" ></center>\n",
|
||
|
|
"</figure>\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Goals\n",
|
||
|
|
"In this lab you will:\n",
|
||
|
|
"- you will implement and explore the `cost` function for linear regression with one variable. \n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Tools\n",
|
||
|
|
"In this lab we will make use of: \n",
|
||
|
|
"- NumPy, a popular library for scientific computing\n",
|
||
|
|
"- Matplotlib, a popular library for plotting data\n",
|
||
|
|
"- local plotting routines in the lab_utils_uni.py file in the local directory"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"import numpy as np\n",
|
||
|
|
"%matplotlib widget\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"from lab_utils_uni import plt_intuition, plt_stationary, plt_update_onclick, soup_bowl\n",
|
||
|
|
"plt.style.use('./deeplearning.mplstyle')"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Problem Statement\n",
|
||
|
|
"\n",
|
||
|
|
"You would like a model which can predict housing prices given the size of the house. \n",
|
||
|
|
"Let's use the same two data points as before the previous lab- a house with 1000 square feet sold for \\\\$300,000 and a house with 2000 square feet sold for \\\\$500,000.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"| Size (1000 sqft) | Price (1000s of dollars) |\n",
|
||
|
|
"| -------------------| ------------------------ |\n",
|
||
|
|
"| 1 | 300 |\n",
|
||
|
|
"| 2 | 500 |\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"x_train = np.array([1.0, 2.0]) #(size in 1000 square feet)\n",
|
||
|
|
"y_train = np.array([300.0, 500.0]) #(price in 1000s of dollars)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Computing Cost\n",
|
||
|
|
"The term 'cost' in this assignment might be a little confusing since the data is housing cost. Here, cost is a measure how well our model is predicting the target price of the house. The term 'price' is used for housing data.\n",
|
||
|
|
"\n",
|
||
|
|
"The equation for cost with one variable is:\n",
|
||
|
|
" $$J(w,b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})^2 \\tag{1}$$ \n",
|
||
|
|
" \n",
|
||
|
|
"where \n",
|
||
|
|
" $$f_{w,b}(x^{(i)}) = wx^{(i)} + b \\tag{2}$$\n",
|
||
|
|
" \n",
|
||
|
|
"- $f_{w,b}(x^{(i)})$ is our prediction for example $i$ using parameters $w,b$. \n",
|
||
|
|
"- $(f_{w,b}(x^{(i)}) -y^{(i)})^2$ is the squared difference between the target value and the prediction. \n",
|
||
|
|
"- These differences are summed over all the $m$ examples and divided by `2m` to produce the cost, $J(w,b)$. \n",
|
||
|
|
">Note, in lecture summation ranges are typically from 1 to m, while code will be from 0 to m-1.\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The code below calculates cost by looping over each example. In each loop:\n",
|
||
|
|
"- `f_wb`, a prediction is calculated\n",
|
||
|
|
"- the difference between the target and the prediction is calculated and squared.\n",
|
||
|
|
"- this is added to the total cost."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"def compute_cost(x, y, w, b): \n",
|
||
|
|
" \"\"\"\n",
|
||
|
|
" Computes the cost function for linear regression.\n",
|
||
|
|
" \n",
|
||
|
|
" Args:\n",
|
||
|
|
" x (ndarray (m,)): Data, m examples \n",
|
||
|
|
" y (ndarray (m,)): target values\n",
|
||
|
|
" w,b (scalar) : model parameters \n",
|
||
|
|
" \n",
|
||
|
|
" Returns\n",
|
||
|
|
" total_cost (float): The cost of using w,b as the parameters for linear regression\n",
|
||
|
|
" to fit the data points in x and y\n",
|
||
|
|
" \"\"\"\n",
|
||
|
|
" # number of training examples\n",
|
||
|
|
" m = x.shape[0] \n",
|
||
|
|
" \n",
|
||
|
|
" cost_sum = 0 \n",
|
||
|
|
" for i in range(m): \n",
|
||
|
|
" f_wb = w * x[i] + b \n",
|
||
|
|
" cost = (f_wb - y[i]) ** 2 \n",
|
||
|
|
" cost_sum = cost_sum + cost \n",
|
||
|
|
" total_cost = (1 / (2 * m)) * cost_sum \n",
|
||
|
|
"\n",
|
||
|
|
" return total_cost"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Cost Function Intuition"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"<img align=\"left\" src=\"./images/C1_W1_Lab02_GoalOfRegression.PNG\" style=\" width:380px; padding: 10px; \" /> Your goal is to find a model $f_{w,b}(x) = wx + b$, with parameters $w,b$, which will accurately predict house values given an input $x$. The cost is a measure of how accurate the model is on the training data.\n",
|
||
|
|
"\n",
|
||
|
|
"The cost equation (1) above shows that if $w$ and $b$ can be selected such that the predictions $f_{w,b}(x)$ match the target data $y$, the $(f_{w,b}(x^{(i)}) - y^{(i)})^2 $ term will be zero and the cost minimized. In this simple two point example, you can achieve this!\n",
|
||
|
|
"\n",
|
||
|
|
"In the previous lab, you determined that $b=100$ provided an optimal solution so let's set $b$ to 100 and focus on $w$.\n",
|
||
|
|
"\n",
|
||
|
|
"<br/>\n",
|
||
|
|
"Below, use the slider control to select the value of $w$ that minimizes cost. It can take a few seconds for the plot to update."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"plt_intuition(x_train,y_train)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The plot contains a few points that are worth mentioning.\n",
|
||
|
|
"- cost is minimized when $w = 200$, which matches results from the previous lab\n",
|
||
|
|
"- Because the difference between the target and pediction is squared in the cost equation, the cost increases rapidly when $w$ is either too large or too small.\n",
|
||
|
|
"- Using the `w` and `b` selected by minimizing cost results in a line which is a perfect fit to the data."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Cost Function Visualization- 3D\n",
|
||
|
|
"\n",
|
||
|
|
"You can see how cost varies with respect to *both* `w` and `b` by plotting in 3D or using a contour plot. \n",
|
||
|
|
"It is worth noting that some of the plotting in this course can become quite involved. The plotting routines are provided and while it can be instructive to read through the code to become familiar with the methods, it is not needed to complete the course successfully. The routines are in lab_utils_uni.py in the local directory."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Larger Data Set\n",
|
||
|
|
"It's use instructive to view a scenario with a few more data points. This data set includes data points that do not fall on the same line. What does that mean for the cost equation? Can we find $w$, and $b$ that will give us a cost of 0? "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"x_train = np.array([1.0, 1.7, 2.0, 2.5, 3.0, 3.2])\n",
|
||
|
|
"y_train = np.array([250, 300, 480, 430, 630, 730,])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"In the contour plot, click on a point to select `w` and `b` to achieve the lowest cost. Use the contours to guide your selections. Note, it can take a few seconds to update the graph. "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"plt.close('all') \n",
|
||
|
|
"fig, ax, dyn_items = plt_stationary(x_train, y_train)\n",
|
||
|
|
"updater = plt_update_onclick(fig, ax, x_train, y_train, dyn_items)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Above, note the dashed lines in the left plot. These represent the portion of the cost contributed by each example in your training set. In this case, values of approximately $w=209$ and $b=2.4$ provide low cost. Note that, because our training examples are not on a line, the minimum cost is not zero."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Convex Cost surface\n",
|
||
|
|
"The fact that the cost function squares the loss ensures that the 'error surface' is convex like a soup bowl. It will always have a minimum that can be reached by following the gradient in all dimensions. In the previous plot, because the $w$ and $b$ dimensions scale differently, this is not easy to recognize. The following plot, where $w$ and $b$ are symmetric, was shown in lecture:"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"soup_bowl()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"# Congratulations!\n",
|
||
|
|
"You have learned the following:\n",
|
||
|
|
" - The cost equation provides a measure of how well your predictions match your training data.\n",
|
||
|
|
" - Minimizing the cost can provide optimal values of $w$, $b$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.7.6"
|
||
|
|
},
|
||
|
|
"toc-autonumbering": false
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 5
|
||
|
|
}
|