对于multiple features 的问题(设有n个feature),hypothesis 应该改写成

\[\mathit{h} _{\theta}(x) = \theta_{0} + \theta_{1}\cdot x_{1}+\theta_{2}\cdot x_{2}+\theta_{3}\cdot x_{3}+\dots+\theta_{n}\cdot x_{n}
\]

其中:

\[x=\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\\ \vdots \\ x_{n} \end{bmatrix}\in {\Bbb R}^n \;,\; \theta=\begin{bmatrix}\theta_{1}\\ \theta_{2}\\ \theta_{3}\\ \vdots \\ \theta_{n} \end{bmatrix}\in {\Bbb R}^n
\]

为便于表达,可令\(x_{0}=1\),则

\[\mathit{h} _{\theta}(x) = \theta_{0}\cdot x_{0} + \theta_{1}\cdot x_{1}+\theta_{2}\cdot x_{2}+\theta_{3}\cdot x_{3}+\dots+\theta_{n}\cdot x_{n}
\]

\[\quad\; x=\begin{bmatrix}x_{0} \\ x_{1}\\ x_{2}\\ x_{3}\\ \vdots \\ x_{n} \end{bmatrix}\in {\Bbb R}^{n+1}\;,\; \theta=\begin{bmatrix}\theta_{0} \\ \theta_{1}\\ \theta_{2}\\ \theta_{3}\\ \vdots \\ \theta_{n} \end{bmatrix}\in {\Bbb R}^{n+1}
\]

即:

\[h_{\theta}(x) = \theta^{\rm T}x
\]

multivariate linear regression(多元线性回归):\(h_{\theta}(x) = \theta^{\rm T}x\)

cost function:

\[J(\theta_{0}, \theta_{1}) = \frac{1}{2m} \sum_{i=1}^{m} (h_{\theta}(x^{(i)})-y^{(i)})^{2} = \frac{1}{2m} \sum_{i=1}^{m} (\theta^{\rm T}x^{(i)}-y^{(i)})^{2} = \frac{1}{2m} \sum_{i=1}^{m} (\sum_{j=0}^{n} \theta_{j}x_{j}^{(i)}-y^{(i)})^{2}
\]

\(\therefore\) 梯度下降算法的循环内容变成\(\theta_{j}\; \text{:= } \theta_{j} - \alpha\frac{\partial}{\partial \theta_{j}}J(\theta) \qquad (j = 0,1,2...,n)\)

\(\therefore\) gradient descent algorism(\(n \ge 1\), simultaneously update \(\theta_{j}\) for \(j=0,1,2,3\dots,n\)):

\[\text{repeat until convergence}\{\qquad\qquad\qquad\qquad\qquad\\
\qquad\qquad\qquad\qquad \theta_{j}\; \text{:= } \theta_{j} - \alpha\frac{1}{m} \sum_{i=1}^{m} (h_{\theta}(x^{(i)})-y^{(i)})x_{j}^{(i)} \qquad (j = 0,1,2...,n)\\
\}\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
\]

[实现的注意事项:比如作业中的某一道题,循环判断条件我用了\(\sum\Delta\theta_{j}^2>C\),其中\(C\)是且必须是某个很小的常数(如0.0000000001),不然出来的结果不准确,而\(\alpha\)可以相对大点但也不能太大(如0.001)]

技巧:

Feature Scaling(特征缩放)

​ 对x各项元素(也就是feature)除以这个feature的最大值,得到一个百分比的数,这样x就不会因为各元素之间的数量级差异导致梯度算法的性能变差

​ 也就是说,把x各项元素(也就是feature)的值约束到\([-1,1]\)之间

​ 范围在 \([-3,3]\) ~ \([-\frac{1}{3}, \frac{1}{3}]\)的feature是可以接受的,而过小或者过大的feature则需要进行feature scaling

Mean Normalization(均值归一化)

​ replace \(x_{i}​\) with \(x_{i}-\mu_{i}​\) to make features have approximately zero mean (But do not apply to \(x_{0} = 1​\), 因为所有的\(x_0=1​\)故不可能平均值为0)

​ 说明:也就是把feature的均值归一为0,其中\(\mu_{i}\)是\(x_i\)的平均值

​ \(e.g.:x_1= \frac{size-1000}{2000},\quad x_2 = \frac{\#bedrooms-2}{5},\qquad s.t.\, -0.5\le x_1\le 0.5,\; -0.5\le x_2\le 0.5\)

表达出来即是:

\[x_i = \frac{x_i-\mu_i}{s_i}
\]

​ 其中 \(\mu_i \text{ is average value, }s_i \text{ is the range of the feature[ == max(feature) - min(feature)] or feature's Standard Deviation}\)

【啊好吧,到这里讲了如何选择\(\alpha\),不用我自己摸索了】

Declare convergence if \(J(\theta)\) decreases by less than \(10^{-3}\) in one iteration. (#循环的判断条件)

To choose \(\alpha\), try \(\dots,0.001,0.003,0.01,0.03,0.1,0.3,1,\dots\) \((x_{n+1} = x_n * 3)\) (#\(\alpha\)的选择)

Try to pick the largest possible value, or the value just slightly smaller than the largest reasonable value that I found

统合特征,比如用面积代替长和宽

polynomial regression(多项式回归)

例:

​ \(\begin{align}h_{\theta}(x) &= \theta_0 + \theta_1\cdot x_1+ \theta_2\cdot x_2+\theta_3\cdot x_3\\&=\theta_0 + \theta_1\cdot (size)+ \theta_2\cdot (size)^2+\theta_3\cdot (size)^3 \end{align}\)

由于到后面不同指数的size的值相差甚远,因此需要对其进行均值归一化

其实指数不一定要上升,对于只增不减的某些函数而言,也可以选用:

​ \(\begin{align}h_{\theta}(x) &=\theta_0 + \theta_1\cdot (size)+ \theta_2\cdot \sqrt{(size)} \end{align}\)

其均值归一化过程(已知①②③):

①model is \(\begin{align}h_{\theta}(x) &=\theta_0 + \theta_1\cdot (size)+ \theta_2\cdot \sqrt{(size)} \end{align}\)

​ ②size range from 1 to 1000(feet\(^2\))

​ ③implement this by fitting a model \(\begin{align}h_{\theta}(x) &=\theta_0 + \theta_1\cdot x_1+ \theta_2\cdot x_2 \end{align}\)

​ \(\therefore\) \(x_1,x_2\) should satisfy \(x_1 = \frac{size}{1000}, \quad x_2=\frac{\sqrt{(size)}}{\sqrt{1000}}\)

One important thing to keep in mind is, if you choose your features this way then feature scaling becomes very important.

Normal Equation(正规方程)

可以直接求出\(\theta\)的最优解

其实就是,求导,解出导数为0

一般直接想到的解法是:分别求解所有变量的偏导等于零:\(\frac{\partial}{\partial \theta_j}f(\theta) = 0\)

其实可以这么做:

令\(X = \begin{bmatrix}x_{10} & x_{11} &x_{12} & \cdots & x_{1n} \\ x_{20} & x_{21} &x_{22} & \cdots & x_{2n} \\ \vdots & \vdots &\vdots & \ddots & \vdots \\ x_{m0} & x_{m1} &x_{m2} & \cdots & x_{mn} \end{bmatrix}\quad,\quad y = \begin{bmatrix} y_1\\ y_2 \\ \vdots \\ y_m \end{bmatrix} ​\)

则 \(\large\theta = (X^TX)^{-1}X^Ty\)

$ x^{(i)} = \begin{bmatrix} x_0^{(i)} \ x_1^{(i)}\ x_2^{(i)} \ \vdots \ x_n^{(i)} \end{bmatrix}$ 则 design matrix(设计矩阵) \(X = \begin{bmatrix} (x^{(1)})^T \\ (x^{(2)})^T\\ (x^{(3)})^T \\ \vdots \\ (x^{(m)})^T \end{bmatrix}\)

pinv(x'*x)*x'*y

(不需要归一化特征变量)

与Gradient Descent 的比较

Gradient Descent Normal Equation
Need to choose alpha No need to choose alpha
Needs many iterations No need to iterate
\(O (kn^2)\) \(O (n^3)\), need to calculate inverse of \(X^TX\)
Works well when n is large Slow if n is very large

选择:

​ \(lg(n)\ge 4:\text{ gradient descent} \\lg(n)\le 4: \text{ normal equation}\)

计算Normal Equation要\(X^TX\)是可逆的,但是如果它不可逆(Non-invertible)呢?

Octave 中的pinv()inv()都能求逆,但是pinv()能展现数学上的过程,即使矩阵不可逆

如果\(X^TX\)不可逆:

  • 首先看看都没有redundant features, 比如一个feature是单位为feet,而另一个feature仅仅是那个feet单位换算成m,有就删掉redundant的feature
  • check if I may have too many features. 若是, I would either delete some features if I can bear to use fewer features or else I would consider using regularization.

Machine Learning--week2 多元线性回归、梯度下降改进、特征缩放、均值归一化、多项式回归、正规方程与设计矩阵的更多相关文章

  1. machine learning 之 多元线性回归

    整理自Andrew Ng的machine learning课程 week2. 目录: 多元线性回归 Multivariates linear regression /MLR Gradient desc ...

  2. 线性回归 Linear regression(2)线性回归梯度下降中学习率的讨论

    这篇博客针对的AndrewNg在公开课中未讲到的,线性回归梯度下降的学习率进行讨论,并且结合例子讨论梯度下降初值的问题. 线性回归梯度下降中的学习率 上一篇博客中我们推导了线性回归,并且用梯度下降来求 ...

  3. [Machine Learning] 单变量线性回归(Linear Regression with One Variable) - 线性回归-代价函数-梯度下降法-学习率

    单变量线性回归(Linear Regression with One Variable) 什么是线性回归?线性回归是利用数理统计中回归分析,来确定两种或两种以上变量间相互依赖的定量关系的一种统计分析方 ...

  4. 斯坦福机器学习视频笔记 Week2 多元线性回归 Linear Regression with Multiple Variables

    相比于week1中讨论的单变量的线性回归,多元线性回归更具有一般性,应用范围也更大,更贴近实际. Multiple Features 上面就是接上次的例子,将房价预测问题进行扩充,添加多个特征(fea ...

  5. [笔记]线性回归&梯度下降

    一.总述 线性回归算法属于监督学习的一种,主要用于模型为连续函数的数值预测. 过程总得来说就是初步建模后,通过训练集合确定模型参数,得到最终预测函数,此时输入自变量即可得到预测值. 二.基本过程 1. ...

  6. [Machine Learning]学习笔记-线性回归

    模型 假定有i组输入输出数据.输入变量可以用\(x^i\)表示,输出变量可以用\(y^i\)表示,一对\(\{x^i,y^i\}\)名为训练样本(training example),它们的集合则名为训 ...

  7. [笔记]机器学习(Machine Learning) - 01.线性回归(Linear Regression)

    线性回归属于回归问题.对于回归问题,解决流程为: 给定数据集中每个样本及其正确答案,选择一个模型函数h(hypothesis,假设),并为h找到适应数据的(未必是全局)最优解,即找出最优解下的h的参数 ...

  8. Machine Learning - 第3周(Logistic Regression、Regularization)

    Logistic regression is a method for classifying data into discrete outcomes. For example, we might u ...

  9. (原创)Stanford Machine Learning (by Andrew NG) --- (week 1) Linear Regression

    Andrew NG的Machine learning课程地址为:https://www.coursera.org/course/ml 在Linear Regression部分出现了一些新的名词,这些名 ...

随机推荐

  1. java对excel文件内容读写修改操作

    Read.java package domain; import java.io.FileInputStream; import java.io.InputStream; import jxl.Cel ...

  2. java 查看线程死锁

    那我们怎么确定一定是死锁呢?有两种方法. 1>使用JDK给我们的的工具JConsole,可以通过打开cmd然后输入jconsole打开. 1)连接到需要查看的进程.

  3. ASP.NET页面使用JQuery EasyUI生成Dialog后台取值为空

    原因: JQuery EasyUI生成Dialog后原来的文档结构发生了变化,原本在form里的内容被移动form外面,提交到后台后就没有办法取值了. 解决办法: 在生成Dialog后将它append ...

  4. One Technical Problem: Can one process load two different c libraries simutaneously, such as uclibc and glibc?

    For some special reasons, there is a possible case which need to load uclibc and glibc in one proces ...

  5. [dev] 啥是Virtual Private Network

    先来读wiki:https://en.wikipedia.org/wiki/Virtual_private_network 摘要: VPNs can be either remote-access ( ...

  6. 使用moment.js结合filter过滤器格式化时间

    <td>{{item.ctime | timeFormat('yyyy-MM-dd')}}</td> //pattern = "" 形参的默认值,如果传过来 ...

  7. 保存退出vi编辑

    保存命令按i进入编辑模式,编辑完成按ESC键 跳到命令模式,然后: :w 保存文件但不退出vi:w file 将修改另外保存到file中,不退出vi:w! 强制保存,不推出vi:wq 保存文件并退出v ...

  8. 认识拨号计划-dialplan

    拨号计划是 FreeSWITCH 中至关重要的一部分.它的主要作用就是对电话进行路由(从这一点上来说,相当于一个路由表).说的简明一点,就是当一个用户拨号时,对用户所拨的号码进行分析,进而决定下一步该 ...

  9. JavaScript事件起泡与捕获

    // 向 <div> 元素添加事件句柄 document.getElementById("myDIV").addEventListener("mousemov ...

  10. Oracle数据库分组排序

    select row_number() over(partition by oea03 order by oea02 desc) num,oea01,oea02,oea03 from oea_file ...