从最小角回归(LARS)中学到的一个小知识(很短)

【转载请注明出处】http://www.cnblogs.com/mashiqi

(居然有朋友说内容不接地气，那么我就再加一段嘛，请喜欢读笑话的同学直接看第二段)假设这里有一组向量$\left\{ x_i \right\}_{i=1}^n$和一个待投影的向量$u$。假设$u$和每个$x_i$的内积都为正数，也就是说$u$和每个$x_i$的夹角都小于90度。那么当我们把$u$投影到$\left\{ x_i \right\}_{i=1}^n$上时，理所应当地每个$x_i$的系数$\beta_i$也都应该大于零：$$u = x_1\beta_1+\cdots+x_n\beta_n,\beta_i\geq0$$不知道读者们的空间直觉怎么样，反正我最开始就是这么天真的认为的。最近看了Efron的“Least Angle Regression”后，才明白原来不是这样的，自己以前too young了。有些时候系数会变成负的。

(大家好我是第二段)举个栗子，设$$x_1=(0.1793,0.1216,0.9762)^T$$$$x_2=(-0.5947,0.5150,0.6173)^T$$$$x_3=(-0.2013,0.1782,-0.9632)^T$$并令$$u=(3.7952,6.9738,-0.5415)^T$$那么我们可以计算出$$(u,x_1)=(u,x_2)=(u,x_3)=1 > 0$$也就是说$x_1,x_2,x_3$与$u$的夹角都小于90度。但是通过解矩阵方程$X\beta=u$我们可以得到$$\beta = (36.5949,-6.6537,33.3891)^T$$也就是说：$$u = 36.5949x_1 -6.6537x_2 + 33.3891x_3$$看呐！$x_2$君的系数居然是负的！$x_1$和$x_3$一样，$x_2$君也是同样思念着$u$君的，还思念的同样深(内积都是$1$ = =)。可是$x_2$君为了集体的利益甘愿牺牲自己成全整个$\left\{ x_i \right\}_{i=1}^3$集合，这是多么伟大的奉献精神，此处应有掌声，啪啪啪，啪啪啪。。。

下面贴一个小MATLAB代码，自己去体会吧！

 %{
 % This small matlab program show you a unexpected result in
 % high-dimensional geometry: for any set of n-dimensional vectors
 % {x_1,...,x_n}, if these vectors are indepentent, then you can always find
 % an equiangular vector u, so that the inner product (x_i,u)= for all i.
 % BUT, if we project u into {x_i}, some coefficients may be 'negative'!
 %}

 %%
 clear;
 close all;
 %% You can change the following two variable's value
 dimension = 3;
 vectors = 3;

 %% In case of unpredicted problems, dont change the following code.
 if vectors > dimension
 disp('Please set vectors <= dimension.');
 end
 X = randn(dimension,vectors); % every column is a vector
 X = X./repmat(sqrt(sum(X.^2,1)),dimension,1); % standardize
 if rank(X) ~= size(X,2)
 disp('These vectors are not independent. Run again.');
 end
 w = (X'*X)\ones(vectors,);
 u = X*w;
 w % the coefficient
 u % the equiangular vector
 X'*u % the correlation value
 if dimension == 3 % '' is the upper bound of dimensions of humans where we can draw.
 quiver3(0,0,0,X(1,1),X(2,1),X(3,1),'r'),hold on;
 quiver3(0,0,0,X(1,2),X(2,2),X(3,2),'r'),hold on;
 quiver3(0,0,0,X(1,3),X(2,3),X(3,3),'r'),hold on;
 quiver3(0,0,0,u(1),u(2),u(3),'b'),hold off;
 end

总结一句话：高维空间有危险，忽久留= =||