机器学习【算法】：KNN近邻

引言

　　本文讨论的kNN算法是监督学习中分类方法的一种。所谓监督学习与非监督学习，是指训练数据是否有标注类别，若有则为监督学习，若否则为非监督学习。监督学习是根据输入数据（训练数据）学习一个模型，能对后来的输入做预测。在监督学习中，输入变量与输出变量可以是连续的，也可以是离散的。若输入变量与输出变量均为连续变量，则称为回归；输出变量为有限个离散变量，则称为分类；输入变量与输出变量均为变量序列，则称为标注

　　有监督的分类学习

KNN算法的基本要素大致有三个：

　　1、K 值的选择（即输入新实例要取多少个训练实例点作为近邻）

　　2、距离的度量（欧氏距离，曼哈顿距离等）

　　3、分类决策规则（常用的方式是取k个近邻训练实例中类别出现次数最多者作为输入新实例的类别）

使用方式：

　　K 值会对算法的结果产生重大影响。K值较小意味着只有与输入实例较近的训练实例才会对预测结果起作用，容易发生过拟合；如果 K 值较大，优点是可以减少学习的估计误差，缺点是学习的近似误差增大，这时与输入实例较远的训练实例也会对预测起作用，是预测发生错误。在实际应用中，K 值一般选择一个较小的数值，通常采用交叉验证的方法来选择最有的 K 值。随着训练实例数目趋向于无穷和 K=1 时，误差率不会超过贝叶斯误差率的2倍，如果K也趋向于无穷，则误差率趋向于贝叶斯误差率。

　　距离度量一般采用 Lp 距离，当p=2时，即为欧氏距离，在度量之前，应该将每个属性的值规范化，这样有助于防止具有较大初始值域的属性比具有较小初始值域的属性的权重过大（规范）

　　算法中的分类决策规则往往是多数表决，即由输入实例的 K 个最临近的训练实例中的多数类决定输入实例的类别

简述

　　KNN分类算法，是理论上比较成熟的方法，也是最简单的机器学习算法之一。

　　该方法的思路是：如果一个样本在特征空间中的k个最相似(即特征空间中最邻近)的样本中的大多数属于某一个类别，则该样本也属于这个类别。

　　KNN算法中，所选择的邻居都是已经正确分类的对象。该方法在定类决策上只依据最邻近的一个或者几个样本的类别来决定待分样本所属的类别。

　　一个对于KNN算法解释最清楚的图如下所示：

　　蓝方块和红三角均是已有分类数据，当前的任务是将绿色圆块进行分类判断，判断是属于蓝方块或者红三角。

　　当然这里的分类还跟K值是有关的：

　　如果K=3(实线圈)，红三角占比2/3，则判断为红三角；

　　如果K=5(虚线圈)，蓝方块占比3/5，则判断为蓝方块。

　　由此可以看出knn算法实际上根本就不用进行训练，而是直接进行计算的，训练时间为0，计算时间为训练集规模n。

　　knn算法在分类时主要的不足是，当样本不平衡时，如果一个类的样本容量很大，而其他类样本容量很小时，有可能导致当输入一个新样本时，该样本的 K 个邻居中大容量类的样本占多数。

*算法实现

1、函数

输入：训练数据集，其中为训练实例的特征向量（ $x$ i为特征值），为训练实例的类别（ $y$ i为 $x$ i对应的类别）

输出：新输入实例 $x$ 所属类别 $y$ （实例类别）

1、根据给定的距离度量，在训练集T中找到与 $x$ 最近的k个点，涵盖这k个点的邻域记为 $N_k(x)$ （上图中实体圆圈包含的数据）

2、在 $N_k(x)$ 中根据分类决策规则（如多数表决）决定 $x$ 所属的类别 $y$ （即统计 $N_k(x)$ 哪个 $y$ 类别出现的次数多，则取哪个）：

其中I为指示函数，仅当 $y_i = c_j$ 时I(*)的值为1，否则为0.（argmax公式表示后面取值最大时，Cj的变量值，公式的目的也是求此值； $x$ i为 $N_k(x)$ 中的值，Cj为涵盖领域存在的类别；利用求和公式∑，当Cj为C1时， $x$ i∈ $N_k(x)$ 分别代入，得出值; $x$ i为已知数据，Cj值不断变化,求出公式的最大值，得出最终 $y$ 得取值）

2、距离度量方式

较为常用的距离度量方式是欧式距离，定义可以使用其他更为一般的 $L_p$ 距离或闵科夫斯基（Minkowski）距离。

设特征空间 $\chi$ 为n为实数向量空间 $R^n$ ， $x_i,x_j \in \chi,x_i=(x^{(1)}_i,x^{(2)}_i,x^{(3)}_i,...,x^{(n)}_i)^T$ ， $x_j=(x^{(1)}_j,x^{(2)}_j,x^{(3)}_j,...,x^{(n)}_j)^T$ ， $x_i,x_j$ 的 $L_p$ 距离可定义为：

　　 $L_p(x_i,x_j) = (\sum_{l=1}^{n}{|x_i^{(l)}-x_j^{(l)}|^p} )^{1/p}$ 　　　　闵科夫斯基距离

　　 $L_p(x_i,x_j) = (\sum_{l=1}^{n}{|x_i^{(l)}-x_j^{(l)}|^2} )^{1/2}$ 　　　　欧氏距离，p取2

　　 $L_p(x_i,x_j) = \sum_{l=1}^{n}{|x_i^{(l)}-x_j^{(l)}} |$ 　　　　　　　曼哈顿距离，p取1

　　 $L_p(x_i,x_j) = \max \limits_l {|x_i^{(l)}-x_j^{(l)}} |$ 　　　　　　 p取 $\infty$

3、K值选择

　　一般会先选择较小的k值，然后进行交叉验证选取最优的k值。k值较小时，整体模型会变得复杂，且对近邻的训练实例点较为敏感，容易出现过拟合。k值较大时，模型则会趋于简单，此时较远的训练实例点也会起到预测作用，容易出现欠拟合，特殊的，当k取N时，此时无论输入实例是什么，都会将其预测为属于训练实例中最多的类别。

4、分类决策规则

　　KNN学习模型：输入

　　通过学习得到决策函数：输出类别

假设分类损失函数为0-1损失函数，即分类正确时损失函数值为0，分类错误时则为1。假如给预测类别为,即；同时由式子（1）可知k邻域的样本点对学习模型的贡献度是均等的，则kNN学习模型误分类率为

若要最小化误分类率，则应

所以，最大表决规则等价于经验风险最小化。

k值得选取对kNN学习模型有着很大的影响。若k值过小，预测结果会对噪音样本点显得异常敏感。特别地，当k等于1时，kNN退化成最近邻算法，没有了显式的学习过程。若k值过大，会有较大的邻域训练样本进行预测，可以减小噪音样本点的减少；但是距离较远的训练样本点对预测结果会有贡献，以至于造成预测结果错误。下图给出k值的选取对于预测结果的影响：

前面提到过，k邻域的样本点对预测结果的贡献度是相等的；但距离更近的样本点应有更大的相似度，其贡献度应比距离更远的样本点大。可以加上权值进行修正，则最大表决原则变成：

应用案例

对data2.txt中的数据进行运算，取前10%的数据进行验证，90%的数据做训练，根据K值的不同，算出误差率，选出最佳K值；

    8.326976    0.953952
    7.153469    1.673904
    1.441871    0.805124
    13.147394    0.428964
    1.669788    0.134296
    10.141740    1.032955
    6.830792    1.213192
    13.276369    0.543880
    8.631577    0.749278
    12.273169    1.508053
    3.723498    0.831917
    8.385879    1.669485
    4.875435    0.728658
    4.680098    0.625224
    15.299570    0.331351
    1.889461    0.191283
    7.516754    1.269164
    14.239195    0.261333
    0.000000    1.250185
    10.528555    1.304844
    3.540265    0.822483
    2.991551    0.833920
    5.297865    0.638306
    6.593803    0.187108
    2.816760    1.686209
    12.458258    0.649617
    0.000000    1.656418
    9.968648    0.731232
    1.364838    0.640103
    0.230453    1.151996
    11.865402    0.882810
    0.120460    1.352013
    8.545204    1.340429
    5.856649    0.160006
    9.665618    0.778626
    9.778763    1.084103
    4.932976    0.632026
    2.216246    0.587095
    14.305636    0.632317
    12.591889    0.686581
    3.424649    1.004504
    0.000000    0.147573
    8.533823    0.205324
    9.829528    0.238620
    11.492186    0.263499
    3.570968    0.832254
    1.771228    0.207612
    3.513921    0.991854
    4.398172    0.975024
    4.276823    1.174874
    5.946014    1.614244
    13.798970    0.724375
    10.393591    1.663724
    3.007577    0.297302
    1.031938    0.486174
    4.751212    0.064693
    3.692269    1.655113
    10.448091    0.267652
    10.585786    0.329557
    1.604501    0.069064
    3.679497    0.961466
    3.795146    0.696694
    2.531885    1.659173
    9.733340    0.977746
    6.093067    1.413798
    7.712960    1.054927
    11.470364    0.760461
    2.886529    0.934416
    10.054373    1.138351
    9.972470    0.881876
    2.335785    1.366145
    11.375155    1.528626
    0.000000    0.605619
    4.126787    0.357501
    6.319522    1.058602
    8.680527    0.086955
    14.856391    1.129823
    2.454285    0.222380
    7.292202    0.548607
    8.745137    0.857348
    8.579001    0.683048
    2.507302    0.869177
    11.415476    1.505466
    4.838540    1.680892
    10.339507    0.583646
    6.573742    1.151433
    6.539397    0.462065
    2.209159    0.723567
    11.196378    0.836326
    4.229595    0.128253
    9.505944    0.005273
    8.652725    1.348934
    17.101108    0.490712
    7.871839    0.717662
    8.262131    1.361646
    9.015635    1.658555
    9.215351    0.806762
    6.375007    0.033678
    2.262014    1.022169
    5.677110    0.709469
    11.293017    0.207976
    6.590043    1.353117
    4.711960    0.194167
    8.768099    1.108041
    11.502519    0.545097
    4.682812    0.578112
    12.446578    0.300754
    12.908384    1.657722
    12.601108    0.974527
    3.929456    0.025466
    9.751503    1.182050
    3.043767    0.888168
    4.391522    0.807100
    11.695276    0.679015
    7.879742    0.154263
    5.613163    0.933632
    9.140172    0.851300
    4.258644    0.206892
    6.799831    1.221171
    8.752758    0.484418
    1.123033    1.180352
    10.833248    1.585426
    3.051618    0.026781
    5.308409    0.030683
    1.841792    0.028099
    2.261978    1.605603
    11.573696    1.061347
    8.038764    1.083910
    10.734007    0.103715
    9.661909    0.350772
    9.005850    0.548737
    0.000000    0.539131
    5.757140    1.062373
    9.164656    1.624565
    1.318340    1.436243
    14.075597    0.695934
    10.107550    1.308398
    7.960293    1.219760
    6.317292    0.018209
    12.664194    0.595653
    2.906644    0.581657
    2.388241    0.913938
    6.024471    0.486215
    7.226764    1.255329
    4.183997    1.275290
    11.850211    1.096981
    11.661797    1.167935
    3.574967    0.494666
    0.000000    0.107475
    7.937657    0.904799
    3.365027    1.014085
    0.000000    0.367491
    13.860672    1.293270
    10.306714    1.211594
    7.228002    0.670670
    4.508740    1.036192
    0.366328    0.163652
    3.299444    0.575152
    0.573287    0.607915
    9.183738    0.012280
    7.842646    1.060636
    4.750964    0.558240
    11.438702    1.556334
    8.243063    1.122768
    7.949017    0.271865
    7.875477    0.227085
    9.569087    0.364856
    7.750103    0.869094
    0.000000    1.515293
    3.396030    0.633977
    11.916091    0.025294
    0.460758    0.689586
    13.087566    0.476002
    4.589016    1.672600
    8.397217    1.534103
    5.562772    1.689388
    10.905159    0.619091
    1.311441    1.169887
    10.647170    0.980141
    0.000000    0.481918
    8.503025    0.830861
    0.436880    1.395314
    6.127867    1.102179
    12.112492    0.359680
    1.264968    1.141582
    6.067568    1.327047
    8.010964    1.681648
    3.791084    0.304072
    11.773195    1.262621
    8.339588    1.443357
    2.563092    1.464013
    5.954216    0.953782
    9.288374    0.767318
    3.976796    1.043109
    8.585227    1.455708
    1.271946    0.796506
    0.000000    0.242778
    0.000000    0.089749
    11.521298    0.300860
    1.139447    0.415373
    5.699090    1.391892
    2.449378    1.322560
    0.000000    1.228380
    3.168365    0.053993
    10.428610    1.126257
    2.943070    1.446816
    10.441348    0.975283
    12.478764    1.628726
    5.856902    0.363883
    2.476420    0.096075
    1.826637    0.811457
    4.324451    0.328235
    1.376085    1.178359
    5.342462    0.394527
    11.835521    0.693301
    12.423687    1.424264
    12.161273    0.071131
    8.148360    1.649194
    1.531067    1.549756
    3.200912    0.309679
    8.862691    0.530506
    6.370551    0.369350
    2.468841    0.145060
    11.054212    0.141508
    2.037080    0.715243
    13.364030    0.549972
    10.249135    0.192735
    10.464252    1.669767
    9.424574    0.013725
    4.458902    0.268444
    0.000000    0.575976
    9.686082    1.029808
    13.649402    1.052618
    13.181148    0.273014
    3.877472    0.401600
    1.413952    0.451380
    4.248986    1.430249
    8.779183    0.845947
    4.156252    0.097109
    5.580018    0.158401
    15.040440    1.366898
    12.793870    1.307323
    3.254877    0.669546
    10.725607    0.588588
    8.256473    0.765891
    8.033892    1.618562
    10.702532    0.204792
    5.062996    1.132555
    10.772286    0.668721
    1.892354    0.837028
    1.019966    0.372320
    15.546043    0.729742
    11.638205    0.409125
    3.427886    0.975616
    11.246174    1.475586
    0.000000    0.645045
    0.000000    1.424017
    8.242553    0.279069
    8.700060    0.101807
    0.812344    0.260334
    2.448235    1.176829
    13.230078    0.616147
    0.236133    0.340840
    11.155826    0.335131
    11.029636    0.505769
    2.901181    1.646633
    3.924594    1.143120
    2.524806    1.292848
    3.527474    1.449158
    3.384281    0.889268
    0.000000    1.107592
    11.898890    0.406441
    3.529892    1.375844
    11.442677    0.696919
    10.308145    0.422722
    8.540529    0.727373
    7.156949    1.691682
    0.720675    0.847574
    0.229405    1.038603
    3.399331    0.077501
    6.157239    0.580133
    1.239698    0.719989
    6.036854    0.016548
    5.258665    0.933722
    12.393001    1.571281
    9.627613    0.935842
    11.130453    0.597610
    8.842595    0.349768
    10.690010    1.456595
    5.714718    1.674780
    3.052505    1.335804
    0.000000    0.059025
    9.945307    1.287952
    2.719723    1.142148
    11.154055    1.608486
    2.687918    0.660836
    10.037847    0.962245
    12.404762    1.112080
    10.237305    0.633422
    4.745392    0.662520
    4.639461    1.569431
    3.149310    0.639669
    13.406875    1.639194
    6.068668    0.881241
    9.477022    0.899002
    3.897620    0.560201
    5.463615    1.203677
    3.369267    1.575043
    5.234562    0.825954
    0.000000    0.722170
    12.979069    0.504068
    5.376564    0.557476
    13.527910    1.586732
    2.196889    0.784587
    10.691748    0.007509
    1.659242    0.447066
    8.369667    0.656697
    13.157197    0.143248
    8.199667    0.908508
    4.441669    0.439381
    9.846492    0.644523
    0.019540    0.977949
    8.253774    0.748700
    6.038620    1.509646
    6.091587    1.694641
    8.986820    1.225165
    11.508473    1.624296
    8.807734    0.713922
    0.000000    0.816676
    8.889202    1.665414
    3.178117    0.542752
    7.013795    0.139909
    9.605014    0.065254
    1.230540    1.331674
    10.412811    0.890803
    0.000000    0.567161
    9.699991    0.122011
    0.000000    0.061191
    4.455293    0.272135
    3.020977    1.502803
    8.099278    0.216317
    1.157764    1.603217
    10.105396    0.121067
    11.230148    0.408603
    9.070058    0.011379
    0.566460    0.478837
    0.000000    0.487300
    8.956369    1.193484
    1.523057    0.620528
    2.749006    0.169855
    9.235393    0.188350
    10.555573    0.403927
    6.956372    1.519308
    0.636281    1.273984
    3.574737    0.075163
    9.032486    1.461809
    5.958993    0.023012
    2.435300    1.211744
    10.539731    1.638248
    7.646702    0.056513
    20.919349    0.644571
    1.424726    0.838447
    6.748663    0.890223
    2.289167    0.114881
    5.548377    0.402238
    6.057227    0.432666
    10.828595    0.559955
    11.318160    0.271094
    13.265311    0.633903
    0.000000    1.496715
    6.517133    0.402519
    4.934374    1.520028
    10.151738    0.896433
    2.425781    1.559467
    9.778962    1.195498
    12.219950    0.657677
    7.394151    0.954434
    8.518535    0.742546
    2.798700    0.662632
    0.637930    0.617373
    10.750490    0.097415
    0.625382    0.140969
    10.027968    0.282787
    9.817347    0.364197
    0.646828    1.266069
    3.347111    0.914294
    11.816892    0.193798
    0.000000    1.480198
    10.945666    0.993219
    10.244706    0.280539
    2.579801    1.149172
    2.630410    0.098869
    11.746200    1.695517
    8.104232    1.326277
    12.409743    0.790295
    12.167844    1.328086
    3.198408    0.299287
    16.055513    0.541052
    7.138659    0.158481
    4.831041    0.761419
    10.082890    1.373611
    10.066867    0.788470
    8.129538    0.329913
    3.012463    1.138108
    3.720391    0.845974
    0.773493    1.148256
    10.962941    1.037324
    0.177621    0.162614
    3.085853    0.967899
    8.426781    0.202558
    1.825927    1.128347
    2.185155    1.010173
    7.184595    1.261338
    0.000000    0.116525
    8.901752    1.033527
    2.451497    1.358795
    3.213631    0.432044
    3.974739    0.723929
    9.601306    0.619232
    8.363897    0.445341
    6.381484    1.365019
    0.000000    1.403914
    9.609836    1.438105
    9.904741    0.985862
    7.185807    1.489102
    5.466703    1.216571
    0.000000    0.915898
    4.575443    0.535671
    3.277076    1.010868
    10.246623    1.239634
    2.341735    1.060235
    3.201046    0.498843
    6.066013    0.120927
    8.829379    0.895657
    15.833048    1.568245
    13.516711    1.220153
    0.664284    1.116755
    6.325139    0.605109
    8.677499    0.344373
    8.188005    0.964896
    9.414263    0.384030
    9.196547    1.138253
    10.202968    0.452363
    2.119439    1.481661
    13.635078    0.858314
    0.083443    0.701669
    9.149096    1.051446
    1.933803    1.374388
    14.115544    0.676198
    8.933736    0.943352
    2.661254    0.946117
    0.988432    1.305027
    2.063741    1.125946
    2.220590    0.690754
    6.424849    0.806641
    1.156153    1.613674
    3.032720    0.601847
    3.076828    0.952089
    0.000000    0.318105
    7.750480    0.554015
    10.958135    1.482500
    10.222018    0.488678
    2.367988    0.435741
    7.686054    1.381455
    11.464879    1.481589
    11.075735    0.089726
    3.543989    0.345853
    8.123889    1.282880
    4.331769    0.754467
    0.120865    1.211961
    6.116109    0.701523
    7.474534    0.505790
    8.819454    0.649292
    6.802144    0.615284
    12.666325    0.931960
    8.636180    0.399333
    11.730991    1.289833
    8.132449    0.039062
    10.296589    1.496144
    7.583906    1.005764
    9.777806    0.496377
    8.833546    0.513876
    4.907899    1.518036
    8.362736    1.285939
    9.084726    1.606312
    14.164141    0.560970
    9.080683    0.989920
    6.522767    0.038548
    3.690342    0.462281
    3.563706    0.242019
    1.065870    1.141569
    6.683796    1.456317
    1.712874    0.243945
    13.109929    1.280111
    11.327910    0.780977
    4.545711    1.233254
    3.367889    0.468104
    8.326224    0.567347
    8.978339    1.442034
    5.655826    1.582159
    8.855312    0.570684
    6.649568    0.544233
    3.966325    0.850410
    1.924045    1.664782
    6.004812    0.280369
    0.000000    0.375849
    9.923018    0.092192
    2.389084    0.119284
    13.663189    0.133251
    11.434976    0.321216
    0.358270    1.292858
    9.598873    0.223524
    6.375275    0.608040
    11.580532    0.458401
    5.319324    1.598070
    4.324031    1.603481
    2.358370    1.273204
    0.000000    1.182708
    12.824376    0.890411
    1.587247    1.456982
    8.510324    1.520683
    10.428884    1.187734
    8.346618    0.042318
    7.541444    0.809226
    2.540946    1.583286
    9.473047    0.692513
    0.352284    0.474080
    0.000000    0.589826
    12.405171    0.567201
    4.126775    0.871452
    0.034087    0.335848
    1.177634    0.075106
    0.000000    0.479996
    0.994909    0.611135
    11.053664    1.180117
    0.000000    1.679729
    2.495011    1.459589
    11.516831    0.001156
    9.213215    0.797743
    5.332865    0.109288
    0.000000    1.689771
    0.000000    1.126053
    12.640062    1.690903
    2.693142    1.317518
    3.328969    0.268271
    7.193166    1.117456
    6.615512    1.521012
    8.000567    0.835341
    4.017541    0.512104
    13.245859    0.927465
    5.970616    0.813624
    11.668719    0.886902
    4.283237    1.272728
    10.742963    0.971401
    12.326672    1.592608
    0.000000    0.344622
    0.000000    0.922846
    10.602095    0.573686
    10.861859    1.155054
    1.229094    1.638690
    0.410392    1.313401
    14.552711    0.616162
    14.178043    0.616313
    14.136260    0.362388
    0.093534    1.207194
    10.929021    0.403110
    11.432919    0.825959
    9.134527    0.586846
    5.071432    1.421420
    11.460254    1.541749
    11.620039    1.103553
    4.022079    0.207307
    3.057842    1.631262
    7.782169    0.404385
    7.981741    0.929789
    4.601363    0.268326
    2.595564    1.115375
    10.049077    0.391045
    3.265444    1.572970
    11.780282    1.511014
    3.075975    0.286284
    1.795307    0.194343
    11.106979    0.202415
    5.994413    0.800021
    9.706062    1.012182
    10.582992    0.836025
    7.038266    1.458979
    0.023771    0.015314
    12.823982    0.676371
    3.617770    0.493483
    8.346684    0.253317
    6.104317    0.099207
    16.207776    0.584973
    6.401969    1.691873
    2.298696    0.559757
    7.661515    0.055981
    6.353608    1.645301
    10.442780    0.335870
    3.834509    1.346121
    10.998587    0.584555
    2.695935    1.512111
    3.356646    0.324230
    14.677836    0.793183
    1.551934    0.130902
    2.464739    0.223502
    1.533216    1.007481
    12.473921    0.162910
    6.491596    0.032576
    10.506276    1.510747
    4.380388    0.748506
    13.670988    1.687944
    8.317599    0.390409
    0.000000    0.556245
    0.000000    0.290218
    10.095799    1.188148
    0.860695    1.482632
    1.557564    0.711278
    10.072779    0.756030
    0.000000    0.431468
    7.140817    0.883813
    11.384548    1.438307
    3.214568    1.083536
    11.720655    0.301636
    6.374475    1.475925
    5.749684    0.198875
    3.871808    0.552602
    8.336309    0.636238
    9.710442    1.503735
    1.532611    1.433898
    9.785785    0.984614
    2.633627    1.097866
    9.238935    0.494701
    1.205656    1.398803
    3.124909    1.670121
    7.935489    1.585044
    12.746636    1.560352
    10.732563    0.545321
    3.977403    0.766103
    4.194426    0.450663
    9.610286    0.142912
    4.797555    1.260455
    1.615279    0.093002
    4.614771    1.027105
    0.000000    1.369726
    0.608457    0.512220
    6.558239    0.667579
    12.315116    0.197068
    7.014973    1.494616
    8.822304    1.194177
    10.086796    0.570455
    7.241614    1.661627
    4.602395    1.511768
    7.434921    0.079792
    10.467570    1.595418
    9.948127    0.003663
    2.478529    1.568987
    5.938545    0.878540
    0.000000    0.948004
    5.559181    1.357926
    9.776654    0.535966
    3.092056    0.490906
    0.000000    1.623311
    4.459495    0.538867
    8.334306    1.646600
    11.226654    0.384686
    3.904737    1.597294
    7.038205    1.211329
    9.836120    1.054340
    1.990976    0.378081
    9.005302    0.485385
    1.772510    1.039873
    0.458674    0.819560
    10.003919    0.231658
    0.520807    1.476008
    10.678214    1.431837
    4.425992    1.363842
    12.035355    0.831222
    10.606732    1.253858
    1.568653    0.684264
    2.545434    0.024271
    10.264062    0.982593
    9.866276    0.685218
    0.142704    0.057455
    9.853270    1.521432
    6.596604    1.653574
    2.602287    1.321481
    10.411776    0.664168
    7.083449    0.622589
    2.080068    1.254441
    0.522844    1.622458
    10.362000    1.544827
    3.412967    1.035410
    6.796548    1.112153
    4.092035    0.075804
    2.763811    1.564325
    12.547439    1.402443
    5.708052    1.596152
    4.558025    0.375806
    11.642307    0.438553
    3.222443    0.121399
    4.736156    0.029871
    10.839526    0.836323
    4.194791    0.235483
    14.936259    0.888582
    3.310699    1.521855
    2.971931    0.034321
    9.261667    0.537807
    7.791833    1.111416
    1.480470    1.028750
    3.677287    0.244167
    2.202967    1.370399
    5.796735    0.935893
    3.063333    0.144089
    11.233094    0.492487
    1.965570    0.005697
    8.616719    0.137419
    6.609989    1.083505
    1.712639    1.086297
    10.117445    1.299319
    0.000000    1.104178
    9.824777    1.346821
    1.653089    0.980949
    18.178822    1.473671
    6.781126    0.885340
    8.206750    1.549223
    10.081853    1.376745
    6.288742    0.112799
    3.695937    1.543589
    6.726151    1.069380
    12.969999    1.568223
    2.661390    1.531933
    7.072764    1.117386
    9.123366    1.318988
    3.743946    1.039546
    2.341300    0.219361
    0.541913    0.592348
    2.310828    1.436753
    6.226597    1.427316
    7.277876    0.489252
    0.000000    0.389459
    7.218221    1.098828
    8.777129    1.111464
    2.813428    0.819419
    2.268766    1.412130
    6.283627    0.571292
    7.520081    1.626868
    11.739225    0.027138
    3.746883    0.877350
    12.089835    0.521631
    12.310404    0.259339
    0.000000    0.671355
    2.728800    0.331502
    10.814342    0.607652
    12.170268    0.844205
    6.698371    0.240084
    3.632672    1.643479
    10.059991    0.892361
    1.887674    0.756162
    8.229125    0.195886
    7.817082    0.476102
    12.277230    0.076805
    10.055337    1.115778
    3.596002    1.485952
    2.755530    1.420655
    7.780991    0.513048
    0.093705    0.391834
    8.481567    0.520078
    3.865584    0.110062
    9.683709    0.779984
    10.617255    1.359970
    7.203216    1.624762
    7.601414    1.215605
    1.386107    1.417070
    9.129253    0.594089
    1.363447    0.620841
    3.181399    0.359329
    13.365414    0.217011
    4.207717    1.289767
    4.088395    0.870075
    3.327371    1.142505
    1.303323    1.235650
    7.999279    1.581763
    2.217488    0.864536
    7.751808    0.192451
    14.149305    1.591532
    8.765721    0.152808
    3.408996    0.184896
    1.251021    0.112340
    6.160619    1.537165
    1.034538    1.585162
    0.000000    1.034635
    2.355051    0.542603
    6.614543    0.153771
    10.245062    1.450903
    3.467074    1.231019
    7.487678    1.572293
    4.624115    1.185192
    8.995957    1.436479
    11.564476    0.007195
    3.440948    0.078331
    1.673603    0.732746
    4.719341    0.699755
    10.304798    1.576488
    2.086915    1.199312
    6.338220    1.131305
    8.254926    0.710694
    16.067108    0.974142
    1.723201    0.310488
    3.785045    0.876904
    2.557561    0.123738
    9.852220    1.095171
    3.679147    1.557205
    9.789681    0.852971
    14.958998    0.526707
    11.182148    1.288459
    7.528533    1.657487
    5.253802    1.378603
    13.946752    1.426657
    15.557263    1.430029
    12.483550    0.688513
    2.317302    1.411137
    10.069724    0.766119
    5.792231    1.615483
    4.138435    0.475994
    12.929517    0.304378
    9.378238    0.307392
    8.361362    1.643204
    7.939406    1.325042
    10.735384    0.705788
    11.592723    0.286188
    10.098356    0.704748
    9.299025    0.545337
    11.158297    0.218067
    16.143900    0.558388
    10.971700    1.221787
    0.000000    0.681478
    3.178961    1.292692
    17.625350    0.339926
    1.995833    0.267826
    10.640467    0.416181
    9.628339    0.985462
    4.662664    0.495403
    5.754047    1.382742
    0.000000    0.037146
    9.334332    0.198118
    3.846162    0.619968
    10.685084    0.678179
    4.752134    0.359205
    0.697630    0.966786
    10.365836    0.505898
    0.461478    0.352865
    11.339537    1.068740
    5.420280    0.127310
    3.469955    1.619947
    8.517067    0.994858
    8.306512    0.413690
    2.628690    0.444320
    0.000000    0.802985
    0.000000    1.170397
    7.298767    1.582346
    7.331319    1.277988
    9.392269    0.151617
    5.541201    1.180596
    15.149460    0.537540
    5.515189    0.250562
    7.728898    0.920494
    11.318785    1.510979
    3.574709    1.531514
    7.350965    0.026332
    7.122363    1.630177
    1.828412    1.013702
    10.117989    1.156862
    11.309897    0.086291
    8.342034    1.388569
    0.241714    0.715577
    10.482619    1.694972
    9.289510    1.428879
    4.269419    0.134181
    0.000000    0.189456
    0.817119    0.143668
    1.508394    0.652651
    9.359918    0.052262
    10.052333    0.550423
    11.111660    0.989159
    11.265971    0.724054
    10.383830    0.254836
    3.878569    1.377983
    13.679237    0.025346
    10.526846    0.781569
    0.000000    0.924198
    4.106727    1.085669
    8.118856    1.470686
    7.796874    0.052336
    2.789669    1.093070
    6.226962    0.287251
    10.169548    1.660104
    0.000000    1.370549
    7.513353    0.137348
    8.240793    0.099735
    14.612797    1.247390
    3.562976    0.445386
    3.230482    1.331698
    3.612548    1.551911
    0.000000    0.332365
    3.931299    0.487577
    14.752342    1.155160
    10.261887    1.628085
    2.787266    1.570402
    15.112319    1.324132
    5.184553    0.223382
    3.868359    0.128078
    3.507965    0.028904
    11.019254    0.427554
    3.812387    0.655245
    11.056784    0.378725
    8.826880    1.002328
    11.173861    1.478244
    11.506465    0.421993
    7.798138    0.147917
    10.155081    1.370039
    10.645275    0.693453
    9.663200    1.521541
    10.790404    1.312679
    2.810534    0.219962
    9.825999    1.388500
    1.421316    0.677603
    11.123219    0.809107
    13.402206    0.661524
    1.212255    0.836807
    1.568446    1.297469
    3.343473    1.312266
    5.400155    0.193494
    3.818754    0.590905
    7.973845    0.307364
    9.078824    0.734876
    0.153467    0.766619
    8.325167    0.028479
    7.092089    1.216733
    5.192485    1.094409
    10.340791    1.087721
    2.077169    1.019775
    10.151966    0.993105
    0.046826    0.809614
    11.221874    1.395015
    14.497963    1.019254
    3.554508    0.533462
    3.522673    0.086725
    14.531655    0.380172
    3.027528    0.885457
    1.845967    0.488985
    10.226164    0.804403
    10.965926    1.212328
    2.129921    1.477378
    0.000000    1.606849
    9.489005    0.827814
    0.000000    1.020797
    0.000000    1.270167
    6.556676    0.055183
    9.959588    0.060020
    7.436056    1.479856
    0.404888    0.459517
    9.952942    1.650279
    15.600252    0.021935
    2.723846    0.387455
    0.513866    1.323448
    0.000000    0.861859
    7.280602    1.438470
    9.161978    1.110180
    0.991725    0.730979
    7.398380    0.684218
    12.149747    1.389088
    9.149678    0.874905
    9.666576    1.370330
    3.620110    0.287767
    5.238800    1.253646
    14.715782    1.503758
    14.445740    1.211160
    13.609528    0.364240
    3.141585    0.424280
    0.000000    0.120947
    0.454750    1.033280
    0.510310    0.016395
    3.864171    0.616349
    6.724021    0.563044
    4.289375    0.012563
    0.000000    1.437030
    3.733617    0.698269
    2.002589    1.380184
    2.502627    0.184223
    6.382129    0.876581
    8.546741    0.128706
    2.694977    0.432818
    3.951256    0.333300
    9.856183    0.329181
    2.068962    0.429927
    3.410627    0.631838
    9.974715    0.669787
    10.650102    0.866627
    9.134528    0.728045
    7.882601    1.332446

data.txt

代码:

import numpy as np

import operator

import matplotlib.pyplot as plt

def knn_classifier(inX, train_data, labels, k):

    """

    Knn_Classifier k近邻核心实现代码

    :param inX: 输入待测试实例

    :param train_data: 输入的训练数据集

    :param labels: 训练数据集对应的标签

    :param k: 分为k类

    :return: 返回对应的类别

    """

    train_data_size = train_data.shape[0]  # 获取数据集的

    diff_matrix = np.tile(inX, (train_data_size, 1)) - train_data  # 计算inx与训练数据集中所有数据的差值

    lp_diff_matrix = diff_matrix**2  # 这里采用的是欧氏距离

    lp_distances = lp_diff_matrix.sum(axis=1)

    distances = lp_distances**0.5

    sorted_dist_indicies = np.argsort(distances)  # 按照欧氏距离的大小进行排序

    class_count = {}

    for i in range(k):

        vote_I_label = labels[sorted_dist_indicies[i]]  # 依次存入欧氏距离前k个最小距离的编号

        class_count[vote_I_label] = class_count.get(vote_I_label, 0) + 1  # 对应一个字典

    sorted_class_count = sorted(class_count.items(), key=operator.itemgetter(1), reverse=True)  # 对字典进行排序

    return sorted_class_count[0][0]  # 最高值对应类别即为inX的类别

def file_to_matrix(filename):

    '''

    文件数据转换为矩阵类型

    :param filename:

    :return:

    '''

    f = open(filename)

    lines = f.readlines()

    lines_number = len(lines)

    feature = len(lines[1].split('\t'))

    matrix = np.zeros((lines_number, feature - 1))

    class_label_vector = []

    index = 0

    for line in lines:

        line = line.strip()

        item = line.split('\t')

        matrix[index, :] = item[0:3]

        class_label_vector.append(item[-1])

        index += 1

    return matrix, class_label_vector

def auto_norm(data_set):

    """

    auto_norm 归一化函数，消除量纲不同，不同属性的大小差异过大的影响

    归一化公式 newValue = (oldValue - min)/(max - min)  取值0-1

    :param dataSet:

    :return:

    """

    min_vals = data_set.min(0)

    max_vals = data_set.max(0)

    ranges = max_vals - min_vals

    # print(type(min_vals))

    m = data_set.shape[0]

    data_set = data_set - np.tile(min_vals, (m, 1))

    data_set = data_set/(np.tile(ranges, (m, 1)))

    return data_set, ranges, min_vals

def dating_class_test():

    """

    dating_class_test 测试约会数据集data2的聚类预测结果

    return: null

    """

    ho_ratio = 0.1

    dating_matrix, dating_labels = file_to_matrix('data2.txt')  # 读入数据

    norm_matrix, ranges, min_vals = auto_norm(dating_matrix)  # 归一化数据

    m = norm_matrix.shape[0]  # 获取行数

    num_test_vecs = int(m * ho_ratio)  # 取出其中的10%作为测试数据

    error_count = 0.0  # 记录错误率

    for i in range(num_test_vecs):

        classifier = knn_classifier(norm_matrix[i, :], norm_matrix[num_test_vecs:m, :], dating_labels[num_test_vecs:m], 4)  #利用KNN聚类算法进行预测

        print("the classifier came back with: " +classifier + " the real answer is: " + dating_labels[i])

        if(classifier != dating_labels[i]) :

            error_count += 1.0

    print('the total error rate is: %f' % (error_count/float(num_test_vecs)))

if __name__ == "__main__":

    dating_class_test()