Seven Techniques for Data Dimensionality Reduction

Seven Techniques for Data Dimensionality Reduction

12 May, 2015 - 12:38 — rs

The recent explosion of data set size, in number of records and attributes, has triggered the development of a number of big data platforms as well as parallel data analytics algorithms. At the same time though, it has pushed for usage of data dimensionality reduction procedures. Indeed, more is not always better. Large amounts of data might sometimes produce worse performances in data analytics applications.

One of my most recent projects happened to be about churn prediction and to use the 2009 KDD Challenge large data set. The particularity of this data set consists of its very high dimensionality with 15K data columns. Most data mining algorithms are column-wise implemented, which makes them slower and slower on a growing number of data columns. The first milestone of the project was then to reduce the number of columns in the data set and lose the smallest amount of information possible at the same time.

Using the project as an excuse, we started exploring the state-of-the-art on dimensionality reduction techniques currently available and accepted in the data analytics landscape.

  • Missing Values Ratio. Data columns with too many missing values are unlikely to carry much useful information. Thus data columns with number of missing values greater than a given threshold can be removed. The higher the threshold, the more aggressive the reduction.
  • Low Variance Filter. Similarly to the previous technique, data columns with little changes in the data carry little information. Thus all data columns with variance lower than a given threshold are removed. A word of caution: variance is range dependent; therefore normalization is required before applying this technique.
  • High Correlation Filter. Data columns with very similar trends are also likely to carry very similar information. In this case, only one of them will suffice to feed the machine learning model. Here we calculate the correlation coefficient between numerical columns and between nominal columns as the Pearson’s Product Moment Coefficient and thePearson's chi square value respectively. Pairs of columns with correlation coefficient higher than a threshold are reduced to only one. A word of caution: correlation is scale sensitive; therefore column normalization is required for a meaningful correlation comparison.
  • Random Forests / Ensemble Trees. Decision Tree Ensembles, also referred to as random forests, are useful for feature selection in addition to being effective classifiers.  One approach to dimensionality reduction is to generate a large and carefully constructed set of trees against a target attribute and then use each attribute’s usage statistics to find the most informative subset of features.  Specifically, we can generate a large set (2000) of very shallow trees (2 levels), with each tree being trained on a small fraction (3) of the total number of attributes. If an attribute is often selected as best split, it is most likely an informative feature to retain. A score calculated on the attribute usage statistics in the random forest tells us ‒ relative to the other attributes ‒ which are the most predictive attributes.
  • Principal Component Analysis (PCA)Principal Component Analysis (PCA) is a statistical procedure that orthogonally transforms the original n coordinates of a data set into a new set of n coordinates called principal components. As a result of the transformation, the first principal component has the largest possible variance; each succeeding component has the highest possible variance under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding components. Keeping only the first m < ncomponents reduces the data dimensionality while retaining most of the data information, i.e. the variation in the data. Notice that the PCA transformation is sensitive to the relative scaling of the original variables. Data column ranges need to be normalized before applying PCA. Also notice that the new coordinates (PCs) are not real system-produced variables anymore. Applying PCA to your data set loses its interpretability. If interpretability of the results is important for your analysis, PCA is not the transformation for your project.
  • Backward Feature Elimination. In this technique, at a given iteration, the selected classification algorithm is trained on n input features. Then we remove one input feature at a time and train the same model on n-1 input features n times. The input feature whose removal has produced the smallest increase in the error rate is removed, leaving us with n-1 input features. The classification is then repeated using n-2 features, and so on. Each iteration k produces a model trained on n-k features and an error rate e(k). Selecting the maximum tolerable error rate, we define the smallest number of features necessary to reach that classification performance with the selected machine learning algorithm.
  • Forward Feature Construction. This is the inverse process to the Backward Feature Elimination. We start with 1 feature only, progressively adding 1 feature at a time, i.e. the feature that produces the highest increase in performance. Both algorithms, Backward Feature Elimination and Forward Feature Construction, are quite time and computationally expensive. They are practically only applicable to a data set with an already relatively low number of input columns.

We picked this chance to compare those techniques on the smaller data set of the 2009 KDD challenge in terms of reduction ratio, degrading accuracy, and speed. The final accuracy and its degradation depend, of course, on the model selected for the analysis. Thus, the compromise between reduction ratio and final accuracy is optimized against a bag of three specific models: decision tree, neural networks, and naïve Bayes.

Running the optimization loop, the best cutoffs, in terms of lowest number of columns and best accuracy, were determined for each one of the seven dimensionality reduction techniques and for the best performing model. The final best model performance, as accuracy and Area under the ROC Curve, was compared with the performance of the baseline algorithm using all input features. Results of this comparison are reported in the table below.

Dimensionality Reduction Reduction Rate Accuracy on validation set Best Threshold AuC Notes
Baseline 0% 73% - 81% Baseline models are using all input features
Missing Values Ratio 71% 76% 0.4 82% -
Low Variance Filter 73% 82% 0.03 82% Only for numerical columns
High Correlation Filter 74% 79% 0.2 82% No correlation available between numerical and nominal columns
PCA 62% 74% - 72% Only for numerical columns
Random Forrest / Ensemble Trees 86% 76% - 82% -
Backward Feature Elimination + missing values ratio 99% 94% - 78% Backward Feature Elimination and Forward Feature Construction are prohibitively slow on high dimensional data sets. It becomes practical to use them, only if following other dimensionality reduction techniques, like here the one based on the number of missing values.
Forward Feature Construction + missing values ratio 91% 83% - 63%

Notice that the highest reduction ratio without performance degradation is obtained by analyzing the decision cuts in many random forests (Random Forests/Ensemble Trees). However, even just counting the number of missing values, measuring the column variance, and measuring the correlation of pairs of columns can lead to a satisfactory reduction rate while keeping performance unaltered with respect to the baseline models.

What we have learned from this little review exercise, is that dimensionality reduction is not only useful to speed up algorithm execution, but also to improve model performance. The Area under the Curve (AuC) in the table shows a slight increase on the test data, when the missing value ratio, the low variance filter, the high correlation filter criteria, or the random forests are applied.

Indeed, in the era of big data, when more is axiomatically better, we have re-discovered that too many noisy or even faulty input data columns often lead to a less than desirable algorithm performance. Removing un-informative or even worse dis-informative input attributes might help build a model on more extensive data regions, with more general classification rules, and overall with better performances on new unseen data.

Recently, we asked data analysts on a LinkedIn group (https://www.linkedin.com/grp/post/35222-5998794653007171586) for the most used dimensionality reduction techniques, besides the seven described in this blog post. The answers involved Random Projections, NMF, (Stacked) Auto-encoders, Chi-square or Information Gain, Multidimensional Scaling, Correspondence Analysis, Factor Analysis, Clustering, and Bayesian Models. Thanks to Asterios StergioudisRaoul Savos, and Michael Will who provided the suggestions on the LinkedIn group.

The workflows described in this blog post are available on the KNIME EXAMPLES server under 003_Preprocessing/003005_dimensionality_reduction.

Both small and large data sets from the 2009 KDD Challenge can be downloaded from http://www.sigkdd.org/kdd-cup-2009-customer-relationship-prediction.

This is just a brief summary of the whole project. If you are interested in all the tiny details, you can always read the related whitepaper, in the Whitepapers section on the KNIME web site:https://www.knime.org/files/knime_seventechniquesdatadimreduction.pdf

Below are the ROC curves for all the evaluated dimensionality reduction techniques and the best performing machine learning algorithm. The value of the area under the curve is shown in the legend.

Further Reading:

Seven Techniques for Data Dimensionality Reduction的更多相关文章

  1. dimensionality reduction动机---data compression(使算法提速)

    data compression可以使数据占用更少的空间,并且能使算法提速 什么是dimensionality reduction(维数约简)    例1:比如说我们有一些数据,它有很多很多的feat ...

  2. 可视化MNIST之降维探索Visualizing MNIST: An Exploration of Dimensionality Reduction

    At some fundamental level, no one understands machine learning. It isn’t a matter of things being to ...

  3. Stanford机器学习笔记-10. 降维(Dimensionality Reduction)

    10. Dimensionality Reduction Content  10. Dimensionality Reduction 10.1 Motivation 10.1.1 Motivation ...

  4. [Scikit-learn] 4.4 Dimensionality reduction - PCA

    2.5. Decomposing signals in components (matrix factorization problems) 2.5.1. Principal component an ...

  5. 第八章——降维(Dimensionality Reduction)

    机器学习问题可能包含成百上千的特征.特征数量过多,不仅使得训练很耗时,而且难以找到解决方案.这一问题被称为维数灾难(curse of dimensionality).为简化问题,加速训练,就需要降维了 ...

  6. 壁虎书8 Dimensionality Reduction

    many Machine Learning problems involve thousands or even millions of features for each training inst ...

  7. [UFLDL] Dimensionality Reduction

    博客内容取材于:http://www.cnblogs.com/tornadomeet/archive/2012/06/24/2560261.html Deep learning:三十五(用NN实现数据 ...

  8. 单细胞数据高级分析之初步降维和聚类 | Dimensionality reduction | Clustering

    个人的一些碎碎念: 聚类,直觉就能想到kmeans聚类,另外还有一个hierarchical clustering,但是单细胞里面都用得不多,为什么?印象中只有一个scoring model是用kme ...

  9. 【原】Coursera—Andrew Ng机器学习—课程笔记 Lecture 14—Dimensionality Reduction 降维

    Lecture 14 Dimensionality Reduction 降维 14.1 降维的动机一:数据压缩 Data Compression 现在讨论第二种无监督学习问题:降维. 降维的一个作用是 ...

随机推荐

  1. 《软件工程和Python》PYTHON效能分析和Django

    资料汇总网站:http://www.yzhiliao.com/my/course/55 一..作业下面两个题目任选一题: (1)运用jieba库分词(或者你喜欢的其他库),并把代码发到git上去(不发 ...

  2. 转 理解vuex -- vue的状态管理模式

    转自:https://segmentfault.com/a/1190000012015742 vuex是什么? 先引用vuex官网的话: Vuex 是一个专为 Vue.js 应用程序开发的状态管理模式 ...

  3. js数组遍历 千万不要使用for...in...

    昨天做个下拉框 扩充了一下数组的方法 Array.prototype.remove = function (val) { var index = this.indexOf(val); if (inde ...

  4. vue 过渡效果

    Vue中提供了`<transition>`和`<transition-group>`来为元素增加过渡动画.文档写的很清楚,但是实际使用起来还是费了一番功夫.这里做一个简单的记录 ...

  5. HBase 架构与工作原理4 - 压缩、分裂与故障恢复

    本文系转载,如有侵权,请联系我:likui0913@gmail.com Compacation HBase 在读写的过程中,难免会产生无效的数据以及过小的文件,比如:MemStore 在未达到指定大小 ...

  6. 检查cpu是否支持VT-X(HAXM)

    可以使用CPU-Z进行检测,CPU是否支持VT-X,如果不支持,就不要浪费时间安装HAXM了. 可以试一下Genymotion模拟器.

  7. ACM数论之旅9---中国剩余定理(CRT)(壮哉我大中华╰(*°▽°*)╯)

    中国剩余定理,又名孙子定理o(*≧▽≦)ツ 能求解什么问题呢? 问题: 一堆物品 3个3个分剩2个 5个5个分剩3个 7个7个分剩2个 问这个物品有多少个 解这题,我们需要构造一个答案 我们需要构造这 ...

  8. SecureCRT8.1下载+注册机+破解教程

    [下载]下载SecureCRT + SecureFX 8.1 Bundle版本软件,官网下载较麻烦,因此在此提供百度云连接. 链接:http://pan.baidu.com/s/1hsIjtSK 密码 ...

  9. 转:关于python文件操作大全

    转自:http://www.cnblogs.com/rollenholt/archive/2012/04/23/2466179.html python中对文件.文件夹(文件操作函数)的操作需要涉及到o ...

  10. [AT2384] [agc015_f] Kenus the Ancient Greek

    题目链接 AtCoder:https://agc015.contest.atcoder.jp/tasks/agc015_f 洛谷:https://www.luogu.org/problemnew/sh ...