[Machine Learning with Python] Data Preparation by Pandas and Scikit-Learn
In this article, we dicuss some main steps in data preparation.
Drop Labels
Firstly, we drop labels for train set. Here we use drop() method in Pandas library.
housing = strat_train_set.drop("median_house_value", axis=1) # drop labels for training set
housing_labels = strat_train_set["median_house_value"].copy()
Here are some tips:
- The drop funtion deletes rows by default. If you want to delete columns, don't forget to set the parameter axis=1.
- The
drop
function doesn't change the DataFrame by default. And instead, returns to you a copy of the DataFrame with the given rows/columns removed. Or you can set inplace = True. - Note the function copy() here. It creates a copy that will not affect the original DataFrame
Impute Missing Values
Firstly, let's check the missing values:
sample_incomplete_rows = housing[housing.isnull().any(axis=1)].head()
Here give three methods to impute missing values:
Option 1: drop the rows
sample_incomplete_rows.dropna(subset=["total_bedrooms"])
Option 2: drop the columns
sample_incomplete_rows.drop("total_bedrooms", axis=1)
Option 3: impute with the median value
median = housing["total_bedrooms"].median()
sample_incomplete_rows["total_bedrooms"].fillna(median, inplace=True)
Alternatively, we can import sklearn.impute.SimpleImputer
class in Scikit-Learn 0.20.
try:
from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+
except ImportError:
from sklearn.preprocessing import Imputer as SimpleImputer imputer = SimpleImputer(strategy="median")
# Remove the text attribute because median can only be calculated on numerical attributes
housing_num = housing.drop('ocean_proximity', axis=1)
# alternatively: housing_num = housing.select_dtypes(include=[np.number])
imputer.fit(housing_num)
We can check the statistcs by imputer.statistics_ and the strategy by imputer.strategy
Finally, transform the train set:
X = imputer.transform(housing_num)
housing_tr = pd.DataFrame(X, columns=housing_num.columns,
index = list(housing.index.values))
Encode Categorical Attributes
We need to convert text labels to numbers. There are two methods.
Option 1: Label Encoding
Conver a categorical attribute into an interger attribute.
try:
from sklearn.preprocessing import OrdinalEncoder
except ImportError:
from future_encoders import OrdinalEncoder # Scikit-Learn < 0.20 ordinal_encoder = OrdinalEncoder()
housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)
Option2: One-Hot Encoding
Convert a categorical attribute into a series of binary intergers.
try:
from sklearn.preprocessing import OrdinalEncoder # just to raise an ImportError if Scikit-Learn < 0.20
from sklearn.preprocessing import OneHotEncoder
except ImportError:
from future_encoders import OneHotEncoder # Scikit-Learn < 0.20 cat_encoder = OneHotEncoder()
housing_cat_1hot = cat_encoder.fit_transform(housing_cat)
By default, the OneHotEncoder
class returns a sparse array, but we can convert it to a dense array if needed by calling the toarray()
method:
housing_cat_1hot.toarray()
Alternatively, you can set sparse=False
when creating the OneHotEncoder
:
cat_encoder = OneHotEncoder(sparse=False)
housing_cat_1hot = cat_encoder.fit_transform(housing_cat)
Feature Engineering
Sometimes, we need to add some features to better describe the variation of the target variable. Let's create a custom transformer to add extra attributes and implement three methods: fit()(returning self), transform(), and fit_transform(). You can get the last one for free by simply adding TransformerMixin as a base class. Also, if you add BaseEstima tor as a base class (and avoid *args and **kargs in your constructor) you will get two extra methods (get_params() and set_params()) that will be useful for auto‐ matic hyperparameter tuning.
from sklearn.base import BaseEstimator, TransformerMixin # column index
rooms_ix, bedrooms_ix, population_ix, household_ix = 3, 4, 5, 6 class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs
self.add_bedrooms_per_room = add_bedrooms_per_room
def fit(self, X, y=None):
return self # nothing else to do
def transform(self, X, y=None):
rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
population_per_household = X[:, population_ix] / X[:, household_ix]
if self.add_bedrooms_per_room:
bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
return np.c_[X, rooms_per_household, population_per_household,
bedrooms_per_room]
else:
return np.c_[X, rooms_per_household, population_per_household] attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)
[Machine Learning with Python] Data Preparation by Pandas and Scikit-Learn的更多相关文章
- [Machine Learning with Python] Data Preparation through Transformation Pipeline
In the former article "Data Preparation by Pandas and Scikit-Learn", we discussed about a ...
- [Machine Learning with Python] Data Visualization by Matplotlib Library
Before you can plot anything, you need to specify which backend Matplotlib should use. The simplest ...
- Python (1) - 7 Steps to Mastering Machine Learning With Python
Step 1: Basic Python Skills install Anacondaincluding numpy, scikit-learn, and matplotlib Step 2: Fo ...
- Getting started with machine learning in Python
Getting started with machine learning in Python Machine learning is a field that uses algorithms to ...
- 《Learning scikit-learn Machine Learning in Python》chapter1
前言 由于实验原因,准备入坑 python 机器学习,而 python 机器学习常用的包就是 scikit-learn ,准备先了解一下这个工具.在这里搜了有 scikit-learn 关键字的书,找 ...
- 【Machine Learning】Python开发工具:Anaconda+Sublime
Python开发工具:Anaconda+Sublime 作者:白宁超 2016年12月23日21:24:51 摘要:随着机器学习和深度学习的热潮,各种图书层出不穷.然而多数是基础理论知识介绍,缺乏实现 ...
- In machine learning, is more data always better than better algorithms?
In machine learning, is more data always better than better algorithms? No. There are times when mor ...
- Coursera, Big Data 4, Machine Learning With Big Data (week 1/2)
Week 1 Machine Learning with Big Data KNime - GUI based Spark MLlib - inside Spark CRISP-DM Week 2, ...
- Machine Learning的Python环境设置
Machine Learning目前经常使用的语言有Python.R和MATLAB.如果采用Python,需要安装大量的数学相关和Machine Learning的包.一般安装Anaconda,可以把 ...
随机推荐
- 网络流 EK算法模板。
这篇博客讲得很好 #include<queue> #include<stdio.h> #include<string.h> using namespace std; ...
- HDU 3896 Greatest TC 双连通分量
题意 给一个连通的无向图,有两种询问: \(a, b, c, d\),问如果删掉\(c,d\)之间的边,\(a,b\)之间是否还连通 \(a, b, c\),问如果删掉顶点\(c\),\(a,b\)之 ...
- OpenResty安装与hello world
安装环境:CentOS 7.0 1. 安装编译工具.依赖库 yum -y install readline-devel pcre-devel openssl-devel gcc 2. 下载openre ...
- Java中BigInteger类型
BigInteger是java.math包提供的处理大整数类型,实现了大整数的存储,四则运算,判断素数的方法,求幂,求模,求逆元,求最大公约数等方法.本文主要分析下BigInteger对于大整数的存储 ...
- java web知识点
java web知识点 1.Java知识点 基本数据类型,面向对象,异常,IO,NIO,集合,多线程,JVM,高级特性. 2.web知识点 JSP,Serlvet,JDBC,Http 掌握Cookie ...
- mini购物车程序
product_list=[("Iphohe",5800),("Mac Pro Book",12900), ("xiaomi 4c",120 ...
- PostgreSQL查看索引的使用情况
查看某个表的索引使用情况 select relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch from pg_stat_user_i ...
- camelot工具进行pdf表格解析重建
camelot内置生成html文件的方法,但表格数据转化成pandas.dataframe的过程中,丢失了跨行跨列的结构信息,故生成html的表格无跨行跨列结构. 于是我在输出部分选择直接手写html ...
- 【转】netstat 查看端口占用情况
netstat用来查看系统当前系统网络状态信息,包括端口,连接情况等,常用方式如下: netstat -atunlp,各参数含义如下: -t : 指明显示TCP端口 -u : 指明显示UDP端口 -l ...
- 字符串匹配算法(在字符串T中查找是否有与字符串P相同的子串)
T称为目标串(Target)或主串 ,P称为模式串(Pattren) 或子串 1. 简单字符串模式匹配算法 原理:用字符串P的字符依次与字符串T中的字符进行比较,首先将字符串P从第0个位置起与主串T的 ...