Text classifcation with Naïve Bayes

In this section we will try to classify newsgroup messages using a dataset that can be retrieved from within scikit-learn. This dataset consists of around 19,000 newsgroup messages from 20 different topics ranging from politics and religion to sports and science.

As usual, we frst start by importing our pylab environment:

%pylab inline

Our dataset can be obtained by importing the fetch_20newgroups function from the sklearn.datasets module. We have to specify if we want to import a part or all of the set of instances (we will import all of them).

from sklearn.datasets import fetch_20newsgroups
news = fetch_20newsgroups(subset='all')

If we look at the properties of the dataset, we will fnd that we have the usual ones: DESCR, data, target, and target_names. The difference now is that data holds a list of text contents, instead of a numpy matrix:

print(type(news.data), type(news.target), type(news.target_names))
print(news.target_names)
print(len(news.data))
print(len(news.target))
<class 'list'> <class 'numpy.ndarray'> <class 'list'>
['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc']
18846
18846

If you look at, say, the frst instance, you will see the content of a newsgroup message, and you can get its corresponding category:

print(news.data[0])
print(news.target[0], news.target_names[news.target[0]])
From: Mamatha Devineni Ratnam <mr47+@andrew.cmu.edu>
Subject: Pens fans reactions
Organization: Post Office, Carnegie Mellon, Pittsburgh, PA
Lines: 12
NNTP-Posting-Host: po4.andrew.cmu.edu I am sure some bashers of Pens fans are pretty confused about the lack
of any kind of posts about the recent Pens massacre of the Devils. Actually,
I am bit puzzled too and a bit relieved. However, I am going to put an end
to non-PIttsburghers' relief with a bit of praise for the Pens. Man, they
are killing those Devils worse than I thought. Jagr just showed you why
he is much better than his regular season stats. He is also a lot
fo fun to watch in the playoffs. Bowman should let JAgr have a lot of
fun in the next couple of games since the Pens are going to beat the pulp out of Jersey anyway. I was very disappointed not to see the Islanders lose the final
regular season game. PENS RULE!!! 10 rec.sport.hockey

Preprocessing the data

Our machine learning algorithms can work only on numeric data, so our next step will be to convert our text-based dataset to a numeric dataset. Currently we only have one feature, the text content of the message; we need some function that transforms a text into a meaningful set of numeric features.

Intuitively one could try to look at which are the words (or more precisely, tokens, including numbers or punctuation signs) that are used in each of the text categories, and try to characterize

each category with the frequency distribution of each of those words. The sklearn.

feature_extraction.text
module has some useful utilities to build numeric feature vectors from text documents.

Before starting the transformation, we will have to partition our data into training and testing set. The loaded data is already in a random order, so we only have to split the data into, for example, 75 percent for training and the rest 25 percent for testing:

If you look inside the sklearn.feature_extraction.text module, you will fnd three different classes that can transform text into numeric features: CountVectorizer, HashingVectorizer, and TfidfVectorizer.

The difference between them resides in the calculations they perform to obtain the numeric features.

  • CountVectorizer主要从语料库创建了一个单词的字典,将每一个样本转化成一个关于每个单词在文档中出现次数的向量。
  • HashingVectorizer不是在内存中压缩和维护字典,而是实现了一个将标记映射到特征索引的哈希函数,然后同一样计数。
  • TfidfVectorizer与CountVectorizer类似,不过它使用一种更高级的计算方式,叫做Term Frequency Inverse Document Frequency (TF-IDF)。这是一种统计来测量一个单词在文档或语料库中的重要性。直观地,它寻找在当前文档中与整个文档集相比更加频繁的单词。你可以把它看做一种方式,用来标准化结果和避免单词太频繁,因此不能用来描述样本。

Training a Naïve Bayes classifer

We will create a Naïve Bayes classifer that is composed of a feature vectorizer and the actual Bayes classifer. We will use the MultinomialNB class from the sklearn.naive_bayes module. Scikitlearn has a very useful class called Pipeline (available in the sklearn.pipeline module) that eases the construction of a compound classifer, which consists of several vectorizers and classifers.

We will create three different classifers by combining MultinomialNB with the three different text vectorizers just mentioned, and compare which one performs better using the default parameters:

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer, CountVectorizer
clf_1 = Pipeline([
('vect', CountVectorizer()),
('clf', MultinomialNB()),
])
clf_2 = Pipeline([
('vect', HashingVectorizer(non_negative=True)),
('clf', MultinomialNB()),
])
clf_3 = Pipeline([
('vect', TfidfVectorizer()),
('clf', MultinomialNB()),
])

We will defne a function that takes a classifer and performs the K-fold crossvalidation over the specifed X and y values:

from sklearn.cross_validation import cross_val_score, KFold
from scipy.stats import sem
def evaluate_cross_validation(clf, X, y, K):
# create a k-fold cross validation iterator of k=5 folds
cv = KFold(len(y), K, shuffle=True, random_state=0)
# by default the score used is the one returned by score method of the estimator (accuracy)
scores = cross_val_score(clf, X, y, cv=cv)
print(scores)
print(("Mean score: {0:.3f} (+/-{1:.3f})").format(np.mean(scores), sem(scores)))

Then we will perform a fve-fold cross-validation by using each one of the classifers.

clfs = [clf_1, clf_2, clf_3]
for clf in clfs:
evaluate_cross_validation(clf, news.data, news.target, 5)
[ 0.85782493  0.85725657  0.84664367  0.85911382  0.8458477 ]
Mean score: 0.853 (+/-0.003)
[ 0.75543767 0.77659857 0.77049615 0.78508888 0.76200584]
Mean score: 0.770 (+/-0.005)
[ 0.84482759 0.85990979 0.84558238 0.85990979 0.84213319]
Mean score: 0.850 (+/-0.004)

As you can see CountVectorizer and TfidfVectorizer had similar performances, and much better than HashingVectorizer.

Let's continue with TfidfVectorizer; we could try to improve the results by trying to parse the text documents into tokens with a different regular expression.

clf_4 = Pipeline([
('vect', TfidfVectorizer(
token_pattern=ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('clf', MultinomialNB()),
])
(不知道为什么报错SyntaxError: invalid syntax)

The default regular expression: ur"\b\w\w+\b" considers alphanumeric characters and the underscore. Perhaps also considering the slash and the dot could improve the tokenization, and begin considering tokens as Wi-Fi and site.com. The new regular expression could be: ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b". If you have queries about how to defne regular expressions, please refer to the Python re module documentation. Let's try our new classifer:

evaluate_cross_validation(clf_4, news.data, news.target, 5)

We have a slight improvement from 0.84 to 0.85.

Another parameter that we can use is stop_words: this argument allows us to pass a list of words we do not want to take into account, such as too frequent words, or words we do not a priori expect to provide information about the particular topic.

We will defne a function to load the stop words from a text fle as follows:

def get_stop_words():
result = set()
for line in open('stopwords_en.txt', 'r').readlines():
result.add(line.strip())
return result

And create a new classifer with this new parameter as follows:

clf_5 = Pipeline([
('vect', TfidfVectorizer(
stop_words=get_stop_words(),
token_pattern=ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('clf', MultinomialNB()),
])
evaluate_cross_validation(clf_5, news.data, news.target, 5)

The preceding code shows another improvement from 0.85 to 0.87.

Let's keep this vectorizer and start looking at the MultinomialNB parameters. This classifer has few parameters to tweak; the most important is the alpha parameter, which is a smoothing parameter. Let's set it to a lower value; instead of setting alpha to 1.0 (the default value), we will set it to 0.01:

clf_6 = Pipeline([
('vect', TfidfVectorizer(
stop_words=get_stop_words(),
token_pattern=ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('clf', MultinomialNB(alpha=0.01)),
]) evaluate_cross_validation(clf_6, X_train, Y_train, 5)

The results had an important boost from 0.89 to 0.92, pretty good. At this point, we could continue doing trials by using different values of alpha or doing new modifcations of the vectorizer.

Evaluating the performance

If we decide that we have made enough improvements in our model, we are ready to evaluate its performance on the testing set.

We will defne a helper function(见http://www.cnblogs.com/iamxyq/p/5912048.html train_and_evaluate函数) that will train the model in the entire training setand evaluate the accuracy in the training and in the testing sets.

We will evaluate our best classifer.

train_and_evaluate(clf_7, X_train, X_test, y_train, y_test)

If we look inside the vectorizer, we can see which tokens have been used to create our dictionary:

print len(clf_7.named_steps['vect'].get_feature_names())

Let's print the feature names.

clf_7.named_steps['vect'].get_feature_names()

The following table presents an extract of the results:

You can see that some words are semantically very similar, for example, sand and sands, sanctuaries and sanctuary. Perhaps if the plurals and the singulars are counted to the same bucket, we would better represent the documents. This is a very common task, which could be solved using stemming, a technique that relates two words having the same lexical root.

sklearn学习笔记2的更多相关文章

  1. sklearn学习笔记之简单线性回归

    简单线性回归 线性回归是数据挖掘中的基础算法之一,从某种意义上来说,在学习函数的时候已经开始接触线性回归了,只不过那时候并没有涉及到误差项.线性回归的思想其实就是解一组方程,得到回归函数,不过在出现误 ...

  2. sklearn学习笔记3

    Explaining Titanic hypothesis with decision trees decision trees are very simple yet powerful superv ...

  3. sklearn学习笔记1

    Image recognition with Support Vector Machines #our dataset is provided within scikit-learn #let's s ...

  4. sklearn学习笔记

    用Bagging优化模型的过程:1.对于要使用的弱模型(比如线性分类器.岭回归),通过交叉验证的方式找到弱模型本身的最好超参数:2.然后用这个带着最好超参数的弱模型去构建强模型:3.对强模型也是通过交 ...

  5. sklearn学习笔记(一)——数据预处理 sklearn.preprocessing

    https://blog.csdn.net/zhangyang10d/article/details/53418227 数据预处理 sklearn.preprocessing 标准化 (Standar ...

  6. sklearn学习笔记之岭回归

    岭回归 岭回归是一种专用于共线性数据分析的有偏估计回归方法,实质上是一种改良的最小二乘估计法,通过放弃最小二乘法的无偏性,以损失部分信息.降低精度为代价获得回归系数更为符合实际.更可靠的回归方法,对病 ...

  7. sklearn学习笔记之开始

    简介   自2007年发布以来,scikit-learn已经成为Python重要的机器学习库了.scikit-learn简称sklearn,支持包括分类.回归.降维和聚类四大机器学习算法.还包含了特征 ...

  8. sklearn学习笔记(1)--make_blobs函数及相应参数简介

    make_blobs方法: sklearn.datasets.make_blobs(n_samples=100,n_features=2,centers=3, cluster_std=1.0,cent ...

  9. Google TensorFlow深度学习笔记

    Google Deep Learning Notes Google 深度学习笔记 由于谷歌机器学习教程更新太慢,所以一边学习Deep Learning教程,经常总结是个好习惯,笔记目录奉上. Gith ...

随机推荐

  1. 转 git安装配置

    Win7上Git安装及配置过程 一.安装说明 1.Git在windows平台上安装说明. Git 是 Linux Torvalds 为了帮助管理 Linux 内核开发而开发的一个开放源码的版本控制软件 ...

  2. RPM方式安装MySQL5.6

    原文转自:http://blog.csdn.net/liumm0000/article/details/18841197 未整理! RPM方式安装MySQL5.6 a. 检查MySQL及相关RPM包, ...

  3. ThinkPHP 3.2 获取页面运行时间

    在ThinkPHP中,可以通过在config.php中配置'SHOW_PAGE_TRACE' =>true,打开页面调试,实现页面载入时间的显示.但显示在页面右下角TP的LOGO显然不能适用于我 ...

  4. Java设计模式(十三) 别人再问你设计模式,叫他看这篇文章

    原创文章,转载请务注明出处 OOP三大基本特性 封装 封装,也就是把客观事物封装成抽象的类,并且类可以把自己的属性和方法只让可信的类操作,对不可信的进行信息隐藏. 继承 继承是指这样一种能力,它可以使 ...

  5. 编写安装中断7ch的中断例程:将一个以0结尾的字符串,转化为大写

    中断的安装程序: assume cs:code code segment start: mov ax, cs mov ds, ax mov si, offset chstr mov es, ax mo ...

  6. 高效能人士必知铁律--note

    偶然看到了<高效能人士 必知铁律>这本书,我比较少看成功学,但是这本书把很多著名的成功学书籍整理出来,有时会让你耳目一新,有些观点尽管是常识,但是却加深了你对它们的理解,比如: 只要在积极 ...

  7. LeetCode-Subsets

    Given a set of distinct integers, nums, return all possible subsets. Note: The solution set must not ...

  8. [转] "self = [super init]"的解释与潜藏bug

    Objective-C的推荐init方法写法如下: - (id) init { if(self = [super init]) { //为子类增加属性进行初始化 } return self; } 这里 ...

  9. 扩展 ASP.NET MVC 模型扩展 – ASP.NET MVC 4 系列

           大部分人不能将核心运行时(System.Web 中的类)和 ASP.NET Web Forms 应用程序平台(System.Web.UI 中的类)区分开来.        ASP.NET ...

  10. mac快捷键

    切换 1. 应用程序切换 command tab 2.应用程序中的窗口间切换 command ~ ===================== 通用 1. 隐藏窗口 command H 2. 最小化窗口 ...