Spark机器学习基础一

特征工程

对连续值处理

0.binarizer/二值化

from __future__ import print_function

from pyspark.sql import SparkSession

from pyspark.ml.feature import Binarizer

spark = SparkSession\

        .builder\

        .appName("BinarizerExample")\

        .getOrCreate()

# 创建DataFrame

continuousDataFrame = spark.createDataFrame([

    (0, 1.1),

    (1, 8.5),

    (2, 5.2)

], ["id", "feature"])

# 给定阈值，对连续特征列进行二值化。

binarizer = Binarizer(threshold=5.1, inputCol="feature", outputCol="binarized_feature")

binarizedDataFrame = binarizer.transform(continuousDataFrame)

print("Binarizer output with Threshold = %f" % binarizer.getThreshold())

binarizedDataFrame.show()

spark.stop()

Binarizer output with Threshold = 5.100000

+---+-------+-----------------+

| id|feature|binarized_feature|

+---+-------+-----------------+

|  0|    1.1|              0.0|

|  1|    8.5|              1.0|

|  2|    5.2|              1.0|

+---+-------+-----------------+

1.按照给定边界离散化

from __future__ import print_function

from pyspark.sql import SparkSession

from pyspark.ml.feature import Bucketizer

spark = SparkSession\

    .builder\

    .appName("BucketizerExample")\

    .getOrCreate()

splits = [-float("inf"), -0.5, 0.0, 0.5, float("inf")]

data = [(-999.9,), (-0.5,), (-0.3,), (0.0,), (0.2,), (999.9,)]

dataFrame = spark.createDataFrame(data, ["features"])

# 将连续特性列映射到特征桶。

bucketizer = Bucketizer(splits=splits, inputCol="features", outputCol="bucketedFeatures")

# 按照给定的边界进行分桶

bucketedData = bucketizer.transform(dataFrame)

print("Bucketizer output with %d buckets" % (len(bucketizer.getSplits())-1))

bucketedData.show()

spark.stop()

Bucketizer output with 4 buckets

+--------+----------------+

|features|bucketedFeatures|

+--------+----------------+

|  -999.9|             0.0|

|    -0.5|             1.0|

|    -0.3|             1.0|

|     0.0|             2.0|

|     0.2|             2.0|

|   999.9|             3.0|

+--------+----------------+

2.quantile_discretizer/按分位数离散化

from __future__ import print_function

from pyspark.ml.feature import QuantileDiscretizer

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("QuantileDiscretizerExample")\

    .getOrCreate()

data = [(0, 18.0), (1, 19.0), (2, 8.0), (3, 5.0), (4, 2.2), (5, 9.2), (6, 14.4)]

df = spark.createDataFrame(data, ["id", "hour"])

df = df.repartition(1)

# 分成3个桶进行离散化

discretizer = QuantileDiscretizer(numBuckets=3, inputCol="hour", outputCol="result")

result = discretizer.fit(df).transform(df)

result.show()

spark.stop()

+---+----+------+

| id|hour|result|

+---+----+------+

|  0|18.0|   2.0|

|  1|19.0|   2.0|

|  2| 8.0|   1.0|

|  3| 5.0|   0.0|

|  4| 2.2|   0.0|

|  5| 9.2|   1.0|

|  6|14.4|   2.0|

+---+----+------+

3.最大最小值幅度缩放

from __future__ import print_function

from pyspark.ml.feature import MaxAbsScaler

from pyspark.ml.linalg import Vectors

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("MaxAbsScalerExample")\

    .getOrCreate()

dataFrame = spark.createDataFrame([

    (0, Vectors.dense([1.0, 0.1, -8.0]),),

    (1, Vectors.dense([2.0, 1.0, -4.0]),),

    (2, Vectors.dense([4.0, 10.0, 8.0]),)

], ["id", "features"])

# 通过除以每个特征的最大绝对值，将每个特征单独缩放到范围[- 1,1]。

scaler = MaxAbsScaler(inputCol="features", outputCol="scaledFeatures")

# 计算最大最小值用于缩放

scalerModel = scaler.fit(dataFrame)

# 缩放幅度到[-1, 1]之间

scaledData = scalerModel.transform(dataFrame)

scaledData.select("features", "scaledFeatures").show()

spark.stop()

+--------------+----------------+

|      features|  scaledFeatures|

+--------------+----------------+

|[1.0,0.1,-8.0]|[0.25,0.01,-1.0]|

|[2.0,1.0,-4.0]|  [0.5,0.1,-0.5]|

|[4.0,10.0,8.0]|   [1.0,1.0,1.0]|

+--------------+----------------+

4.标准化

from __future__ import print_function

from pyspark.ml.feature import StandardScaler

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("StandardScalerExample")\

    .getOrCreate()

dataFrame = spark.read.format("libsvm").load("sample_libsvm_data.txt")

# 使用训练集中样本的列汇总统计信息，通过删除均值并缩放到单位方差来标准化特征。

scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures", withStd=True, withMean=False)

# 计算均值方差等参数

scalerModel = scaler.fit(dataFrame)

# 标准化

scaledData = scalerModel.transform(dataFrame)

scaledData.show()

spark.stop()

+-----+--------------------+--------------------+

|label|            features|      scaledFeatures|

+-----+--------------------+--------------------+

|  0.0|(692,[127,128,129...|(692,[127,128,129...|

|  1.0|(692,[158,159,160...|(692,[158,159,160...|

|  1.0|(692,[124,125,126...|(692,[124,125,126...|

|  1.0|(692,[152,153,154...|(692,[152,153,154...|

|  1.0|(692,[151,152,153...|(692,[151,152,153...|

|  0.0|(692,[129,130,131...|(692,[129,130,131...|

|  1.0|(692,[158,159,160...|(692,[158,159,160...|

|  1.0|(692,[99,100,101,...|(692,[99,100,101,...|

|  0.0|(692,[154,155,156...|(692,[154,155,156...|

|  0.0|(692,[127,128,129...|(692,[127,128,129...|

|  1.0|(692,[154,155,156...|(692,[154,155,156...|

|  0.0|(692,[153,154,155...|(692,[153,154,155...|

|  0.0|(692,[151,152,153...|(692,[151,152,153...|

|  1.0|(692,[129,130,131...|(692,[129,130,131...|

|  0.0|(692,[154,155,156...|(692,[154,155,156...|

|  1.0|(692,[150,151,152...|(692,[150,151,152...|

|  0.0|(692,[124,125,126...|(692,[124,125,126...|

|  0.0|(692,[152,153,154...|(692,[152,153,154...|

|  1.0|(692,[97,98,99,12...|(692,[97,98,99,12...|

|  1.0|(692,[124,125,126...|(692,[124,125,126...|

+-----+--------------------+--------------------+

only showing top 20 rows

from __future__ import print_function

from pyspark.ml.feature import StandardScaler

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("StandardScalerExample")\

    .getOrCreate()

dataFrame = spark.createDataFrame([

    (0, Vectors.dense([1.0, 0.1, -8.0]),),

    (1, Vectors.dense([2.0, 1.0, -4.0]),),

    (2, Vectors.dense([4.0, 10.0, 8.0]),)

], ["id", "features"])

# 计算均值方差等参数

scalerModel = scaler.fit(dataFrame)

# 标准化

scaledData = scalerModel.transform(dataFrame)

scaledData.show()

spark.stop()

+---+--------------+--------------------+

| id|      features|      scaledFeatures|

+---+--------------+--------------------+

|  0|[1.0,0.1,-8.0]|[0.65465367070797...|

|  1|[2.0,1.0,-4.0]|[1.30930734141595...|

|  2|[4.0,10.0,8.0]|[2.61861468283190...|

+---+--------------+--------------------+

5.添加多项式特征

from __future__ import print_function

from pyspark.ml.feature import PolynomialExpansion

from pyspark.ml.linalg import Vectors

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("PolynomialExpansionExample")\

    .getOrCreate()

df = spark.createDataFrame([

    (Vectors.dense([2.0, 1.0]),),

    (Vectors.dense([0.0, 0.0]),),

    (Vectors.dense([3.0, -1.0]),)

], ["features"])

# 在多项式空间中进行特征展开。

polyExpansion = PolynomialExpansion(degree=3, inputCol="features", outputCol="polyFeatures")

polyDF = polyExpansion.transform(df)

polyDF.show(truncate=False)

spark.stop()

+----------+------------------------------------------+

|features  |polyFeatures                              |

+----------+------------------------------------------+

|[2.0,1.0] |[2.0,4.0,8.0,1.0,2.0,4.0,1.0,2.0,1.0]     |

|[0.0,0.0] |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]     |

|[3.0,-1.0]|[3.0,9.0,27.0,-1.0,-3.0,-9.0,1.0,3.0,-1.0]|

+----------+------------------------------------------+

对离散型处理

0.独热向量编码

from __future__ import print_function

from pyspark.ml.feature import OneHotEncoder, StringIndexer

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("OneHotEncoderExample")\

    .getOrCreate()

df = spark.createDataFrame([

    (0, "a"),

    (1, "b"),

    (2, "c"),

    (3, "a"),

    (4, "a"),

    (5, "c")

], ["id", "category"])

# 将分类值转换为类别索引

stringIndexer = StringIndexer(inputCol="category", outputCol="categoryIndex")

model = stringIndexer.fit(df)

indexed = model.transform(df)

# 独热向量编码

encoder = OneHotEncoder(inputCol="categoryIndex", outputCol="categoryVec")

encoded = encoder.transform(indexed)

encoded.show()

spark.stop()

+---+--------+-------------+-------------+

| id|category|categoryIndex|  categoryVec|

+---+--------+-------------+-------------+

|  0|       a|          0.0|(2,[0],[1.0])|

|  1|       b|          2.0|    (2,[],[])|

|  2|       c|          1.0|(2,[1],[1.0])|

|  3|       a|          0.0|(2,[0],[1.0])|

|  4|       a|          0.0|(2,[0],[1.0])|

|  5|       c|          1.0|(2,[1],[1.0])|

+---+--------+-------------+-------------+

对文本型处理

0.去停用词

from __future__ import print_function

from pyspark.ml.feature import StopWordsRemover

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("StopWordsRemoverExample")\

    .getOrCreate()

sentenceData = spark.createDataFrame([

    (0, ["I", "saw", "the", "red", "balloon"]),

    (1, ["Mary", "had", "a", "little", "lamb"])

], ["id", "raw"])

# 去停用词

remover = StopWordsRemover(inputCol="raw", outputCol="filtered")

remover.transform(sentenceData).show(truncate=False)

spark.stop()

+---+----------------------------+--------------------+

|id |raw                         |filtered            |

+---+----------------------------+--------------------+

|0  |[I, saw, the, red, balloon] |[saw, red, balloon] |

|1  |[Mary, had, a, little, lamb]|[Mary, little, lamb]|

+---+----------------------------+--------------------+

1.Tokenizer 分词器

from __future__ import print_function

from pyspark.ml.feature import Tokenizer, RegexTokenizer

from pyspark.sql.functions import col, udf

from pyspark.sql.types import IntegerType

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("TokenizerExample")\

    .getOrCreate()

sentenceDataFrame = spark.createDataFrame([

    (0, "Hi I heard about Spark"),

    (1, "I wish Java could use case classes"),

    (2, "Logistic,regression,models,are,neat")

], ["id", "sentence"])

# 分词

tokenizer = Tokenizer(inputCol="sentence", outputCol="words")

# 通过使用提供的正则表达式模式分割文本

regexTokenizer = RegexTokenizer(inputCol="sentence", outputCol="words", pattern="\\W")

countTokens = udf(lambda words: len(words), IntegerType())

tokenized = tokenizer.transform(sentenceDataFrame)

tokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(truncate=False)

regexTokenized = regexTokenizer.transform(sentenceDataFrame)

regexTokenized.select("sentence", "words").withColumn("tokens", countTokens(col("words"))).show(truncate=False)

spark.stop()

+-----------------------------------+------------------------------------------+------+

|sentence                           |words                                     |tokens|

+-----------------------------------+------------------------------------------+------+

|Hi I heard about Spark             |[hi, i, heard, about, spark]              |5     |

|I wish Java could use case classes |[i, wish, java, could, use, case, classes]|7     |

|Logistic,regression,models,are,neat|[logistic,regression,models,are,neat]     |1     |

+-----------------------------------+------------------------------------------+------+

+-----------------------------------+------------------------------------------+------+

|sentence                           |words                                     |tokens|

+-----------------------------------+------------------------------------------+------+

|Hi I heard about Spark             |[hi, i, heard, about, spark]              |5     |

|I wish Java could use case classes |[i, wish, java, could, use, case, classes]|7     |

|Logistic,regression,models,are,neat|[logistic, regression, models, are, neat] |5     |

+-----------------------------------+------------------------------------------+------+

2.count_vectorizer

from __future__ import print_function

from pyspark.sql import SparkSession

from pyspark.ml.feature import CountVectorizer

spark = SparkSession\

    .builder\

    .appName("CountVectorizerExample")\

    .getOrCreate()

df = spark.createDataFrame([

    (0, "a b c".split(" ")),

    (1, "a b b c a".split(" "))

], ["id", "words"])

# 从文档集合中提取词汇表并生成CountVectorizerModel

cv = CountVectorizer(inputCol="words", outputCol="features", vocabSize=3, minDF=2.0)

model = cv.fit(df)

result = model.transform(df)

result.show(truncate=False)

spark.stop()

+---+---------------+-------------------------+

|id |words          |features                 |

+---+---------------+-------------------------+

|0  |[a, b, c]      |(3,[0,1,2],[1.0,1.0,1.0])|

|1  |[a, b, b, c, a]|(3,[0,1,2],[2.0,2.0,1.0])|

+---+---------------+-------------------------+

3.TF-IDF权重

from __future__ import print_function

from pyspark.ml.feature import HashingTF, IDF, Tokenizer

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("TfIdfExample")\

    .getOrCreate()

sentenceData = spark.createDataFrame([

    (0.0, "Hi I heard about Spark"),

    (0.0, "I wish Java could use case classes"),

    (1.0, "Logistic regression models are neat")

], ["label", "sentence"])

# 分词

tokenizer = Tokenizer(inputCol="sentence", outputCol="words")

wordsData = tokenizer.transform(sentenceData)

# 使用哈希技巧将一系列项映射到它们的项频率。

hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=20)

featurizedData = hashingTF.transform(wordsData)

# 计算给定文档集合的逆文档频率(IDF)。

idf = IDF(inputCol="rawFeatures", outputCol="features")

idfModel = idf.fit(featurizedData)

rescaledData = idfModel.transform(featurizedData)

rescaledData.select("label", "features").show()

spark.stop()

+-----+--------------------+

|label|            features|

+-----+--------------------+

|  0.0|(20,[0,5,9,17],[0...|

|  0.0|(20,[2,7,9,13,15]...|

|  1.0|(20,[4,6,13,15,18...|

+-----+--------------------+

4.n-gram语言模型

from __future__ import print_function

from pyspark.ml.feature import NGram

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("NGramExample")\

    .getOrCreate()

#Hanmeimei loves LiLei

#LiLei loves Hanmeimei

wordDataFrame = spark.createDataFrame([

    (0, ["Hi", "I", "heard", "about", "Spark"]),

    (1, ["I", "wish", "Java", "could", "use", "case", "classes"]),

    (2, ["Logistic", "regression", "models", "are", "neat"])

], ["id", "words"])

# 将字符串输入数组转换为n-grams数组的特征转换器。

ngram = NGram(n=2, inputCol="words", outputCol="ngrams")

ngramDataFrame = ngram.transform(wordDataFrame)

ngramDataFrame.select("ngrams").show(truncate=False)

spark.stop()

+------------------------------------------------------------------+

|ngrams                                                            |

+------------------------------------------------------------------+

|[Hi I, I heard, heard about, about Spark]                         |

|[I wish, wish Java, Java could, could use, use case, case classes]|

|[Logistic regression, regression models, models are, are neat]    |

+------------------------------------------------------------------+

高级变换

0.SQL变换

from __future__ import print_function

from pyspark.ml.feature import SQLTransformer

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("SQLTransformerExample")\

    .getOrCreate()

df = spark.createDataFrame([

    (0, 1.0, 3.0),

    (2, 2.0, 5.0)

], ["id", "v1", "v2"])

# 实现由SQL语句定义的转换。

sqlTrans = SQLTransformer(statement="SELECT *, (v1 + v2) AS v3, (v1 * v2) AS v4 FROM __THIS__")

sqlTrans.transform(df).show()

spark.stop()

+---+---+---+---+----+

| id| v1| v2| v3|  v4|

+---+---+---+---+----+

|  0|1.0|3.0|4.0| 3.0|

|  2|2.0|5.0|7.0|10.0|

+---+---+---+---+----+

1.R公式变换

from __future__ import print_function

from pyspark.ml.feature import RFormula

from pyspark.sql import SparkSession

spark = SparkSession\

    .builder\

    .appName("RFormulaExample")\

    .getOrCreate()

dataset = spark.createDataFrame(

    [(7, "US", 18, 1.0),

     (8, "CA", 12, 0.0),

     (9, "NZ", 15, 0.0)],

    ["id", "country", "hour", "clicked"])

# 实现根据R模型公式拟合数据集所需的转换。

formula = RFormula(

    formula="clicked ~ country + hour",

    featuresCol="features",

    labelCol="label")

output = formula.fit(dataset).transform(dataset)

output.select("features", "label").show()

spark.stop()

+--------------+-----+

|      features|label|

+--------------+-----+

|[0.0,0.0,18.0]|  1.0|

|[0.0,1.0,12.0]|  0.0|

|[1.0,0.0,15.0]|  0.0|

+--------------+-----+

Spark机器学习基础一的更多相关文章

spark 机器学习基础数据类型
spark的机器学习库,包含常见的学习算法和工具如分类.回归.聚类.协同过滤.降维等使用算法时都需要指定相应的数据集,下面为大家介绍常用的spark ml 数据类型.1.本地向量(Local Vect ...
Spark机器学习基础三
监督学习 0.线性回归(加L1.L2正则化) from __future__ import print_function from pyspark.ml.regression import Linea ...
Spark机器学习基础二
无监督学习 0.K-means from __future__ import print_function from pyspark.ml.clustering import KMeans #from ...
Spark机器学习基础-监督学习
监督学习 0.线性回归(加L1.L2正则化) from __future__ import print_function from pyspark.ml.regression import Linea ...
Spark机器学习基础-无监督学习
0.K-means from __future__ import print_function from pyspark.ml.clustering import KMeans#硬聚类 #from p ...
Spark机器学习基础-特征工程
对连续值处理 0.binarizer/二值化 from __future__ import print_function from pyspark.sql import SparkSession fr ...
Spark机器学习4·分类模型(spark-shell)
线性模型逻辑回归--逻辑损失(logistic loss) 线性支持向量机(Support Vector Machine, SVM)--合页损失(hinge loss) 朴素贝叶斯(Naive Ba ...
掌握Spark机器学习库（课程目录）
第1章初识机器学习在本章中将带领大家概要了解什么是机器学习.机器学习在当前有哪些典型应用.机器学习的核心思想.常用的框架有哪些,该如何进行选型等相关问题. 1-1 导学 1-2 机器学习概述 1- ...
Spark机器学习MLlib系列１（for python）－－数据类型，向量，分布式矩阵，API
Spark机器学习MLlib系列1(for python)--数据类型,向量,分布式矩阵,API 关键词:Local vector,Labeled point,Local matrix,Distrib ...

随机推荐

pandas(三)
合并数据集: 创建一个能创建dataframe的函数 def make_data(cols,ind): data={c:[strc(c)+str(i) for i in ind] for c in c ...
Hibernate处理事务并发问题
在Hibernate中设置事务的隔离级别.<property name="hibernate.connection.isolation">2</property& ...
2019年 Gratner数据分析平台对比 - PowerBI大幅领先
先睹为快,看看你正在用的工具在哪里? 文末见2017-2018图对比2019年, 1.ThoughtSpot好像发展很快 2.IBM...... 3.Microstrategy好像表现还不错 4.L ...
less is more，so 只记 less
less + 文件名 1.Enter键 :向下翻一行 2.空格键 :向下翻一屏 3.j键 :想下翻一行 4.k键 :向上翻一行 5.f键 :向下翻一屏 6.b键 : 向上翻一屏 7.d键 :向下翻半屏 ...
Keras RetinaNet github项目
https://github.com/fizyr/keras-retinanet 根据此网站的方法,利用Pascal VOC 2007数据集开始训练,出现error: D:\JupyterWorkSp ...
Linux给MySQL创建用户，并分配权限
//登录MYSQL 使用root账号登录 @>mysql -u root -p @>密码 //创建用户 mysql> insert into mysql.user(Host,User ...
kali在vbox上运行设置共享文件夹
mount -t vboxsf VBoxShared /root/Desktop/vbox 0x00 使用共享文件夹的前提需要自行安装增强功能:https://jingyan.baidu.com/a ...
2019微信浏览器跳转外部浏览器下载app打开任意站实现方法
很多朋友问我怎么解决微信内点击链接或扫描二维码可以直接跳出微信在外部浏览器打开网页链接和下载APP,其实这并不难,只要我们实现微信跳转功能即可.下面给大家介绍这个功能方案实现教程: 功能目的生成微 ...
关于element-ui表格table设置header-cell-class-name样式不起作用的原因分析
在编写表格的时候想给表头添加样式,使用 header-cell-class-name怎么都添加不上样式,检查元素发现连class都没添加上,查了很多资料有人说element之前版本不支持这属性,但我使 ...
es6 遍历总结
1.for in / for of for in --> index是key值 var array = [1,2,3,4,5]; for(let index in array) { consol ...

Spark机器学习基础一

特征工程

对连续值处理

0.binarizer/二值化

1.按照给定边界离散化

2.quantile_discretizer/按分位数离散化

3.最大最小值幅度缩放

4.标准化

5.添加多项式特征

对离散型处理

0.独热向量编码

对文本型处理

0.去停用词

1.Tokenizer 分词器

2.count_vectorizer

3.TF-IDF权重

4.n-gram语言模型

高级变换

0.SQL变换

1.R公式变换

Spark机器学习基础一的更多相关文章

随机推荐

热门专题