object ScalaApp { def main(args: Array[String]): Unit = { var conf = new SparkConf() conf.setMaster("local") var sc = new SparkContext(conf) val lines: RDD[String] = sc.textFile("./words.txt") lines.flatMap(line=>line.split()).reduc…

spark  JavaDirectKafkaWordCount 例子分析: 1. KafkaUtils.createDirectStream( jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topicsSet );后面参数意思: 源码是这样 @param ssc StreamingContext object * @param kafkaParams Kafka <…

[学习笔记] reduce将RDD中元素前两个传给输入函数,产生一个新的return值,将新产生的return值与RDD中下一个元素(即第三个元素)组成两个元素,再被传给输入函数,这样递归运作,直到最后只有一个值为止.*/    val rdd07 = sc.parallelize(1 to 10)    val sum = rdd07.reduce((x, y) => x + y)    println("sum is " + sum) 文章转载自原文:https://blog…

spark用的是cdh spark-2.0.1 package main.scala import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} /* Created by tomtang on 2016/12/16. */ object tomhelloworld { def FILE_NAME:String = "word_count_results_" def main(arg…

统计效果: 代码部分: import org.apache.spark.sql.hive.HiveContext import org.apache.spark.{Logging, SparkConf, SparkContext} import org.apache.spark.sql.{DataFrame, Row, SaveMode, _} import com.alibaba.fastjson.{JSON, JSONObject} import org.apache.hadoop.conf…

import org.apache.spark._import SparkContext._import java.util.{Calendar,Properties,Date,Locale}import java.text.SimpleDateFormat import java.math.BigDecimal;import java.math.RoundingMode;import java.text.DecimalFormat;import java.text.NumberFormat;i…

scala集合转化为DS/DF case class TestPerson(name: String, age: Long, salary: Double) val tom = TestPerson(,35.5) val sam = TestPerson(,40.5) val PersonList = mutable.MutableList[TestPerson]() PersonList += tom PersonList += sam val personDS = PersonList.to…

综合案例分析 现有数据集 department.json与employee.json,以部门名称和员工性别为粒度,试计算每个部门分性别平均年龄与平均薪资. department.json如下: {"id":1,"name":"Tech Department"} {"id":2,"name":"Fina Department"} {"id":3,"name&q…

NetworkWordCount.scala /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF lice…

1.利用scala语言开发spark的worcount程序(本地运行) package com.zy.spark import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} //todo:利用scala语言来实现spark的wordcount程序 object WordCount { def main(args: Array[String]): Unit = { //1.创建SparkConf…