【Spark】通过SparkStreaming实现从socket接受数据,并进行简单的单词计数
步骤
一、创建maven工程并导入jar包
<properties>
<scala.version>2.11.8</scala.version>
<spark.version>2.2.0</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
<!-- <verbal>true</verbal>-->
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
二、安装并启动生产者
在node01安装nc工具
yum -y install nc
使用nc工具向指定端口发送数据
nc -lk 9999
三、开发SparkStreaming代码
import org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.streaming.{Seconds, StreamingContext}
object WordCountTest {
def main(args: Array[String]): Unit = {
//获取SparkConf
val sparkConf: SparkConf = new SparkConf().setAppName("Streaming_WordCountTest").setMaster("local[4]").set("spark.driver.host", "localhost")
//获取SparkContext
val sparkContext: SparkContext = new SparkContext(sparkConf)
//设置日志级别
sparkContext.setLogLevel("WARN")
//获取StreamingContext 需要两个参数 SparkContext和duration,后者就是间隔时间
val streamContext: StreamingContext = new StreamingContext(sparkContext, Seconds(5))
//从socket获取数据
val stream: ReceiverInputDStream[String] = streamContext.socketTextStream("node01", 9999)
//对数据进行计数操作
val result: DStream[(String, Int)] = stream.flatMap(x => x.split(" ")).map((_, 1)).reduceByKey(_ + _)
//输出数据
result.print()
//启动程序
streamContext.start()
streamContext.awaitTermination()
}
}
四、查看结果
nc工具发送的数据
控制台结果
-----------------------------------------
Time: 1586852050000 ms
-------------------------------------------
(hive,1)
(wro,1)
(hadoop,2)
(hello,4)
(java,1)
(ja,1)
(world,1)
-------------------------------------------
Time: 1586852055000 ms
-------------------------------------------
-------------------------------------------
Time: 1586852060000 ms
-------------------------------------------
20/04/14 16:14:23 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:23 WARN BlockManager: Block input-0-1586852063400 replicated to only 0 peer(s) instead of 1 peers
20/04/14 16:14:24 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:24 WARN BlockManager: Block input-0-1586852064000 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 1586852065000 ms
-------------------------------------------
(,2)
20/04/14 16:14:29 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:29 WARN BlockManager: Block input-0-1586852069600 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 1586852070000 ms
-------------------------------------------
(456,1)
(123,1)
20/04/14 16:14:31 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:31 WARN BlockManager: Block input-0-1586852071200 replicated to only 0 peer(s) instead of 1 peers
20/04/14 16:14:34 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:34 WARN BlockManager: Block input-0-1586852073800 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 1586852075000 ms
-------------------------------------------
(zhao,1)
(456,1)
(123,1)
20/04/14 16:14:36 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
20/04/14 16:14:36 WARN BlockManager: Block input-0-1586852076200 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 1586852080000 ms
-------------------------------------------
(zhao,2)
-------------------------------------------
Time: 1586852085000 ms
-------------------------------------------
-------------------------------------------
Time: 1586852090000 ms
-------------------------------------------
【Spark】通过SparkStreaming实现从socket接受数据,并进行简单的单词计数的更多相关文章
- C# Socket 接受数据不全的处理
由于Socket 一次传输数据有限,因此需要多次接受数据传输. 解决办法一: int numberOfBytesRead = 0; int totalNumberOfBytes = 0 ...
- spark-streaming集成Kafka处理实时数据
在这篇文章里,我们模拟了一个场景,实时分析订单数据,统计实时收益. 场景模拟 我试图覆盖工程上最为常用的一个场景: 1)首先,向Kafka里实时的写入订单数据,JSON格式,包含订单ID-订单类型-订 ...
- Spark Streaming源码解读之流数据不断接收和全生命周期彻底研究和思考
本节的主要内容: 一.数据接受架构和设计模式 二.接受数据的源码解读 Spark Streaming不断持续的接收数据,具有Receiver的Spark 应用程序的考虑. Receiver和Drive ...
- spark or sparkstreaming的内存泄露问题?
关于sparkstreaming的无法正常产生数据---->到崩溃---->到数据读写极为缓慢(或块丢失?)问题 前两阶段请看我的博客:https://www.cnblogs.com/wa ...
- 3 python3 编码解码问题 upd接受数据
1.python3下的中文乱码:send_data.encode("utf-8") from socket import * udp_socket = socket(AF_INET ...
- 【Spark】SparkStreaming与flume进行整合
文章目录 注意事项 SparkStreaming从flume中poll数据 步骤 一.开发flume配置文件 二.启动flume 三.开发sparkStreaming代码 1.创建maven工程,导入 ...
- C#上位机制作之串口接受数据(利用接受事件)
前面设计好了界面,现在就开始写代码了,首先定义一个串口对象.. SerialPort serialport = new SerialPort();//定义串口对象 添加串口扫描函数,扫描出来所有可用串 ...
- dsp28377控制DM9000收发数据——第三版程序,通过外部引脚触发来实现中断接受数据,优化掉帧现象
//-------------------------------------------------------------------------------------------- - //D ...
- PHP+socket游戏数据统计平台发包接包类库
<?php /** * @title: PHP+socket游戏数据统计平台发包接包类库 * @version: 1.0 * @author: perry <perry@1kyou.com ...
随机推荐
- Julia基础语法复数和分数
1.复数 2.分数
- PAS
一.概念 二.安装 打开Delphi,在主菜单上选择Component,单击Install Component,出现图所示的对话框.有两个选择,装到已经存在的包里面和装到新的包里面.我们选择后者,单击 ...
- 复习python的__call__ __str__ __repr__ __getattr__函数 整理
class Www: def __init__(self,name): self.name=name def __str__(self): return '名称 %s'%self.name #__re ...
- Linux学习笔记(二)文件操作命令
文件操作命令 touch stat cat more less head tail ln touch 英文原意: change file timestamps 功能: 修改文件的时间戳 语法: tou ...
- [YII2] 增删改查2
一.新增 使用model::save()操作进行新增数据 $user= new User; $user->username =$username; $user->password =$pa ...
- golang实现并发爬虫二(简单调度器)
上篇文章当中实现了单任务版爬虫. 那么这篇文章就大概说下,如何在上一个版本中进行升级改造,使之成为一个多任务版本的爬虫.加快我们爬取的速度. 话不多说,先看图: 其实呢,实现方法就是加了一个sched ...
- ORM之单表、多表操作
参考1 参考2 表与表之间的关系: 一对一(OneToOneField):一对一字段无论建在哪张关系表里面都可以,但是推荐建在查询频率比较高的那张表里面 一对多(ForeignKey):一对多字段建在 ...
- CSS躬行记(7)——合成
在图形编辑软件中,可以按特定地方式处理不同图层的合成,最新的CSS规范也引入了该功能,并提供了mix-blend-mode和background-blend-mode两个属性.混合模式(blendin ...
- 大数据作业之利用MapRedeuce实现简单的数据操作
Map/Reduce编程作业 现有student.txt和student_score.txt.将两个文件上传到hdfs上.使用Map/Reduce框架完成下面的题目 student.txt 20160 ...
- 【Linux题目】第六关
[定时任务规则] 1. 如果在某用户的crontab文件中有以下记录,该行中的命令多久执行一次(RHCE考试题)?( ) 30 4 * * 3 mycmd A. 每小时. B. 每周. C. 每年三月 ...