Spark wordcount开发并提交到集群运行

使用的ide是eclipse

package com.luogankun.spark.base
 
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
 
/**
 * 统计字符出现次数
 */
object WordCount {
  def main(args: Array[String]) {
    if (args.length < 1) {
      System.err.println("Usage: <file>")
      System.exit(1)
    }
 
    val conf = new SparkConf()
    val sc = new SparkContext(conf)
    val line = sc.textFile(args(0))
 
    line.flatMap(_.split("\t")).map((_, 1)).reduceByKey(_+_).collect.foreach(println)
 
    sc.stop
  }
}

导出成jar包。

提交到集群运行脚本: WordCount.sh

#!/bin/bash
 
set -x
 
cd $SPARK_HOME/bin
 
spark-submit \
--name WordCount \
--class com.luogankun.spark.base.WordCount \
--master spark://hadoop000:7077 \
--executor-memory 1G \
--total-executor-cores 1 \
/home/spark/lib/spark.jar \
hdfs://hadoop000:8020/hello.txt

执行WordCount.sh脚本