配好了Spark集群后,先用pyspark写了两个小例子，但是发现Tab键没有提示，于是打算转到scala上试试，在spark-shell下有提示了，但是发现不能退格，而且提示也不是复写，而是追加，这样根本就没法写程序.

      解决办法：

1.打开会话选项

2.终端-仿真    在终端中选择Linux

3.映射键   勾选两个选项

4.至此已经成功了,但是如果远程长时间未操作 就会中断连接，下次再操作时需要等待，其实也很影响使用，在这里也附上解决办法(可选)

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

spark-shell下有提示了，但是发现不能退格的更多相关文章

  1. 02、体验Spark shell下RDD编程

    02.体验Spark shell下RDD编程 1.Spark RDD介绍 RDD是Resilient Distributed Dataset,中文翻译是弹性分布式数据集.该类是Spark是核心类成员之 ...

  2. ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法

    ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法 转载:http://blog.csdn.net/yao_qinwei/article/details/8761777 在插入模式下,按删 ...

  3. ubuntu 下使用vi时方向键乱码，退格键不能使用

    ubuntu 下使用vi时方向键乱码,退格键不能使用的解决方法 问题表现:Ubuntu下,初始使用vi,编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现A B C D 之类的字母,并且 ...

  4. 在Scala IDEA for Eclipse或IDEA里程序编译实现与在Spark Shell下的对比（其实就是那么一回事）

    不多说,直接上干货! 比如,我这里拿主成分分析(PCA). 1.主成分分析(PCA)的概念介绍 主成分分析(PCA) 是一种对数据进行旋转变换的统计学方法,其本质是在线性空间中进行一个基变换,使得变换 ...

  5. Cygwin下安装vim后，vim中退格键无法正常使用

    问题描述: 在Cygwin中安装完vim后 进入vim,发现上下左右键和退格键都无法正常使用 问题分析: 首先考虑到的就是缺少vim的配置文件,首先查看/etc路径下是否有vim的配置文件 admin ...

  6. Ubuntu下使用Vi时方向键变乱码 退格键不能使用的解决方法

    要在Ubuntu下编辑一些文件,这就涉及到了vi这个编辑器了.在Ubuntu下,初始使用vi的时候有点问题,就是在编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现[A [B [C [D ...

  7. 解决Ubuntu 下 vi编辑器不能使用方向键和退格键问题

    转自:http://blog.csdn.net/sky101010ws/article/details/51012103 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多字母 这个问题主要是 ...

  8. 解决Ubuntu14.04下vi编辑器不能使用方向键和退格键问题

    参考:http://blog.sina.com.cn/s/blog_7d0c2fed01010zbi.html 系统:Ubuntu14.04 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多 ...

  9. ubuntu 下解决sublime v3 中文输入法时 退格键删除不了拼音的问题

    ubuntu下,sulime想要支持中文需要这样设置: 1.安装中文输入解决的github git clone https://github.com/lyfeyaj/sublime-text-imfi ...

随机推荐

  1. wpf中的数据模板

    wpf中的模板分为数据模板和控件模板,我们可以通过我们自己定制的数据模板来制定自己想要的数据表现形式.例如:时间的显示可以通过图片,也可以通过简单数字表现出来. 例如: (1)先在Demo这个命名空间 ...

  2. Adobe Flex初记

    公司项目要用Flex,之前没有接触过,菜鸟只好白手起家,把项目拉下来的同时配置下Flex的环境,以下是一篇参考: http://blog.sina.com.cn/s/blog_4c4a24db0100 ...

  3. Python爬虫加速神器的小试

    大名鼎鼎的aiohttp,相信如果你学习Python或者爬虫的时候,肯定听说过这个东西.没听过也不要紧,今天看完文章,只要记住,aiohttp这个东西,在写爬虫的时候,很牛逼就行了. aiohttp ...

  4. 如何解决selenium打开chrome提示chromedriver.exe已停止工作

    场景:启动Chrome,打开URL,提示“disconnected: unable to connect to renderer” 解决方法:chromedriver与chrome的对应关系表, 需要 ...

  5. php获取数据库结构

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/ ...

  6. TensorFlow设置GPU占用量

    默认开启Tensorflow的session之后,就会占用几乎所有的显存,进行如下设置即可: 指定GPU编号: import os os.environ["CUDA_VISIBLE_DEVI ...

  7. 自定义指令（v-check、v-focus）的方法有哪些？它有哪些钩子函数？还有哪些钩子函数参数？

    全局定义指令:在vue对象的directive方法里面有两个参数,一个是指令名称,另一个是函数.组件内定义指令:directives: 钩子函数:bind(绑定事件触发).inserted(节点插入的 ...

  8. logback logback.xml常用配置详解（二）<appender>

    转自:http://aub.iteye.com/blog/1101260 logback 常用配置详解(二) <appender> <appender>: <append ...

  9. LeetCode-111.Mininum Depth of Binary Tree

    Given a binary tree, find its minimum depth. The minimum depth is the number of nodes along the shor ...

  10. （1.15）mysql锁问题之InnoDB锁

    关键词:innodb锁,mysql锁 概要: 1.事务的隔离级别 2.InnoDB锁争用 3.innodb锁 [3.1]innodb的行锁模式及加锁方法 其实,默认情况下,mysql 的 select ...