wordcount报错:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist:
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://192.168.25.128:9000/export/yang/log.1
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at hadoop1.WordCount.main(WordCount.java:53)
当本人在运行,Hadoop集群自带的wordcount实例的时候,报错内容为输入路径不存在,在网上找了很久没有解决,最后发现是因为我创建的log.1是在本地创建的,并没有上传到hdfs集群中,所以在运行的时候会报错,解决的办法是:执行命令:
[root@master ~]# hadoop fs -put log.1 / #(将log.1文件上传到/目录下)
操作之后可以再次运行命令:
[root@master ~]# hadoop jar /export/servers/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /1.log /result
执行结果如下:
File System Counters
FILE: Number of bytes read=312
FILE: Number of bytes written=237571
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=300
HDFS: Number of bytes written=206
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=7544
Total time spent by all reduces in occupied slots (ms)=5156
Total time spent by all map tasks (ms)=7544
Total time spent by all reduce tasks (ms)=5156
Total vcore-milliseconds taken by all map tasks=7544
Total vcore-milliseconds taken by all reduce tasks=5156
Total megabyte-milliseconds taken by all map tasks=7725056
Total megabyte-milliseconds taken by all reduce tasks=5279744
Map-Reduce Framework
Map input records=1
Map output records=35
Map output bytes=342
Map output materialized bytes=312
Input split bytes=97
Combine input records=35
Combine output records=25
Reduce input groups=25
Reduce shuffle bytes=312
Reduce input records=25
Reduce output records=25
Spilled Records=50
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=230
CPU time spent (ms)=2110
Physical memory (bytes) snapshot=306843648
Virtual memory (bytes) snapshot=4163534848
Total committed heap usage (bytes)=142278656
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=203
File Output Format Counters
Bytes Written=206
运行成功!
原文链接:https://blog.csdn.net/yangmingtia/article/details/83993206
wordcount报错:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist:的更多相关文章
- 报错org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to org.apache.hadoop.mapred.FileSplit
报错 java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSpli ...
- 用java运行Hadoop程序报错:org.apache.hadoop.fs.LocalFileSystem cannot be cast to org.apache.
用java运行Hadoop例程报错:org.apache.hadoop.fs.LocalFileSystem cannot be cast to org.apache.所写代码如下: package ...
- sparkOnYarn报错org.apache.hadoop.fs.FSDataInputStream
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInpu ...
- 配置MapReduce插件时,弹窗报错org/apache/hadoop/eclipse/preferences/MapReducePreferencePage : Unsupported major.minor version 51.0(Hadoop2.7.3集群部署)
原因: hadoop-eclipse-plugin-2.7.3.jar 编译的jdk版本和eclipse启动使用的jdk版本不一致导致. 解决方案一: 修改myeclipse.ini文件即可解决. ...
- org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/input
原我是这样写的 //输入数据所在的文件目录 FileInputFormat.addInputPath(job, new Path("/input/")); //mapreduce执 ...
- Hadoop报错:org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxx
问题出现原因: 因为远程提交hadoop的任务的情况下如果,没有hadoop 的系统环境变量,就会读取当前主机的用户名,所以Hadoop集群的节点中没有该用户名的权限,所以出现的异常. 解决方法: S ...
- hbase报错org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
这个错误找了好长时间,日志看半天,找度娘没什么用,再看stackoverflow,上面有几篇相关的帖子,但是没人回答. 看半天,没有cause,只好到idea中去看sortLocalSorts源码,结 ...
- 使用eclipse的快捷键自动生成的map或者reduce函数的参数中:“org.apache.hadoop.mapreduce.Reducer.Context context”
今天在测试mapreduce的程序时,就是简单的去重,对照课本上的程序和自己的程序,唯一不同的就是“org.apache.hadoop.mapreduce.Reducer.Context contex ...
- hadoop项目开发运行报错(log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).)
使用hadoop+myeclipse开发项目是测试运行报错: log4j:WARN No appenders could be found for logger (org.apache.hadoop. ...
随机推荐
- 用相对路径有时居然是这样,,加上<%=basePath%>
用相对路径有时居然是这样,所以还是用绝对路径好点,加上<%=basePath%> 比如create页面的action为ssh/pages/User/create,那么create页面的上的 ...
- 分布式系统一致性算法(Paxos)
CAP理论 一致性(Consistency) 可用性(Availability) 分区容错性(网络分区)Partition toleranceCAP理论的特点,就是CAP只能满足其中 ...
- Xftp设置指定记事本(notepad++)打开文件
右键
- Lucene 基础类型
Lucene 索引文件中,用一下基本类型来保存信息:1. Byte:是最基本的类型,长 8 位(bit).2. UInt32:由 4 个 Byte 组成.3. UInt64:由 8 个 Byte 组成 ...
- sigslot之简化版
1.关于 1.1 基于原版sigslot做了部分修改.原版的信号支持最多支持8个参数,改进后,最多支持1个参数,这样肯定不能满足日常需求,但是,我们可以将信号的一个参数改为类型指针,比如信号定义时的模 ...
- 【LeetCode】面试题13. 机器人的运动范围
作者: 负雪明烛 id: fuxuemingzhu 个人博客:http://fuxuemingzhu.cn/ 目录 题目描述 题目大意 解题方法 BFS 日期 题目地址:https://leetcod ...
- 【LeetCode】320. Generalized Abbreviation 解题报告 (C++)
作者: 负雪明烛 id: fuxuemingzhu 个人博客:http://fuxuemingzhu.cn/ 目录 题目描述 题目大意 解题方法 DFS 日期 题目地址:https://leetcod ...
- 【LeetCode】400. Nth Digit 解题报告(Python)
作者: 负雪明烛 id: fuxuemingzhu 个人博客: http://fuxuemingzhu.cn/ 目录 题目描述 题目大意 解题方法 日期 题目地址:https://leetcode.c ...
- hdu 1528-Card Game Cheater(贪心算法)
题意不讲,怕说不清,自己一点点看吧. 思路是贪心,将每个人的牌按从小到大或(从大到小),我是从小到大排的, 然后每次从第二摞排中找比第一摞排的那张大且相差最小的就可以了,每次找到就sum++: 最后s ...
- anaconda 安装 gdown
pip install gdown注意不要使用conda install gdown 这样会报错参考:gdown · PyPI