Spark报错:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的。因为工作需要,需要在windows上先调试该程序,然后再转到linux下。程序运行的过程中,报 Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable
null
\bin\winutils.exe in the Hadoop binaries.
通过断点调试、查看源码发现程序需要根据HADOOP_HOME找到winutils.exe,由于win机器并没有配置该环境变量,所以程序报 null\bin\winutils.exe。
private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
return home;
}
private static String HADOOP_HOME_DIR = checkHadoopHome ();
public static final String getQualifiedBinPath(String executable)
throws IOException {
// construct hadoop bin path to the specified executable
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"
+ File.separator + executable; File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName
+ " in the Hadoop binaries.");
}
return exeFile.getCanonicalPath();
} /** a Windows utility to emulate Unix commands */
public static final String WINUTILS = getWinUtilsPath(); public static final String getWinUtilsPath() {
String winUtilsPath = null; try {
if (WINDOWS) {
winUtilsPath = getQualifiedBinPath("winutils.exe");
}
} catch (IOException ioe) {
LOG.error("Failed to locate the winutils binary in the hadoop binary path",
ioe);
} return winUtilsPath;
}
找到原因后就去网上问了度娘,找到了解决方案,很简单,如下:
1.下载winutils的windows版本
GitHub上,有人提供了winutils的windows的版本,项目地址是: https://github.com/srccodes/hadoop-common-2.2.0-bin ,直接下载此项目的zip包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录
2.配置环境变量
增加用户变量HADOOP_HOME,值是下载的zip包解压的目录,然后在系统变量path里增加$HADOOP_HOME\bin 即可。
再次运行程序,正常执行。
Spark报错:Failed to locate the winutils binary in the hadoop binary path的更多相关文章
- WIN7下运行hadoop程序报:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报Failed to ...
- 解决spark运行中failed to locate the winutils binary in the hadoop binary path的问题
1.下载hadoop-common-2.2.0-bin并解压到某个目录 https://github.com/srccodes/hadoop-common-2.2.0-bin 2.设置hadoop.h ...
- Windows本地运行调试Spark或Hadoop程序失败:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
报错内容 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOExce ...
- Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
问题: windows开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary ...
- Windows7系统运行hadoop报Failed to locate the winutils binary in the hadoop binary path错误
程序运行的过程中,报Failed to locate the winutils binary in the hadoop binary path Java.io.IOException: Could ...
- Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...
- windows本地调试安装hadoop(idea) : ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
1,本地安装hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 下载hadoop对应版本 (我本意是想下载hadoop ...
- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
文章发自:http://www.cnblogs.com/hark0623/p/4170172.html 转发请注明 14/12/17 19:18:53 ERROR Shell: Failed to ...
- spark报错处理
Spark报错处理 1.问题:org.apache.spark.SparkException: Exception thrown in awaitResult 分析:出现这个情况的原因是spark启动 ...
随机推荐
- nyoj1273 河南省第九届省赛_"宣传墙"、状压DP+矩阵幂加速
宣传墙 时间限制:1000 ms | 内存限制:65535 KB 难度:4 描述 ALPHA 小镇风景美丽,道路整齐,干净,到此旅游的游客特别多.CBA 镇长准备在一条道路南 面 4*N 的墙上做 ...
- VM虚拟机上在NAT模式下设置静态IP的做法
1.问题:由于业务需要,个人笔记本电脑上用Vmware安装了3台Ubuntu虚拟机,现要求pc机连入局域网后,四台机器(3台ubuntu虚拟机+1台宿主机)能上网,并且,虚拟机要使用某一网段的固定IP ...
- centos6.9使用yum安装mysql(简单粗暴,亲测有效)
第1步.yum安装mysql[root@stonex ~]# yum -y install mysql-server安装结果:Installed: mysql-server.x86_64 0: ...
- Toncat-OpenSSL双向认证配置(iOS)
OpenSSL生成证书 要生成证书的目录下建立几个文件和文件夹,有./demoCA/ ./demoCA/newcerts/ ./demoCA/private/ ./demoCA/index.txt ( ...
- CMake与Make
大家都知道,写程序大体步骤为: 1.用编辑器编写源代码,如.c文件. 2.用编译器编译代码生成目标文件,如.o. 3.用链接器连接目标代码生成可执行文件,如.exe. 但如果源文件太多,一个一个编译时 ...
- SendMessage wMsg常量值参考
namespace Core.WinAPI { /// <summary> /// wMsg参数常量值: /// </summary> public static class ...
- query更多的筛选用法
比较操作符$eq : = 写法:db.class0.find({age:{$eq:18}},{name:1,_id:0}); 查找年龄等于18$lt :<$lte : <=$gt : & ...
- 【转】Ubuntu13.04配置:Vim+Syntastic+Vundle+YouCompleteMe
原文网址:http://www.cnblogs.com/csuftzzk/p/3435710.html 序言 使用Ubuntu和vim已经有一段时间了,对于Vim下的插件应用,我总是抱着一股狂热的态度 ...
- git push 免密码
git push 免密码 通用情况 使用ssh协议 git add 使用tab键自动补全的中文文件名乱码 jupyter notebook 创建密码 git push 免密码 通用情况 1.使用文件创 ...
- CentOS 6.5 下搭建FastDFS服务
参考网站: http://www.open-open.com/lib/view/open1435468300700.html http://blog.csdn.net/lynnlovemin/arti ...