平时一般是在windows环境下进行开发,在windows 环境下操作hbase可能会出现异常(java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.),以前也遇到过这个问题,今天又有小伙伴遇到这个问题,就顺带记一笔,异常信息如下:

2016-05-23 17:02:13,551 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-23 17:02:13,611 ERROR [org.apache.hadoop.util.Shell] - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:265)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1710)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:806)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:633)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:387)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:282)
at net.shgaoxin.db.hbase.HbaseConnectionFactory.createResource(HbaseConnectionFactory.java:67)
at net.shgaoxin.db.hbase.HbaseConnectionFactory.makeObject(HbaseConnectionFactory.java:40)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at net.shgaoxin.base.AbstractPooledContainer.get(AbstractPooledContainer.java:49)
at net.shgaoxin.db.hbase.HbaseConnectionContainer.getConnection(HbaseConnectionContainer.java:46)
at net.shgaoxin.db.hbase.HbaseConnectionContainer.getConnection(HbaseConnectionContainer.java:14)
at net.shgaoxin.db.hbase.HbaseTemplate.scan(HbaseTemplate.java:398)
at net.shgaoxin.impl.dao.hbase.GenericDaoHbaseImpl.scan(GenericDaoHbaseImpl.java:73)
at net.shgaoxin.impl.service.eastdayminisitesp.ImgUploadServiceImpl.getCurrentStepRowkeys(ImgUploadServiceImpl.java:260)
at net.shgaoxin.impl.context.eastdayminisitesp.AsyncImgUploadContextImpl.doOnStart(AsyncImgUploadContextImpl.java:81)
at net.shgaoxin.impl.context.AbstractProcessQueue.start(AbstractProcessQueue.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

查看hadoop源码:

  /** fully qualify the path to a binary that should be in a known hadoop
* bin location. This is primarily useful for disambiguating call-outs
* to executable sub-components of Hadoop to avoid clashes with other
* executables that may be in the path. Caveat: this call doesn't
* just format the path to the bin directory. It also checks for file
* existence of the composed path. The output of this call should be
* cached by callers.
* */
public static final String getQualifiedBinPath(String executable)
throws IOException {
// construct hadoop bin path to the specified executable
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"
+ File.separator + executable; File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName
+ " in the Hadoop binaries.");
} return exeFile.getCanonicalPath();
}
  private static String HADOOP_HOME_DIR = checkHadoopHome();

 /** Centralized logic to discover and validate the sanity of the Hadoop
* home directory. Returns either NULL or a directory that exists and
* was specified via either -Dhadoop.home.dir or the HADOOP_HOME ENV
* variable. This does a lot of work so it should only be called
* privately for initialization once per process.
**/
private static String checkHadoopHome() { // first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
} try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
} if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
} // check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
} home = homedir.getCanonicalPath(); } catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
} return home;
}
 
 

结合异常不难发现HADOOP_HOME_DIR值为null,基本上可以判断是 HADOOP_HOME环境变量的问题,实际上本来就没有配置hadoop的环境变量,报错也是理所当然了。

配置windows环境变量需要有winutils.exe,有大神提供了一个windows下环境配置所需文件,可参考https://github.com/srccodes/hadoop-common-2.2.0-bin

windows 中使用hbase 异常:java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.的更多相关文章

  1. spark开发常见问题之一:java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    最近在学习研究pyspark机器学习算法,执行代码出现以下异常: 19/06/29 10:08:26 ERROR Shell: Failed to locate the winutils binary ...

  2. java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries

    在已经搭建好的集群环境Centos6.6+Hadoop2.7+Hbase0.98+Spark1.3.1下,在Win7系统Intellij开发工具中调试Spark读取Hbase.运行直接报错: ? 1 ...

  3. Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...

  4. Spark报错java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    Spark 读取 JSON 文件时运行报错 java.io.IOException: Could not locate executable null\bin\winutils.exe in the ...

  5. spark-local-运行异常-Could not locate executable null\bin\winutils.exe in the Hadoop binaries

    windows下-local模式-运行spark: 1.下载winutils的windows版本 GitHub上,有人提供了winutils的windows的版本,项目地址是:https://gith ...

  6. 安装spark 报错:java.io.IOException: Could not locate executable E:\hadoop-2.7.7\bin\winutils.exe

    打开 cmd 输入 spark-shell 虽然可以正常出现 spark 的标志符,但是报错:java.io.IOException: Could not locate executable E:\h ...

  7. hbase异常:java.io.IOException: Unable to determine ZooKeeper ensemble

    项目中用到hbase,有时候可能会报一些异常,比如java.io.IOException: Unable to determine ZooKeeper ensemble 等等,当出现这个问题时,根据个 ...

  8. Windows下运行MapReduce程序出现Could not locate executable null\winutils.exe in the Hadoop binaries.

    运行环境:windows10 64位,虚拟机:Ubuntu Kylin 14.04,Hadoop2.7.1 错误信息: java.io.IOException: Could not locate ex ...

  9. geotools导出shapefile出错: java.io.IOException: Current fid index is null, next must be called before write()

    geotools导出shapefile出错: java.io.IOException: Current fid index is null, next must be called before wr ...

随机推荐

  1. python ranndom模块及生成验证码

    python的random模块用于生成随机数,下面介绍一下random模块的常用方法: 取随机小数: 数学计算 random.random() 用于生成一个0-1的随机浮点数 0<=n<1 ...

  2. 我教你怎么玩转git

    我教你怎么玩转git 1.想要练习解决冲突? 很好办.创建本地分支,a,b, a上面,这样改.b上面那样改. 然后你就解决冲突就可以了. 可以merge 或者cheerypick 2.想要玩一个不要历 ...

  3. Jsoncpp 编译

    1. linux下编译jsoncpp 从(http://jsoncpp.sourceforge.net/)下载源码包“jsoncpp-src-0.5.0.tar.gz”,解压后在其解压后目录中运行 $ ...

  4. centso下如何解压RAR文件

    tar -xvf rarlinux-3.9.3.tar.gz cd rar   make 看见下面这些信息就是安装成功了 mkdir -p /usr/local/bin mkdir -p /usr/l ...

  5. 《Cracking the Coding Interview》——第16章:线程与锁——题目4

    2014-04-27 20:06 题目:设计一个类,只有在不产生死锁的时候才分配资源. 解法:不太清楚这个题是要分配何种资源,以何种形式?所以没能动手写个可运行的代码,只是闲扯了几句理论分析. 代码: ...

  6. Java基础-4变量与数据类型

    变量:变量是Java程序中的一个基本存储单元.变量是一个标识符.类型及一个可选初始值的组合定义.所有的变量都有一个作用域,即变量在某一区域有效. 基本的变量声明方式如下: int a; float b ...

  7. JavaWeb笔记(九)Ajax&Json

    AJAX 实现方式 原生的JS实现方式 //1.创建核心对象 var xmlhttp; if (window.XMLHttpRequest) {// code for IE7+, Firefox, C ...

  8. 软工实践 - 第十六次作业 Alpha 冲刺 (7/10)

    队名:起床一起肝活队 组长博客:https://www.cnblogs.com/dawnduck/p/10013959.html 作业博客:班级博客本次作业的链接 组员情况 组员1(队长):白晨曦 过 ...

  9. 团队Alpha版本冲刺(四)

    目录 组员情况 组员1(组长):胡绪佩 组员2:胡青元 组员3:庄卉 组员4:家灿 组员5:凯琳 组员6:丹丹 组员7:何家伟 组员8:政演 组员9:鸿杰 组员10:刘一好 组员:何宇恒 展示组内最新 ...

  10. poj 3436 网络流构图经典

    ACM Computer Factory Time Limit: 1000MS   Memory Limit: 65536K Total Submissions: 6012   Accepted: 2 ...