hadoop综合问题记录

1、hdfs无法启动

错误:

2018-04-25 14:36:09,293 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://master:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From master/192.168.12.101 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
safemode: Call From master/192.168.12.101 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

解决:

# su hdfs   //切换到hdfs用户,若关闭了hdfs的权限则可略过这一步
# hadoop dfsadmin -safemode leave   //让namenode离开安全模式

2、oozie启动失败

错误:

"Column name pattern can not be NULL or empty"

解决: 网上有两种处理方式,一种是将mysql-connector-java版本更换成5.1.38版本或者更低的版本,这种方式我处理还是失败了,我采用的是第二种方式,配置oozie的Database URL为

jdbc:mysql://master/oozie?nullNamePatternMatchesAll=true

3、hive点击Hive View 2.0 Go to View出现USER HOME CHECK 的错误

错误:

Service 'userhome' check failed:
java.io.FileNotFoundException: File does not exist: /user/admin
at sun.reflect.GeneratedConstructorAccessor226.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

解决:

[root@master ~]# sudo su - hdfs
[root@master ~]# hdfs dfs -mkdir /user/admin
[root@master ~]# hdfs dfs -chown admin:hdfs /user/admin

4、hive的MetaStore启动了以后又自动停止了

错误:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:552)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
... 14 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

解决:hive.metastore.uris尝试连接9083端口,netstat查看该端口确实没有被监听,第一反应是hiveserver没有正常启动。查看hiveserver进程却存在,只是监听10000端口。 查看hive-site.xml配置,hive客户端连接9083端口,而hiveserver默认监听10000,找到问题根源了 ,但是我修改成10000还是不行,查看配置,还有一个参数是hive.server2.thrift.http.port,值是10001,我配置成了10001端口,然后就成功了。后面不能成功的查询,问题出现在了hive的源配置mysql,后面重新安装了新的mysql,就成功了。

5、centos7安装PyGreSQL

错误:

sh: pg_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 88, in <module>
pg_version = pg_version()
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 82, in pg_version
match = re.search(r'(\d+)\.(\d+)', pg_config('version'))
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 74, in pg_config
raise Exception("pg_config tool is not available.")
Exception: pg_config tool is not available.

  解决:安装 postgresql-devel 命令:yum -y install postgresql-devel

hadoop运维问题记录的更多相关文章

  1. Hadoop运维记录系列

    http://slaytanic.blog.51cto.com/2057708/1038676 Hadoop运维记录系列(一) Hadoop运维记录系列(二) Hadoop运维记录系列(三) Hado ...

  2. CM记录-Hadoop运维管理

    1.自动化运维--cloudera manager 2.手动运维 a.启动./start.all.sh,停止./stop-all.sh b.hdfs启动/停止:./start-dfs.sh   ./s ...

  3. Hadoop运维

    简单记录几个hdfs的运维命令 //查看hdfs的状态,是否有missing block,corrupt block等,也可以看datanode的状态 hdfs dfsadmin -report // ...

  4. Hadoop运维操作

    1.      处理hadoop的namenode宕机 处理措施: 进入hadoop的bin目录,重启namenode服务 操作命令: cd path/to/hadoop/bin ./hadoop-d ...

  5. Hadoop运维手记

    1.处理hadoop的namenode宕机 处理措施:进入hadoop的bin目录,重启namenode服务 操作命令:cd path/to/hadoop/bin ./hadoop-daemon.sh ...

  6. hadoop运维经验

    0.优化:http://dongxicheng.org/mapreduce/hadoop-optimization-0/ http://dongxicheng.org/mapreduce/hadoop ...

  7. Hadoop大数据学习视频教程 大数据hadoop运维之hadoop快速入门视频课程

    Hadoop是一个能够对大量数据进行分布式处理的软件框架. Hadoop 以一种可靠.高效.可伸缩的方式进行数据处理适用人群有一定Java基础的学生或工作者课程简介 Hadoop是一个能够对大量数据进 ...

  8. nginx日志相关运维操作记录

    在分析服务器运行情况和业务数据时,nginx日志是非常可靠的数据来源,而掌握常用的nginx日志分析命令的应用技巧则有着事半功倍的作用,可以快速进行定位和统计. 1)Nginx日志的标准格式(可参考: ...

  9. 针对Nginx日志的相关运维操作记录

    在分析服务器运行情况和业务数据时,nginx日志是非常可靠的数据来源,而掌握常用的nginx日志分析命令的应用技巧则有着事半功倍的作用,可以快速进行定位和统计. 1)Nginx日志的标准格式(可参考: ...

随机推荐

  1. C++11中的继承构造函数

    时间:2014.06.19 地点:基地 ------------------------------------------------------------------------- 一.问题描写 ...

  2. Vim-复制选中内容至系统剪贴板,光标移动到指定行的行首和行尾

    1.全选并复制到系统剪贴板 ggVG或ggvG 然后 "+y gg 让光标移到首行,在vim才有效,vi中无效 V 是进入Visual(可视)模式 G 光标移到最后一行 "+y 复 ...

  3. JAVA List 分割

    按指定大小,分隔集合,将集合按规定个数分为n个部分 import java.util.ArrayList; import java.util.Collections; import java.util ...

  4. 常见linux内核线程说明

    ps进程名有方括号的是内核级的进程,执行辅助功能(比如将缓存写入到磁盘):所有其他进程都是使用者进程.您会注意到,就算是在您新安装的(最小化的)系统中,也会有很多进程在运行. 在文档kernel-pe ...

  5. C++基本功之Operator

    废话不多说,这次讲的是 Operator overload. 关于operator, 在 < The C++ Programing Language > 里的描述,可以用做overload ...

  6. asp.net 正在加载/处理(兼容IE Chrome)

    正在加载分两种 一种是页面初始化效果,也就是数据读取前,这个时候需要用ajax先显示读取中数据,当ajax回调的时候,读取页面数据 一种是界面读取中效果,就是用div显示,当OnPreRenderCo ...

  7. keepalive脑裂的处理,从节点发现访问的虚拟IP就报警,同时尝试发送内容到主节点服务器关闭keepalive和nginx,或者关机

    解决keepalived脑裂问题   检测思路:正常情况下keepalived的VIP地址是在主节点上的,如果在从节点发现了VIP,就设置报警信息 脚本如下: 1 2 3 4 5 6 7 8 9 10 ...

  8. VS2013 连接 MySQL

    1.安装必须的工具: mysql-connector-net-6.8.3 mysql-installer-community-5.6.16.0.msi mysql-for-visualstudio-1 ...

  9. ASP.NET MVC4 权限验证

    在ASP.NET MVC4 中继承ActionFilterAttribute 类,重写OnActionExecuting方法 /// <summary> /// 权限拦截 /// < ...

  10. jquery mobile小经验

    现在网站上关于jquery mobile的demo和帖子可真少啊,我刚开始接触,遇到了一些问题,都找不到人请教. 这是我的个人经验总结,或多或少会对刚入门的童鞋有点帮助吧. 如果想一开始进入页面的时候 ...