hadoop综合问题记录

1、hdfs无法启动

错误:

2018-04-25 14:36:09,293 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://master:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From master/192.168.12.101 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
safemode: Call From master/192.168.12.101 to master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

解决:

# su hdfs   //切换到hdfs用户,若关闭了hdfs的权限则可略过这一步
# hadoop dfsadmin -safemode leave   //让namenode离开安全模式

2、oozie启动失败

错误:

"Column name pattern can not be NULL or empty"

解决: 网上有两种处理方式,一种是将mysql-connector-java版本更换成5.1.38版本或者更低的版本,这种方式我处理还是失败了,我采用的是第二种方式,配置oozie的Database URL为

jdbc:mysql://master/oozie?nullNamePatternMatchesAll=true

3、hive点击Hive View 2.0 Go to View出现USER HOME CHECK 的错误

错误:

Service 'userhome' check failed:
java.io.FileNotFoundException: File does not exist: /user/admin
at sun.reflect.GeneratedConstructorAccessor226.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

解决:

[root@master ~]# sudo su - hdfs
[root@master ~]# hdfs dfs -mkdir /user/admin
[root@master ~]# hdfs dfs -chown admin:hdfs /user/admin

4、hive的MetaStore启动了以后又自动停止了

错误:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:552)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
... 14 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

解决:hive.metastore.uris尝试连接9083端口,netstat查看该端口确实没有被监听,第一反应是hiveserver没有正常启动。查看hiveserver进程却存在,只是监听10000端口。 查看hive-site.xml配置,hive客户端连接9083端口,而hiveserver默认监听10000,找到问题根源了 ,但是我修改成10000还是不行,查看配置,还有一个参数是hive.server2.thrift.http.port,值是10001,我配置成了10001端口,然后就成功了。后面不能成功的查询,问题出现在了hive的源配置mysql,后面重新安装了新的mysql,就成功了。

5、centos7安装PyGreSQL

错误:

sh: pg_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 88, in <module>
pg_version = pg_version()
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 82, in pg_version
match = re.search(r'(\d+)\.(\d+)', pg_config('version'))
File "/tmp/pip-build-MZLWmC/PyGreSQL/setup.py", line 74, in pg_config
raise Exception("pg_config tool is not available.")
Exception: pg_config tool is not available.

  解决:安装 postgresql-devel 命令:yum -y install postgresql-devel

hadoop运维问题记录的更多相关文章

  1. Hadoop运维记录系列

    http://slaytanic.blog.51cto.com/2057708/1038676 Hadoop运维记录系列(一) Hadoop运维记录系列(二) Hadoop运维记录系列(三) Hado ...

  2. CM记录-Hadoop运维管理

    1.自动化运维--cloudera manager 2.手动运维 a.启动./start.all.sh,停止./stop-all.sh b.hdfs启动/停止:./start-dfs.sh   ./s ...

  3. Hadoop运维

    简单记录几个hdfs的运维命令 //查看hdfs的状态,是否有missing block,corrupt block等,也可以看datanode的状态 hdfs dfsadmin -report // ...

  4. Hadoop运维操作

    1.      处理hadoop的namenode宕机 处理措施: 进入hadoop的bin目录,重启namenode服务 操作命令: cd path/to/hadoop/bin ./hadoop-d ...

  5. Hadoop运维手记

    1.处理hadoop的namenode宕机 处理措施:进入hadoop的bin目录,重启namenode服务 操作命令:cd path/to/hadoop/bin ./hadoop-daemon.sh ...

  6. hadoop运维经验

    0.优化:http://dongxicheng.org/mapreduce/hadoop-optimization-0/ http://dongxicheng.org/mapreduce/hadoop ...

  7. Hadoop大数据学习视频教程 大数据hadoop运维之hadoop快速入门视频课程

    Hadoop是一个能够对大量数据进行分布式处理的软件框架. Hadoop 以一种可靠.高效.可伸缩的方式进行数据处理适用人群有一定Java基础的学生或工作者课程简介 Hadoop是一个能够对大量数据进 ...

  8. nginx日志相关运维操作记录

    在分析服务器运行情况和业务数据时,nginx日志是非常可靠的数据来源,而掌握常用的nginx日志分析命令的应用技巧则有着事半功倍的作用,可以快速进行定位和统计. 1)Nginx日志的标准格式(可参考: ...

  9. 针对Nginx日志的相关运维操作记录

    在分析服务器运行情况和业务数据时,nginx日志是非常可靠的数据来源,而掌握常用的nginx日志分析命令的应用技巧则有着事半功倍的作用,可以快速进行定位和统计. 1)Nginx日志的标准格式(可参考: ...

随机推荐

  1. (3)FluidMoveBehavior 之模仿 Windows Phone 开始菜单的 Tile 长按后排序

    这个工程和上一篇 (2)中介绍的排序大同小异,只是比上一篇交换复杂一点,不是通过单击进行交换, 而是拖动一个 Tile 到另一个 Tile 上时,判断两个 Tile 的中心距离是否符合条件来判断是否进 ...

  2. saveFile()方法

    saveFile的原理就是将流写入到需要写入的文件,通过可以用“FileOutputStream”创建文件实例,之后过“OutputStreamWriter”流的形式进行存储,举例:public vo ...

  3. FreeRTOS 任务优先级分配方案

    任务优先级说明下面对 FreeRTOS 优先级相关的几个重要知识点进行下说明,这些知识点在以后的使用中务必要掌握牢固.  FreeRTOS 中任务的最高优先级是通过 FreeRTOSConfig.h ...

  4. 基于js全屏动画焦点图幻灯片

    今天给大家分享一款基于js全屏动画焦点图幻灯片.这款焦点图内的内容以动画形式出现和消失.效果图如下: 在线预览   源码下载 实现的代码. html代码: <div class="sl ...

  5. fontDialog-字体对话框和colorDialog-颜色对话框

    private void button1_Click(object sender, EventArgs e) { DialogResult dr = fontDialog1.ShowDialog(); ...

  6. 树莓派+android things+实时音视频传输demo之遥控小车

    做了个测试小车,上面安装了摄像头,通过外网进行视频传输: https://www.bilibili.com/video/av23817880/

  7. zend Studio10.6.2破解注册码

    下载破解包:http://download.csdn.net/detail/gsls200808/7982115 安装方法:破解的jar文件com.zend.verifier_10.6.2.v2014 ...

  8. (转自)视频流中的DTS/PTS到底是什么;

     翻译了一下: Q:hi,这可能是一个弱智问题,但是当我使用bbMEG1.24beta17编码时,一直以来总是遇到这个下溢的问题.我从日志文件中得到的唯一启示就是我应该更改mux率...但是帮助文档却 ...

  9. php -- 判断文件是否存在

    file_exists is_file is_dir 基本上,PHP的 file_exists = is_dir + is_file 写程序验证一下: 分别执行1000次,记录所需时间. ------ ...

  10. Java凝视模板

     设置凝视模板的入口: Window->Preference->Java->Code Style->Code Template 然后展开Comments节点就是全部需设置凝 ...