本文继成上一篇通过hive分析nginx日志文章,详情参考下面链接:

http://www.cnblogs.com/wcwen1990/p/7066230.html

接着来:

创建业务子表:

drop table if exists chavin.nginx_access_log_comm;
create table if not exists chavin.nginx_access_log_comm(
host STRING,
time STRING,
request STRING,
referer STRING
)
row format delimited fields terminated by '\t'
STORED AS orc tblproperties("orc.compress"="SNAPPY");
insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;

结果报错如下:

hive (default)> insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;
Query ID = root_20170623203838_35656104-9f46-4801-9dbd-1d5685de8187
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1498218985196_0009, Tracking URL = http://db01:8088/proxy/application_1498218985196_0009/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0009
Hadoop job information for Stage-1: number of mappers: 4; number of reducers: 0
2017-06-23 20:38:10,702 Stage-1 map = 0%,  reduce = 0%
2017-06-23 20:38:32,584 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1498218985196_0009 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1498218985196_0009_m_000002 (and more) from job job_1498218985196_0009

Task with the most failures(4):
-----
Task ID:
   task_1498218985196_0009_m_000002

URL:
   http://db01:8088/taskdetails.jsp?jobid=job_1498218985196_0009&tipid=task_1498218985196_0009_m_000002
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:415)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
     ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
     ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
     ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)
     ... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)
     at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)
     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
     ... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
     at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:140)
     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:297)
     ... 24 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 4   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive (default)>

从日志看,貌似执行mr因为缺少包而导致的故障。
解决办法:向hive命令行中添加hive-contrib-1.1.0-cdh5.9.2.jar包:

hive (chavin)> add jar /opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar;
Added [/opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar] to class path
Added resources: [/opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar]

再次运行加载数据命令:

hive (chavin)>insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;
Query ID = root_20170623203333_d244452b-9fae-4131-826a-428620219dbe
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1498218985196_0007, Tracking URL = http://db01:8088/proxy/application_1498218985196_0007/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0007
Hadoop job information for Stage-1: number of mappers: 4; number of reducers: 0
2017-06-23 20:33:59,442 Stage-1 map = 0%,  reduce = 0%
2017-06-23 20:34:19,200 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 50.65 sec
2017-06-23 20:34:20,239 Stage-1 map = 25%,  reduce = 0%, Cumulative CPU 53.72 sec
2017-06-23 20:34:21,333 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 56.75 sec
2017-06-23 20:34:22,368 Stage-1 map = 55%,  reduce = 0%, Cumulative CPU 62.82 sec
2017-06-23 20:34:24,451 Stage-1 map = 63%,  reduce = 0%, Cumulative CPU 68.45 sec
2017-06-23 20:34:28,984 Stage-1 map = 88%,  reduce = 0%, Cumulative CPU 88.58 sec
2017-06-23 20:34:31,049 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 91.97 sec
MapReduce Total cumulative CPU time: 1 minutes 31 seconds 970 msec
Ended Job = job_1498218985196_0007
Stage-4 is filtered out by condition resolver.
Stage-3 is selected by condition resolver.
Stage-5 is filtered out by condition resolver.
Starting Job = job_1498218985196_0008, Tracking URL = http://db01:8088/proxy/application_1498218985196_0008/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0008
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
2017-06-23 20:34:38,770 Stage-3 map = 0%,  reduce = 0%
2017-06-23 20:34:44,973 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.46 sec
MapReduce Total cumulative CPU time: 1 seconds 460 msec
Ended Job = job_1498218985196_0008
Loading data to table chavin.nginx_access_log_comm
Table chavin.nginx_access_log_comm stats: [numFiles=1, numRows=2538166, totalSize=6856514, rawDataSize=1598858202]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 4   Cumulative CPU: 91.97 sec   HDFS Read: 1000717778 HDFS Write: 6859275 SUCCESS
Stage-Stage-3: Map: 1   Cumulative CPU: 1.46 sec   HDFS Read: 6989009 HDFS Write: 6856514 SUCCESS
Total MapReduce CPU Time Spent: 1 minutes 33 seconds 430 msec
OK
host    time    request    referer
Time taken: 55.218 seconds

问题解决,但是该解决方法只能对本次Hive会话有用,Hive使用命令exit退出后再进入依旧会出现该问题。

hive中创建子表并插入数据过程初始化MR报错解决方法的更多相关文章

  1. hive导出数据到本地文件报错解决方法

    报错信息: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move so ...

  2. NLTK数据包下载频繁报错——解决方法

    问题描述:Nltk下载数据包,安装频繁报错. import nltk nltk.download() 运行上面的代码,下载nltk的数据包.但是在下载过程中因为各种问题导致网络传输失败,下载不成功. ...

  3. JDBC插入数据时中文变为问号的解决方法

    JDBC插入数据时中文变为问号的解决方法 制作人:全心全意 出现中文变问号的代码: String url = "jdbc:mysql://localhost:3306/test"; ...

  4. ThinkPHP查询数据的时候toArray()报错解决办法

    当查找不到数据时toArray()会报错,如图 解决办法:先查找数据,然后加个判断,如果有数据再转化为数组,如果没有数据就给个空值,不想代码继续往下执行就return false;

  5. hive创建表带中文注释报错解决方法

    hive创建带有中文注释的表报错: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask ...

  6. C#中往数据库插入空值报错解决方法

    C#中的NUll于SQL中的null是不一样的, SQL中的null用C#表示出来就是DBNull.Value 在用C#往数据库里面插入记录的时候, 可能有的字段你不赋值,那么这个字段的值就为null ...

  7. IDEA中创建父子工程与maven打包Springboot聚合工程报错程序包不存在问题处理

    公司新项目需使用java技术栈,便使用IDEA搭建了一个多SpringBoot项目的聚合工程,因为初次使用,遇到了很多问题,maven打包时各种报错,在网上查了好多终于解决了,为巩固记忆,特作此记录. ...

  8. FrameWork模型中引入宏函数报错解决方法

    如下图在Framework的一个简单维度中加入宏函数 解决办法如下图 step1: step2: PS :Cognos 10.1.1中 在cognos connection中创建数据源,为什么没有od ...

  9. .vue文件在webstorm中es6语法报错解决方法

    1 语法支持es6设置 Preferences > Languages & Frameworks > JavaScript 把 Javascript Language versio ...

随机推荐

  1. Spark2.3 HA集群的分布式安装

    一.下载Spark安装包 1.从官网下载 http://spark.apache.org/downloads.html 2.从微软的镜像站下载 http://mirrors.hust.edu.cn/a ...

  2. [svc]arp协议的细枝末节

    tcpdump抓取arp包 tcpdump -nn -i calif24874aae57 -e arp数据包格式 操作字段指出四种操作类型: arp请求 1 arp应答 2 rarp请求 3 rarp ...

  3. Selenium Web 自动化 - 如何找到元素

    Selenium Web 自动化 - 如何找到元素 2016-07-29 1. 什么是元素? 元素:http://www.w3school.com.cn/html/html_elements.asp ...

  4. LeetCode: Best Time to Buy and Sell Stock III 解题报告

    Best Time to Buy and Sell Stock IIIQuestion SolutionSay you have an array for which the ith element ...

  5. sklearn linear_model,svm,tree,naive bayes,ensemble

    sklearn linear_model,svm,tree,naive bayes,ensemble by iris dataset .caret, .dropup > .btn > .c ...

  6. MinGW 使用 mintty 终端替代默认终端以解决界面上复制与粘贴的问题

    使用了一段时间的 cygwin,挺开心的,又尝试了下同类工具 Msys + MinGW,安装好之后发现它居然使用默认的 cmd 作为终端,界面输出内容的复制与粘贴极其不便,我记得 Cygwin 使用的 ...

  7. CFA一级知识点总结

    更多来自:   www.vipcoursea.com   Ethics 部分 Objective of codes and standard:永远是为了maintain public trust in ...

  8. html5中的input和label写法与取值

    demo 效果图如上 label是html5特有的,是定义 input 元素的标注.凡是input前面要有个label标识下,label和input真是一对好兄弟啊.<label> 标签的 ...

  9. linux dns 工具包 -- bind-utils

    https://www.cnblogs.com/274914765qq/p/4817941.html

  10. vue分页组件

    <template> <div> <ul class="pagination"> <li @click="goTo(1)&quo ...