Hive执行count函数失败

1.现象：

  1. 0: jdbc:hive2://192.168.137.12:10000> select count(*) from emp;
  2. INFO  : Number of reduce tasks determined at compile time: 1
  3. INFO  : In order to change the average load for a reducer (in bytes):
  4. INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
  5. INFO  : In order to limit the maximum number of reducers:
  6. INFO  :   set hive.exec.reducers.max=<number>
  7. INFO  : In order to set a constant number of reducers:
  8. INFO  :   set mapreduce.job.reduces=<number>
  9. ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------
  10.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
  11.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
  12.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
  13.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
  14.         at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)
  15.         at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
  16.         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)
  17.         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)
  18.         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
  19.         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  20.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
  21.         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
  22.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
  23.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
  24.         at java.security.AccessController.doPrivileged(Native Method)
  25.         at javax.security.auth.Subject.doAs(Subject.java:422)
  26.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  27.         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
  28. )'
  29. org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------
  30.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
  31.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
  32.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
  33.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
  34.         at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)
  35.         at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
  36.         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)
  37.         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)
  38.         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
  39.         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  40.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
  41.         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
  42.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
  43.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
  44.         at java.security.AccessController.doPrivileged(Native Method)
  45.         at javax.security.auth.Subject.doAs(Subject.java:422)
  46.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  47.         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
  49.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  50.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  51.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  52.         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  53.         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
  54.         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
  55.         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
  56.         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
  57.         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
  58.         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  59.         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
  60.         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
  61.         at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
  62.         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)
  63.         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
  64.         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
  65.         at java.security.AccessController.doPrivileged(Native Method)
  66.         at javax.security.auth.Subject.doAs(Subject.java:422)
  67.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  68.         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
  69.         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
  70.         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
  71.         at java.security.AccessController.doPrivileged(Native Method)
  72.         at javax.security.auth.Subject.doAs(Subject.java:422)
  73.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  74.         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
  75.         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
  76.         at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
  77.         at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
  78.         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
  79.         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
  80.         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
  81.         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
  82.         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
  83.         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
  84.         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
  85.         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
  86.         at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
  87.         at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
  88.         at java.security.AccessController.doPrivileged(Native Method)
  89.         at javax.security.auth.Subject.doAs(Subject.java:422)
  90.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  91.         at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
  92.         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  93.         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  94.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  95.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  96.         at java.lang.Thread.run(Thread.java:748)
  97. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------
  98.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
  99.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
  100.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
  101.         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
  102.         at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)
  103.         at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
  104.         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)
  105.         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)
  106.         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
  107.         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  108.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
  109.         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
  110.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
  111.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
  112.         at java.security.AccessController.doPrivileged(Native Method)
  113.         at javax.security.auth.Subject.doAs(Subject.java:422)
  114.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  115.         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
  117.         at org.apache.hadoop.ipc.Client.call(Client.java:1475)
  118.         at org.apache.hadoop.ipc.Client.call(Client.java:1412)
  119.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
  120.         at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
  121.         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
  122.         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
  123.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  124.         at java.lang.reflect.Method.invoke(Method.java:498)
  125.         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
  126.         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
  127.         at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)
  128.         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
  129.         ... 41 more
  131. Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)
  132. 0: jdbc:hive2://192.168.137.12:10000>

2.问题分析：

根据日志：

  1. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------

​ 显示对于“/tmp/hadoop-yarn/staging/anonymous/.staging”目录的权限为700，匿名用户没有访问权限，那就给个权限吧！这个目录是用来存放job信息的。

3.解决方法：为“/tmp/hadoop-yarn/staging/anonymous/.staging”赋予读写权限

  1.  hadoop fs -chmod -R 777 /tmp/

参考链接：

https://www.cnblogs.com/zqyw/p/10524103.html

Hive执行count函数失败，Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException)的更多相关文章

  1. Hive JDBC:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate anonymous

    今天使用JDBC来操作Hive时,首先启动了hive远程服务模式:hiveserver2 &(表示后台运行),然后到eclipse中运行程序时出现错误: java.sql.SQLExcepti ...

  2. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):

    用windows连接hadoop集群执行mapreduce任务的时候出现以下错误: org.apache.hadoop.security.AccessControlException:Permissi ...

  3. 异常-Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdfs, access=WRITE, inode="/hbase":root:supergroup:drwxr-xr-x

    1 详细异常 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlExce ...

  4. 你遇到了吗？Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException)

    我在使用 Structured Streaming 的 ForeachWriter,写 HDFS 文件时,出现了这个异常 这个异常出现的原因是HDFS作为一个分布式文件系统,支持多线程读,但是不支持多 ...

  5. hive运行query语句时提示错误：org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:

    hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...

  6. Hive执行过程中出现Caused by ： java.lang.ClassNotFoundException: org.cloudera.htrace.Trace的错误解决办法（图文详解）

    不多说,直接上干货! 问题详情 如下 这个错误的意思是缺少 htrace-core-2.04.jar. 解决办法: 将$HBASE_HOME/lib下的htrace-core-2.04.jar拷贝到$ ...

  7. hive执行结果moveTask操作失败

    hive执行结果moveTask操作失败 Apache Hive 2.1.0 ,在执行"INSERT OVERWRITE TABLE ...... select "或者 " ...

  8. [PHP源码阅读]count函数

    在PHP编程中,在遍历数组的时候经常需要先计算数组的长度作为循环结束的判断条件,而在PHP里面对数组的操作是很频繁的,因此count也算是一个常用函数,下面研究一下count函数的具体实现. 我在gi ...

  9. 在MySQL的InnoDB存储引擎中count(*)函数的优化

    写这篇文章之前已经看过了很多数据库方面的优化内容,大部分都是加索引.使用事务.要什么select什么等等.然而,只是停留在阅读的层面上,很少有实践,因为没有遇到真实的项目,一切都是纸上谈兵.实践是检验 ...

随机推荐

  1. Presto 函数开发

    0. 写在前面 Presto Functions 并不能像 Hive UDF 一样动态加载,需要根据 Function 的类型,实现 Presto 内部定义的不同接口,在 Presto 服务启动时进行 ...

  2. .Net Core 常见错误解决记录

    Error: String or binary data would be truncated. The statement has been terminated 数据库出错原因: 表字段创建的太短 ...

  3. 《Python金融大数据分析》高清PDF版|百度网盘免费下载|Python数据分析

    <Python金融大数据分析>高清PDF版|百度网盘免费下载|Python数据分析 提取码:mfku 内容简介 唯一一本详细讲解使用Python分析处理金融大数据的专业图书:金融应用开发领 ...

  4. Oracle数据库出现[23000][2291] ORA-02291: integrity constraint (SIMTH.SYS_C005306) violated异常

    参考链接 这个异常发生在往中间表中插入数据时,这时出现异常是因为关联的某个表没有插入数据,所以给没有插入数据的关联表插入数据,再给中间表插入数据此时异常就会解决.

  5. Maven中出现Could not find artifact ...:pom:0.0.1-SNAPSHOT

    参考链接:https://blog.csdn.net/zpwggi123/article/details/87189959 多模块项目构建时,先将总项目install,之后子项目分别install,注 ...

  6. 社交网站的数据挖掘与分析pdf版本|网盘下载地址附提取码|

    点击此处进入网盘下载地址 提取码:btqx 作者介绍: 马修·罗塞尔(MatthewA.Russell),DigitalReasoningSystems公司的技术副总裁和Zaffra公司的负责人,是热 ...

  7. PHP highlight_file() 函数

    实例 对测试文件("test.php")进行 PHP 语法高亮显示: <html><body><?phphighlight_file("te ...

  8. PDOStatement::fetchColumn

    PDOStatement::fetchColumn — 从结果集中的下一行返回单独的一列.(PHP 5 >= 5.1.0, PECL pdo >= 0.9.0)高佣联盟 www.cgewa ...

  9. Spring学习总结(7)-AOP

    参考资料:https://docs.spring.io/spring-framework/docs/current/spring-framework-reference/core.html#aop 1 ...

  10. 实用！一键生成数据库文档，堪称数据库界的Swagger

    本文收录在个人博客:www.chengxy-nds.top,技术资料共享,同进步 最近部门订单业务调整,收拢其他业务线的下单入口,做个统一大订单平台.需要梳理各业务线的数据表,但每个业务线库都有近百张 ...