hive> select product_id, track_time from trackinfo limit 5;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobInProgress.checkTaskLimits(JobInProgress.java:509)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:485)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
... 10 more at org.apache.hadoop.ipc.Client.call(Client.java:1066)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at org.apache.hadoop.mapred.$Proxy11.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(java.io.IOException: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobInProgress.checkTaskLimits(JobInProgress.java:509)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:485)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
... 10 more
)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask



错误原因:
因为trackinfo表的数据太庞大,并且写的sql语句
select product_id, track_time from trackinfo limit 5
性能耗损太大了,将会导致调度的Map数太多,超出了job的限制。

来看看hive中trackinfo表的数据量有多大:
-bash-3.2$ hadoop fs -dus /data/share/trackinfo
Warning: $HADOOP_HOME is deprecated. hdfs://yhd-hadoop06.int.yihaodian.com:9000/data/share/trackinfo 19387740988708
能够看到,trackinfo的数据量大概有19TB这么大

改写sql语句,指定where条件,性能得到了提升
select product_id, track_time from trackinfo where ds='2014-5-13' limit 5
问题同一时候也被攻克了

hive运行query语句时提示错误:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:的更多相关文章

  1. Hive JDBC:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate anonymous

    今天使用JDBC来操作Hive时,首先启动了hive远程服务模式:hiveserver2 &(表示后台运行),然后到eclipse中运行程序时出现错误: java.sql.SQLExcepti ...

  2. 运行基准测试hadoop集群中的问题:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /benchmarks/TestDFSIO/io_data/test_

    在master(即:host2)中执行 hadoop jar hadoop-test-1.1.2.jar DFSCIOTest -write -nrFiles 12 -fileSize 10240 - ...

  3. Hive执行count函数失败,Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException)

    Hive执行count函数失败 1.现象: 0: jdbc:hive2://192.168.137.12:10000> select count(*) from emp; INFO : Numb ...

  4. hadoop拷贝文件时 org.apache.hadoop.ipc.RemoteException异常的解决

    1.系统或hdfs是否有空间 2.datanode数是否正常 3.是否在safemode 4.防火墙关闭 5.配置方面 6.把NameNode的tmp文件清空,然后重新格式化NameNode

  5. 在Ubuntu 12.04 - 64bit中安装CodeSourcery时提示错误

    安装时提示错误,Your 64-bit Linux host is missing the 32-bit libraries requied to install and use Sourcery C ...

  6. Oracle登录时提示错误,导致用户无法登录

    Oracle登录时提示错误,导致用户无法登录,错误如下 ------------------------------------------------------------------------ ...

  7. Ubuntu 使用apt-get时提示错误:无法获得锁 /var/lib/dpkg/lock

    推荐博客:http://blog.sina.com.cn/s/blog_5c1450a8010188ju.html Ubuntu 使用apt-get时提示错误:无法获得锁 /var/lib/dpkg/ ...

  8. 在执行context.getContentResolver.query()方法时出现错误。

    1. 在执行context.getContentResolver.query()方法时出现错误. 07-15 18:46:13.470: E/AndroidRuntime(13624): FATAL ...

  9. 使用图形界面管理工具Navicat for MySQL连接Mysql数据库时提示错误:Can't connect to MySQL server (10060)

    版权声明:本文为 testcs_dn(微wx笑) 原创文章,非商用自由转载-保持署名-注明出处,谢谢. https://blog.csdn.net/testcs_dn/article/details/ ...

随机推荐

  1. C# 禁止 Webbrowser 控件的弹出脚本错误对话框

    当IE浏览器遇到脚本错误时浏览器,左下 角会出现一个黄色图标,点击可以查看脚本错误的详细信息,并不会有弹出的错误信息框.当我们使用 WebBrowser控件时有错误信息框弹出,这样程序显的很不友好,而 ...

  2. String、StringBuffer和StringBuilder——个人学习

    1.首先说一下他们的名称区别: String——字符串常量,StringBuffer——字符串变量(线程安全),StringBuilder——字符串变量(非线程安全) 从名称就可以很明显的看出他们的基 ...

  3. 根据http协议传送数据

    发送的内容: [50 4f 53 54 20 2f 64 65 78 2f 66 69 72 65 70 6f 77 65 72 20 48 54 54 50 2f 31 2e 31 0d 0a 43 ...

  4. Ireport 报表导出 Poi + ireport 导出pdf, word ,excel ,htm

    Ireport 报表导出 Poi + ireport 导出pdf, doc ,excel ,html 格式 下面是报表导出工具类reportExportUtils 需要导出以上格式的报表 只需要调用本 ...

  5. html5自定义标签选择器

    * E[attr]:只使用属性名,但没有确定任何属性值 * E[attr="value"]:指定属性名,并指定了该属性的属性值.必须和元素的属性完全匹配 * E[attr~=&qu ...

  6. django作models的UPDATE时,注意有过滤外键的情况

    就是如果要过滤的时候,存在外键,则需要取到原始字段的名称,不然就是ID. 原始名称以__(双下划线)引用. 如: def path2db(release_version, develop_versio ...

  7. 用正则式判断URL是否合法-python

    import sys import re #Make sure we have a single URL argument. if len(sys.argv) != 2: print (sys.std ...

  8. HDU 4430 Yukari's Birthday (二分+枚举)

    题意:给定一个n(18 ≤ n ≤ 10^12),一个等比数列k + k^2 + .......+ k^r = n 或者 = n-1,求出最小的k*r,如果最小的不唯一,则取r更小的 分析:两个未知数 ...

  9. wcf资料

    WCF服务安全控制之netTcpBinding的用户名密码验证http://www.cnblogs.com/wengyuli/archive/2011/05/14/wcf-nettcpbinding- ...

  10. google font和@font-face

    会使用google字体 网址: http://www.google.com/fonts/ 选择字体, quick use 引用css: <link href='http://fonts.goog ...