impala在一个select中执行多个count distinct时会报错,比如执行 select key, count(distinct column_a), count(distinct column_b) from test_table group by key 会报错 Query submitted at: 2019-09-28 00:34:20 (Coordinator: http://DataOne-001:25000)ERROR: AnalysisException: all D

老李分享:JVM调优   poptest是国内唯一一家培养测试开发工程师的培训机构,以学员能胜任自动化测试,性能测试,测试工具开发等工作为目标.如果对课程感兴趣,请大家咨询qq:908821478,咨询电话010-84505200.poptest 不断优化课堂的案例,培养学员实际解决问题的能力. 当Java程序申请内存,超出VM可分配内纯的时候,VM首先可能会GC,如果GC完还是不够,或者申请的直接超够VM可能有的,就会抛出内 存溢出异常.从VM规范中我们可以得到,一下几种异常. java.la

kudu写入压力大时报错 19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed118785e3(server002:7050) for tablet 4259921cdcca4776b37771659a8cafb3: Service unavailable: Soft memory limit exceeded (at 106.05% of capacity). See htt

oozie中通过shell执行impala,脚本如下: $ cat test_impala.sh #!/bin/sh /usr/bin/kinit -kt /tmp/impala.keytab impala/server04 /usr/bin/impala-shell -i server04:21000 -q 'show databases' 直接执行shell脚本正常,在oozie中执行报错: Traceback (most recent call last): File "/usr/lib/

cloudera manager安装hue后想开启访问mysql(librdbms)需要在这里配置(hue_safety_valve.ini) 添加配置如下 [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and o

hive及其他组件(比如spark.impala等)都会依赖hive metastore,依赖的配置文件位于hive-site.xml hive metastore重要配置 hive.metastore.warehouse.dirhive2及之前版本默认为/user/hive/warehouse/,创建数据库或表时会在该目录下创建对应的目录 javax.jdo.option.ConnectionURLjavax.jdo.option.ConnectionDriverNamejavax.jdo.o

常用格式 textfile 需要定义分隔符,占用空间大,读写效率最低,非常容易发生冲突(分隔符)的一种格式,基本上只有需要导入数据的时候才会使用,比如导入csv文件: ROW FORMAT DELIMITED FIELDS TERMINATED BY '\u0001' LINES TERMINATED BY '\n' STORED AS TEXTFILE json hive3.0后官方支持json格式,之前需要使用第三方,导入jar,http://www.congiu.net/hive-json

用户提交任务到yarn时有可能遇到下面的错误: 1) Requested user anything is not whitelisted and has id 980,which is below the minimum allowed 1000 这是因为yarn中配置min.user.id=1000,yarn认为id小于1000的是超级用户,yarn禁止超级用户提交任务: Each account must have a user ID that is greater than or equ

kerberos安装详见:https://www.cnblogs.com/barneywill/p/10394164.html 一 为CM创建用户 # kadmin.local -q "addprinc scm/admin" 名字和密码任意,后续配置中会使用 二 CM配置过程 1 启用Kerberos 2 全部选中 3 按照/etc/krb5.conf填写 4 5 填写刚才创建的用户名密码 6 下一步 7 下一步 8 可以查看cm创建的用户 # kadmin.local -q 'lis

kudu安装后运行不正常,master中找不到任何tserver,查看tserver日志发现有很多报错: Failed to heartbeat to master:7051: Invalid argument: Failed to ping master at master:7051: Client connection negotiation failed: client connection to master:7051: unable to find SASL plugin: PLAIN

一 对比 存储空间对比: 查询性能对比: 二 设计方案 将数据拆分为:历史数据(hdfs+parquet+snappy)+ 近期数据(kudu),可以兼具各种优点: 1)整体低于10%的磁盘占用: 2)更少的查询耗时: 3)近期数据实时更新: 4)近期数据可修改: 5)kudu集群重启时间降低90%: 6)impala并行scan:scan kudu + scan hdfs: 三 改造方案 利用视图 create view v_table asselect * from parquet_tabl

kudu的副本数量是在表上设置,可以通过命令查看 # sudo -u kudu kudu cluster ksck $master ... Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------------------------------------------+----+--------

从impala中创建kudu表之后,如果想从hive或spark sql直接读取,会报错: Caused by: java.lang.ClassNotFoundException: com.cloudera.kudu.hive.KuduStorageHandler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424

spark-2.4.2kudu-1.7.0 开始尝试 1)自己手工将jar加到classpath spark-2.4.2-bin-hadoop2.6+kudu-spark2_2.11-1.7.0-cdh5.16.1.jar # bin/spark-shell scala> val df = spark.read.options(Map("kudu.master" -> "master:7051", "kudu.table" ->

docker container启动失败,报错:Exited (137) *** ago,比如 Exited (137) 16 seconds ago 这时通过docker logs查不到任何日志,从mesos上看stderr相关的只有一句 I0409 16:56:26.408077 8583 executor.cpp:736] Container exited with status 137 通过docker inspect查看container状态为 "State": { &quo

hue中使用hue用户(hue admin)访问hdfs报错: Cannot access: /. Note: you are a Hue admin but not a HDFS superuser, "hdfs" or part of HDFS supergroup, "supergroup". 其他症状:oozie editor页面卡住不动 检查过程如下: 1 hdfs配置 hadoop.proxyuser.hue.hosts=*hadoop.proxyuse

yarn开启日志归集功能,除了配置之外 yarn.log-aggregation-enable=true 还要检查/tmp/logs目录是否存在以及权限,尤其是在开启kerberos之后,有些目录可能不能自动创建成功,需要手工创建: $ hdfs dfs -mkdir /tmp$ hdfs dfs -chmod 777 /tmp 每个应用的hdfs日志目录: /tmp/logs/$user/logs/$applicationId

提前安装好elk(elasticsearch.logstach.kibana) 一 启动logstash $LOGSTASH_HOME默认位于/usr/share/logstash或/opt/logstash 1 nginx日志使用默认格式 log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer"

redhat6系统默认安装的glibc-2.12,有的软件依赖的是glibc-2.14,这时需要升级glibc,下载安装 http://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.gz # ./configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin# make# make install 结果make

hive通过外部表读写elasticsearch数据,和读写hbase数据差不多,差别是需要下载elasticsearch-hadoop-hive-6.6.2.jar,然后使用其中的EsStorageHandler: Connect the massive data storage and deep processing power of Hadoop with the real-time search and analytics of Elasticsearch. The Elasticsea