elasticsearch-hadoop是一个深度集成Hadoop和ElasticSearch的项目,也是ES官方来维护的一个子项目,通过实现Hadoop和ES之间的输入输出,可以在Hadoop里面对ES集群的数据进行读取和写入,充分发挥Map-Reduce并行处理的优势,为Hadoop数据带来实时搜索的可能。 
项目网址:http://www.elasticsearch.org/overview/hadoop/

运行环境: 
CDH4、ElasticSearch0.90.2

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Quick-Start/cdh4qs_topic_3_3.html

https://github.com/medcl/elasticsearch-rtf

Hive和ES的互操作: 
#安装,HIVE里面添加ElasticSearch-Hadoop的JAR路径 
#下载hadoop-es jar包,https://download.elasticsearch.org/hadoop/hadoop-latest.zip

#Hive加载的JAR路径为本地路径

[medcl@node- ~]$ ls
elasticsearch-hadoop-1.3..M1.jar
[medcl@node- ~]$ pwd
/home/medcl
[medcl@node- ~]$ hive -hiveconf hive.aux.jars.path=/home/medcl/elasticsearch-hadoop-1.3..M1.jar
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/medcl/hive_job_log_94db3616-e210-4aab-b07b-6fb159e217ec_1758848920.txt

#ElasticSearch集群名为"elasticsearch",和Hadoop在一个机器上

#Hive里面创建一个Table(user),并使用Hadoop-ElasticSearch关联一个索引(/index/user),2个字段,id和name

CREATE EXTERNAL TABLE user  (id INT, name STRING,site STRING)
STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
TBLPROPERTIES('es.resource' = 'index/user/',
'es.index.auto.create' = 'true')
在medcl用下操作:
CREATE EXTERNAL TABLE user  (id INT, name STRING)
STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
TBLPROPERTIES('es.resource' = '/index/user/',
'es.index.auto.create' = 'true');
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive> CREATE EXTERNAL TABLE user (id INT, name STRING)
> STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
> TBLPROPERTIES('es.resource' = 'medcl/',
> 'es.index.auto.create' = 'false');
FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=medcl, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
 
#擦,看下权限
[medcl@node- ~]$ hadoop fs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxrwxrwt - hdfs supergroup -- : /tmp
drwxr-xr-x - hdfs supergroup -- : /user
drwxr-xr-x - medcl supergroup -- : /user/medcl
drwxr-xr-x - medcl supergroup -- : /user/medcl/input
-rw-r--r-- medcl supergroup -- : /user/medcl/input/file1.txt
drwxr-xr-x - medcl supergroup -- : /user/medcl/lib
-rw-r--r-- medcl supergroup -- : /user/medcl/lib/elasticsearch-hadoop-1.3..M1.jar
drwxr-xr-x - hdfs supergroup -- : /var
drwxr-xr-x - hdfs supergroup -- : /var/lib
#原来user目录权限是hdfs,ok,切换hdfs,jar也换个hdfs用户可以访问到的位置,就/tmp吧
[root@node- medcl]# cp elasticsearch-hadoop-1.3..M1.jar /tmp/
[root@node- medcl]# ^C
[root@node- medcl]# sudo -u hdfs hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/hdfs/hive_job_log_bdad4d7a-f929-43d7-a56e-e026fdd7e3b4_1219802521.txt
hive> CREATE EXTERNAL TABLE user (id INT, name STRING)
> STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
> TBLPROPERTIES('es.resource' = '/index/user/',
> 'es.index.auto.create' = 'false');
-- ::29.560 GMT Thread[main,,main] java.io.FileNotFoundException: derby.log (Permission denied)
----------------------------------------------------------------
-- ::29.877 GMT:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.4.2.0 - (): instance a816c00e--fc62-4b5c-000000cec758
on database directory /var/lib/hive/metastore/metastore_db in READ ONLY mode Database Class Loader started - derby.database.classpath=''
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code from org.apache.hadoop.hive.ql.exec.DDLTask #ok,干掉lock
[root@node- ~]# ls /var/lib/hive/metastore/metastore_db
dbex.lck db.lck log seg0 service.properties tmp
[root@node- ~]# rm /var/lib/hive/metastore/metastore_db/dbex.lck
rm: remove regular file `/var/lib/hive/metastore/metastore_db/dbex.lck'? y
[root@node- ~]# rm /var/lib/hive/metastore/metastore_db/db.lck
rm: remove regular file `/var/lib/hive/metastore/metastore_db/db.lck'? y

#另外忘记关另外一个hive实例了,难怪呢。

[root@node- tmp]# ps -aux|grep hive
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2./FAQ
root 0.0 0.1 pts/ S+ : : sudo -u hdfs hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
hdfs 1.8 5.7 pts/ Sl+ : : /usr/lib/jvm/java-openjdk/bin/java -Xmx256m -Dhadoop.log.dir=/usr/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-cli-0.10.-cdh4.5.0.jar org.apache.hadoop.hive.cli.CliDriver -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
 
#权限问题
[root@node- tmp]# ll /var/lib/hive/metastore/metastore_db/
total
drwxrwxr-x medcl medcl Dec : log
drwxrwxr-x medcl medcl Dec : seg0
-rw-rw-r-- medcl medcl Dec : service.properties
drwxrwxr-x medcl medcl Dec : tmp
[root@node- tmp]# sudo -u hdfs hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar^C
[root@node- tmp]# chmod /var/lib/hive/metastore/metastore_db/ -R
[root@node- tmp]# sudo -u hdfs hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/hdfs/hive_job_log_d5749cb0-fde0-4da2--c85cf4673885_252074310.txt
hive> show tables;
OK
Time taken: 6.934 seconds
hive> CREATE EXTERNAL TABLE user (id INT, name STRING)
> STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
> TBLPROPERTIES('es.resource' = '/index/user/',
> 'es.index.auto.create' = 'true');
OK
Time taken: 1.115 seconds #ok,创建成功了
hive> show tables;
OK
user
Time taken: 0.15 seconds
hive> #权限问题是Hive默认仓库路径造成的,生疏了
[root@node- tmp]# sudo su hdfs
bash-4.1$ hadoop fs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxrwxrwt - hdfs supergroup -- : /tmp
drwxr-xr-x - hdfs supergroup -- : /user
drwxr-xr-x - hdfs supergroup -- : /user/hive
drwxr-xr-x - hdfs supergroup -- : /user/hive/warehouse
drwxr-xr-x - hdfs supergroup -- : /user/hive/warehouse/user #好了,开始往HIVE里面倒数据了,先来几行数据
[root@node- tmp]# cat files1.txt
,medcl
,lcdem
,tom
,jack #传上去
[root@node- tmp]# sudo su hdfs
bash-4.1$ hadoop fs -put files1.txt /tmp/
bash-4.1$ hadoop fs -ls /tmp/
Found items
-rw-r--r-- hdfs supergroup -- : /tmp/files1.txt #加载到Hive里面
hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
#LOAD DATA LOCAL INPATH '/tmp/files1.txt' OVERWRITE INTO TABLE user_source;
#CREATE EXTERNAL TABLE user_source (id INT, name STRING); #不是原始Hive表,还不能直接LOAD
bash-4.1$ hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/hdfs/hive_job_log_a9516f87-6e2d-44db-9d38-18eed77d9dec_1583221137.txt
hive> LOAD DATA LOCAL INPATH '/tmp/files1.txt' OVERWRITE INTO TABLE user;
FAILED: SemanticException [Error ]: A non-native table cannot be used as target for LOAD
hive> CREATE EXTERNAL TABLE user_source (id INT, name STRING);
OK
Time taken: 1.104 seconds
hive> LOAD DATA LOCAL INPATH '/tmp/files1.txt' OVERWRITE INTO TABLE user_source;
Copying data from file:/tmp/files1.txt
Copying file: file:/tmp/files1.txt
Loading data to table default.user_source
Table default.user_source stats: [num_partitions: , num_files: , num_rows: , total_size: , raw_data_size: ]
OK
Time taken: 0.911 seconds
hive> show tables;
OK
user
user_source
Time taken: 0.226 seconds #下面这个错误是因为es-hadoop的jar文件没有传到HDFS上面,看来本地和HDFS都要上传,并且路径要一致
hive> select id,name from user_source;
Total MapReduce jobs =
Launching Job out of
Number of reduce tasks is set to since there's no reduce operator
java.io.FileNotFoundException: File does not exist: /tmp/elasticsearch-hadoop-1.3..M1.jar
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:)
at org.apache.hadoop.filecache.DistributedCache.getFileStatus(DistributedCache.java:)
at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestamps(TrackerDistributedCacheManager.java:)
at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestampsAndCacheVisibilities(TrackerDistributedCacheManager.java:)
at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:)
at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:)
at org.apache.hadoop.mapred.JobClient.access$(JobClient.java:)
at org.apache.hadoop.mapred.JobClient$.run(JobClient.java:)
at org.apache.hadoop.mapred.JobClient$.run(JobClient.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.util.RunJar.main(RunJar.java:)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /tmp/elasticsearch-hadoop-1.3.0.M1.jar)'
FAILED: Execution Error, return code from org.apache.hadoop.hive.ql.exec.MapRedTask #ok,再看看
bash-4.1$ hadoop fs -put elasticsearch-hadoop-1.3..M1.jar /tmp/
bash-4.1$ hive -hiveconf hive.aux.jars.path=/tmp/elasticsearch-hadoop-1.3..M1.jar
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/hdfs/hive_job_log_28ea1fbc-dc3b-4e62-9f47-1a88eed30069_1310993479.txt
hive> select id,name from user_source;
Total MapReduce jobs =
Launching Job out of
Number of reduce tasks is set to since there's no reduce operator
Starting Job = job_201312162220_0004, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201312162220_0004
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201312162220_0004
Hadoop job information for Stage-: number of mappers: ; number of reducers:
-- ::, Stage- map = %, reduce = %
-- ::, Stage- map = %, reduce = %, Cumulative CPU 0.88 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 0.88 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 0.88 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 0.88 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 0.88 sec
MapReduce Total cumulative CPU time: msec
Ended Job = job_201312162220_0004
MapReduce Jobs Launched:
Job : Map: Cumulative CPU: 0.88 sec HDFS Read: HDFS Write: SUCCESS
Total MapReduce CPU Time Spent: msec
OK
NULL NULL
NULL NULL
NULL NULL
NULL NULL
Time taken: 25.999 seconds #慢,数据怎么是空的,建成外表了(EXTERNAL),没有设置默认的分隔符,好纠结
hive> drop table user_source;
OK
Time taken: 0.649 seconds
hive> CREATE TABLE user_source (id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
OK
Time taken: 0.109 seconds
hive> LOAD DATA LOCAL INPATH '/tmp/files1.txt' INTO TABLE user_source;
Copying data from file:/tmp/files1.txt
Copying file: file:/tmp/files1.txt
Loading data to table default.user_source
Table default.user_source stats: [num_partitions: , num_files: , num_rows: , total_size: , raw_data_size: ]
OK
Time taken: 0.348 seconds
hive> select * from user_source;
OK
medcl
lcdem
tom
jack
Time taken: 0.155 seconds #源表现在有了,导入到ES所在的表里面去 hive> INSERT OVERWRITE TABLE user
> SELECT s.id, s.name FROM user_source s;
Total MapReduce jobs =
Launching Job out of
Number of reduce tasks is set to since there's no reduce operator
Starting Job = job_201312162220_0005, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201312162220_0005
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201312162220_0005
Hadoop job information for Stage-: number of mappers: ; number of reducers:
-- ::, Stage- map = %, reduce = %
-- ::, Stage- map = %, reduce = %, Cumulative CPU 1.16 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 1.16 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 1.16 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 1.16 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 1.16 sec
MapReduce Total cumulative CPU time: seconds msec
Ended Job = job_201312162220_0005
Rows loaded to user
MapReduce Jobs Launched:
Job : Map: Cumulative CPU: 1.16 sec HDFS Read: HDFS Write: SUCCESS
Total MapReduce CPU Time Spent: seconds msec
OK
Time taken: 21.849 seconds
hive> select * from user;
OK
Failed with exception java.io.IOException:java.lang.IllegalStateException: [GET] on [/index/user/&search_type=scan&scroll=10m&size=&preference=_shards:;_only_node:MP7Zl3owTRm8O2V6cWvOSg] failed; server[http://10.0.2.15:9200] returned [{"_index":"index","_type":"user","_id":"&search_type=scan&scroll=10m&size=50&preference=_shards:4;_only_node:MP7Zl3owTRm8O2V6cWvOSg","exists":false}]
Time taken: 0.387 seconds #可以看出来hadoop-elasticsearch翻译出来的查询语句好像有问题!不过elasticsearch里面已经有数据了,反正暂时不需要用hive来执行查询,先官方发个issue吧。 #ES查询结果
bash-4.1$ curl localhost:/index/user/_search?q=*&pretty=true
[]
bash-4.1$ {"took":,"timed_out":false,"_shards":{"total":,"successful":,"failed":},"hits":{"total":,"max_score":1.0,"hits":[{"_index":"index","_type":"user","_id":"3x4bEcriRvS6AHkX2Sb7UA","_score":1.0, "_source" : {"id":,"name":"lcdem"}},{"_index":"index","_type":"user","_id":"_3rGVWhaTSCixYxRzBUSLQ","_score":1.0, "_source" : {"id":,"name":"jack"}},{"_index":"index","_type":"user","_id":"T-Q_icjgR8ehsH3IV-twWw","_score":1.0, "_source" : {"id":,"name":"medcl"}},{"_index":"index","_type":"user","_id":"Vdz0sryBT5u0e9hfoMY8Tg","_score":1.0, "_source" : {"id":,"name":"tom"}}]}}
#接下来试试大量数据bulk导入的性能,是不是真的做到data locality。

elasticsearch-hadoop下载地址:https://github.com/elastic/elasticsearch-hadoop

elasticsearch-hadoop使用的更多相关文章

  1. es第十篇:Elasticsearch for Apache Hadoop

    es for apache hadoop(elasticsearch-hadoop.jar)允许hadoop作业(mapreduce.hive.pig.cascading.spark)与es交互. A ...

  2. Hadoop vs Elasticsearch – Which one is More Useful

    Hadoop vs Elasticsearch – Which one is More Useful     Difference Between Hadoop and Elasticsearch H ...

  3. 轻量级OLAP(二):Hive + Elasticsearch

    1. 引言 在做OLAP数据分析时,常常会遇到过滤分析需求,比如:除去只有性别.常驻地标签的用户,计算广告媒体上的覆盖UV.OLAP解决方案Kylin不支持复杂数据类型(array.struct.ma ...

  4. 使用Hive或Impala执行SQL语句,对存储在Elasticsearch中的数据操作(二)

    CSSDesk body { background-color: #2574b0; } /*! zybuluo */ article,aside,details,figcaption,figure,f ...

  5. 使用Hive或Impala执行SQL语句,对存储在Elasticsearch中的数据操作

    http://www.cnblogs.com/wgp13x/p/4934521.html 内容一样,样式好的版本. 使用Hive或Impala执行SQL语句,对存储在Elasticsearch中的数据 ...

  6. elasticsearch + hive环境搭建

    一.环境介绍: elasticsearch:2.3.1 hive:0.12 二.环境搭建 2.1 首先获取elasticsearc-hadoop的jar包 链接地址:http://jcenter.bi ...

  7. 使用hive访问elasticsearch的数据

    使用hive访问elasticsearch的数据 1.配置 将elasticsearch-hadoop-2.1.1.jar拷贝到hive/lib hive -hiveconf hive.aux.jar ...

  8. elasticsearch+spark+hbase 整合

    1.用到的maven依赖 <dependency> <groupId>org.apache.spark</groupId> <artifactId>sp ...

  9. Awesome Hadoop

    A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources. Inspired by Awesome PHP,  ...

  10. Elasticsearch 快速开始

    Elasticsearch 是一个分布式的 RESTful 风格的搜索和数据分析引擎. 查询 : Elasticsearch 允许执行和合并多种类型的搜索 - 结构化.非结构化.地理位置.度量指标 - ...

随机推荐

  1. 学习 TList 类的实现[4]

    现在准备一步步地模拟 TList 类, 建立一个自己的 TMyList. 首先, 这个类中应该包括前面提到的那个 Pointer 数组(TPointerList)的指针(PPointerList): ...

  2. Go 内嵌静态资源

    http://fuxiaohei.me/2016/10/1/go-binary-embed-asset.html

  3. nuget修改配置文件

    https://www.cnblogs.com/seejoy/p/8093837.html 然后将文件解压到需要打包的工程解决方案根目录下. 然后修改nuget文件夹下的 UploadNupkg.ex ...

  4. 7 -- Spring的基本用法 -- 3... Spring 的核心机制 : 依赖注入

    7.3 Spring 的核心机制 : 依赖注入 Spring 框架的核心功能有两个. Spring容器作为超级大工厂,负责创建.管理所有的Java对象,这些Java对象被称为Bean. Spring容 ...

  5. python2.0_s12_day9_协程&Gevent协程

    Python之路,Day9 - 异步IO\数据库\队列\缓存 本节内容 Gevent协程 Select\Poll\Epoll异步IO与事件驱动 Python连接Mysql数据库操作 协程 1.协程,又 ...

  6. Linux系统故障排除

    可能出现的故障: 1,管理员密码忘记 进入单用户模式修改密码 2.系统无法正常启动 a.grub损坏(MBR损坏,grub配置文件丢失) b.系统初始化故障(某文件系统无法正常挂载.驱动不兼容) c. ...

  7. Nginx遇上Access Denied提示怎么解决

    这几天在摆弄linux下面的各种服务器,对nginx非常有兴趣. 于是把phpmyadmin传上去了,先是phpmyadmin配了半天,结果配好之后发现phpmyadmin一些logo.css.js文 ...

  8. Android 使用CheckBox实现多选效果

    CheckBox:复选框1.有两种状态: 选中状态(true),未选中状态(false)2.属性: android:id="@+id/checkbox" android:layou ...

  9. 【HTML】改变鼠标样式图片css

    你需要一张图   .ico 的 格式 如果一开始你要解决的是怎么去用png 格式图片转成 ICO格式 先做一张32*32的PNG格式图片 然后 打开http://www.easyicon.net/co ...

  10. 【Mysql】大数据处理优化方法

    1.应尽量避免在 where 子句中使用 != 或  <>  操作符,否则将引擎放弃使用索引而进行全表扫描. 2.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 orde ...