GoldenGate实时投递数据到大数据平台（4）- ElasticSearch 2.x
ES 2.x
ES 2.x安装
下载elasticSearch 2.4.5, https://www.elastic.co/downloads/elasticsearch
解压下载后的压缩包，启动ES
./elasticsearch
| [2017-12-26 11:46:31,662][INFO ][node ] [Doctor Dorcas] version[2.4.5], pid[4424], build[c849dd1/2017-04-24T16:18:17Z]
[2017-12-26 11:46:31,663][INFO ][node ] [Doctor Dorcas] initializing ... [2017-12-26 11:46:32,665][INFO ][plugins ] [Doctor Dorcas] modules [reindex, lang-expression, lang-groovy], plugins [], sites [] [2017-12-26 11:46:32,748][INFO ][env ] [Doctor Dorcas] using [1] data paths, mounts [[/u02 (/dev/sdb1)]], net usable_space [36.3gb], net total_space [39.2gb], spins? [possibly], types [ext4] [2017-12-26 11:46:32,749][INFO ][env ] [Doctor Dorcas] heap size [1007.3mb], compressed ordinary object pointers [true] [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] initialized [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] starting ... [2017-12-26 11:46:35,197][INFO ][transport ] [Doctor Dorcas] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2017-12-26 11:46:35,204][INFO ][discovery ] [Doctor Dorcas] elasticsearch/R6MDYLdHRReeDckGGa9snw [2017-12-26 11:46:38,283][INFO ][cluster.service ] [Doctor Dorcas] new_master {Doctor Dorcas}{R6MDYLdHRReeDckGGa9snw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2017-12-26 11:46:38,363][INFO ][http ] [Doctor Dorcas] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2017-12-26 11:46:38,363][INFO ][gateway ] [Doctor Dorcas] recovered [0] indices into cluster_state [2017-12-26 11:46:38,363][INFO ][node ] [Doctor Dorcas] started |
查看ES服务状态
curl -GET http://localhost:9200/
| {
"name" : "Doctor Dorcas", "cluster_name" : "elasticsearch", "cluster_uuid" : "RXsZqkc7RvKejoXO3pVbdg", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
安装认证插件
默认GoldenGate是通过shield组件访问ES 2.x，所以要有认证模块才能投递数据。
停止ES进程
bin/plugin install license
bin/plugin install shield
重新启动ES进程
创建OGG访问的用户，并指定为admin用户。
bin/shield/esusers useradd elastic –r damin
Enter new password: changeme
Retype new password: changeme
查看用户及角色
[oracle@ol73 shield]$ ./esusers list
elastic : admin,user
确认新建的用户有权限访问
此时匿名访问将报错
curl -GET http://localhost:9200/
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"shield\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"shield\" charset=\"UTF-8\""}},"status":401}
需要带上用户密码即可访问
curl -u elastic:changeme -GET http://localhost:9200/
| {
"name" : "Cardinal", "cluster_name" : "elasticsearch", "cluster_uuid" : "8DEhE0t0R_u9tN6ZtwagkA", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
OGG投递测试
OGG配置
设置环境变量
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server
使用示例自带的队列文件投递数据
GGSCI> add replicat res, exttrail AdapterExamples/trail/tr
进程内容，用系统自带的示例即可
| REPLICAT res
-- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat res, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/elasticsearch2x.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1000 MAP QASOURCE.*, TARGET QASOURCE.*; |
其中elasticsearch2x.props 内容如下
| gg.handlerlist=elasticsearch
gg.handler.elasticsearch.type=elasticsearch ## Handler properties for Elasticsearch 2.x gg.handler.elasticsearch.ServerAddressList=localhost:9300 gg.handler.elasticsearch.clientSettingsFile=client2x.properties gg.handler.elasticsearch.version=2.x #gg.handler.elasticsearch.bulkWrite=true gg.classpath=/u02/elastic/lib/*:/u02/elastic/plugins/shield/* goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm |
client2x.properties
cluster.name=elasticsearch
shield.user=elastic:changeme
启动ogg res
GGSCI (ol73) 16> start res
Sending START request to MANAGER ...
REPLICAT RES starting
查看状态
GGSCI (ol73) 28> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:10 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
First Record RBA 0
GGSCI (ol73) 29> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:07 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
2015-11-06 02:45:39.000000 RBA 5660
已经处理完示例队列文件，查看统计信息
GGSCI (ol73) 30> stats res, total
Sending STATS request to REPLICAT RES ...
Start of Statistics at 2017-12-27 16:37:57.
Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 1.00
Total deletes 0.00
Total discards 0.00
Total operations 6.00
Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 3.00
Total deletes 2.00
Total discards 0.00
Total operations 10.00
End of Statistics.
可以看到，数据有正常写入。
数据验证
通过ES Rest进行查询
curl -GET http://localhost:9200/_cat/indices?v
| health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open qasource_tcustmer 5 1 5 1 14.8kb 14.8kb yellow open qasource_tcustord 5 1 3 3 21.4kb 21.4kb |
可以看到有新建2个索引：qasource_tcustmer, qasource_tcustord.
查看索引的mapping信息
curl -XGET 'http://localhost:9200/_mapping?pretty=true'
|
{ "qasource_tcustord" : { "mappings" : { "TCUSTORD" : { "properties" : { "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ORDER_DATE" : { "type" : "date" }, "ORDER_ID" : { "type" : "float" }, "PRODUCT_AMOUNT" : { "type" : "long" }, "PRODUCT_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "PRODUCT_PRICE" : { "type" : "float" }, "TRANSACTION_ID" : { "type" : "float" } } } } }, "qasource_tcustmer" : { "mappings" : { "TCUSTMER" : { "properties" : { "CITY" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "NAME" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "STATE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } |
可以看到每个索引的各个字段属性
查询其中一张表的所有数据
$ curl -s -XGET 'http://localhost:9200/qasource_tcustord/_search?q=*&pretty=true'
{
"took" : 83,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1995-12-31 15:00:00_CAR_765",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 14000.0,
"ORDER_DATE" : "1995-12-31T15:00:00",
"ORDER_ID" : 765.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1996-01-01 00:00:00_TRUCK_333",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 25000.0,
"ORDER_DATE" : "1996-01-01T00:00:00",
"ORDER_ID" : 333.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 15,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "TRUCK"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "WILL_1994-09-30 15:33:00_CAR_144",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 16520.0,
"ORDER_DATE" : "1994-09-30T15:33:00",
"ORDER_ID" : 144.0,
"CUST_CODE" : "WILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
}
]
}
}
ES中可正常查询写入的数据，测试完成。
GoldenGate实时投递数据到大数据平台（4）- ElasticSearch 2.x的更多相关文章
- GoldenGate实时投递数据到大数据平台（2）- Cassandra
简介 GoldenGate是一款可以实时投递数据到大数据平台的软件,针对apache cassandra,经过简单配置,即可实现从关系型数据将增量数据实时投递到Cassandra,以下介绍配置过程. ...
- GoldenGate实时投递数据到大数据平台（5） - Kafka
Oracle GoldenGate是Oracle公司的实时数据复制软件,支持关系型数据库和多种大数据平台.从GoldenGate 12.2开始,GoldenGate支持直接投递数据到Kafka等平台, ...
- 大数据学习---大数据的学习【all】
大数据介绍 什么是大数据以及有什么特点 大数据:是指无法在一定时间内用常规软件工具对其内容进行抓取.管理和处理的数据集合. 大数据是一种方法论:“一切都被记录,一切都被数字化,从数据中寻找需求,寻找知 ...
- 转 开启“大数据”时代--大数据挑战与NoSQL数据库技术 iteye
一直觉得“大数据”这个名词离我很近,却又很遥远.最近不管是微博上,还是各种技术博客.论坛,碎碎念大数据概念的不胜枚举. 在我的理解里,从概念理解上来讲,大数据的目的在于更好的数据分析,否则如此大数据的 ...
- GoldenGate实时投递数据到大数据平台（6）– HDFS
GoldenGate可以实时将RDBMS的数据投递到HDFS中,在前面的文章中,已经配置过投递到kafka, mongodb等数据平台,本文通过OGG for bigdata的介质中自带的示例演示实时 ...
- GoldenGate实时投递数据到大数据平台（3）- Apache Flume
Apache Flume Flume NG是一个分布式.可靠.可用的系统,它能够将不同数据源的海量日志数据进行高效收集.聚合,最后存储到一个中心化数据存储系统中,方便进行数据分析.事实上flume也可 ...
- GoldenGate实时投递数据到大数据平台（7）– Apache Hbase
Apache Hbase安装及运行 安装hbase1.4,确保在这之前hadoop是正常运行的.设置相应的环境变量, export HADOOP_HOME=/u01/hadoop export HBA ...
- GoldenGate实时投递数据到大数据平台（1）-MongoDB
mongodb安装 安装 linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 进行安装. 也可以在windows上 ...
- [转载] 使用 Twitter Storm 处理实时的大数据
转载自http://www.ibm.com/developerworks/cn/opensource/os-twitterstorm/ 流式处理大数据简介 Storm 是一个开源的.大数据处理系统,与 ...
随机推荐
- 怎样打开U盘最安全
为了避免电脑使用U盘时,通过双击,或者右击盘符时,导致把病毒感染至整个电脑,因此使用下面的方法,可使U盘病毒不被激活传播. 在取消了U盘自动运行的情况下(在组策略中一定要关闭自动运行功能,否则只要一插 ...
- 前端框架之Vue(10)-全家桶简单使用实例
vue-router官方文档 vuex官方文档 安装 npm install vue-router --save 使用实例 vue-router初使用(webpack-simple模板) 1.切换到指 ...
- TCP接收缓存大小的手动调整
给出了几个可调节的参数,它们可以帮助您提高 Linux TCP/IP 栈的性能. 表 1. TCP/IP 栈性能使用的可调节内核参数 可调节的参数 默认值 选项说明 /proc/sys/net/cor ...
- Scala泛型详解
在Scala中你可以使用类型参数来实现类和函数,这样的类和函数可以用于多种类型.比如Array[T] 你可以存放任意指定类型T的数据. 类.特质.函数都可以有类型参数:将类型参数放在名字后面用方括号括 ...
- LuaFileSystem
bokeyuan_1234 lua的文件管理 lua没有自己的文件管理 只有读取和写入文件,但是可以通过调用lfs(LuaFileSystem),lfs是一个 用于lua进行文件访问的库,支持lua5 ...
- vue watch 监听element upload组件上传成功返回的url列表
因为 on-success 上传成功返回的是一个异步的结果....如果父组件需要这个结果的话 必须用watch 监听 然后里面建立一个 save()方法 save方法里面再调用接口 传需要的上传之后的 ...
- sparse.coo_matrix()
coo_matrix.tocsr(copy = False ) 将此矩阵转换为压缩稀疏行格式,重复的条目将汇总在一起. 举例: from numpy import array from scipy.s ...
- 20165321 实验三 敏捷开发与XP实践-4
Java密码学程序 import java.util.*; public class Lfsr { public static void main(String[] args) { System.ou ...
- ROSETTA使用技巧随笔--Full Atom Representation和Centroid Representation
Full Atom Representation vs Centroid Representation Full Atom Representation即全原子标识,氨基酸残基的所有相关原子,均原封不 ...
- 从零开始一起学习SLAM | SLAM有什么用？
SLAM是 Simultaneous Localization And Mapping的 英文首字母组合,一般翻译为:同时定位与建图.同时定位与地图构建. 「同时定位与地图构建」这几个词,乍一听起来非 ...