GoldenGate实时投递数据到大数据平台（4）- ElasticSearch 2.x
ES 2.x
ES 2.x安装
下载elasticSearch 2.4.5, https://www.elastic.co/downloads/elasticsearch
解压下载后的压缩包，启动ES
./elasticsearch
| [2017-12-26 11:46:31,662][INFO ][node ] [Doctor Dorcas] version[2.4.5], pid[4424], build[c849dd1/2017-04-24T16:18:17Z]
[2017-12-26 11:46:31,663][INFO ][node ] [Doctor Dorcas] initializing ... [2017-12-26 11:46:32,665][INFO ][plugins ] [Doctor Dorcas] modules [reindex, lang-expression, lang-groovy], plugins [], sites [] [2017-12-26 11:46:32,748][INFO ][env ] [Doctor Dorcas] using [1] data paths, mounts [[/u02 (/dev/sdb1)]], net usable_space [36.3gb], net total_space [39.2gb], spins? [possibly], types [ext4] [2017-12-26 11:46:32,749][INFO ][env ] [Doctor Dorcas] heap size [1007.3mb], compressed ordinary object pointers [true] [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] initialized [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] starting ... [2017-12-26 11:46:35,197][INFO ][transport ] [Doctor Dorcas] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2017-12-26 11:46:35,204][INFO ][discovery ] [Doctor Dorcas] elasticsearch/R6MDYLdHRReeDckGGa9snw [2017-12-26 11:46:38,283][INFO ][cluster.service ] [Doctor Dorcas] new_master {Doctor Dorcas}{R6MDYLdHRReeDckGGa9snw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2017-12-26 11:46:38,363][INFO ][http ] [Doctor Dorcas] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2017-12-26 11:46:38,363][INFO ][gateway ] [Doctor Dorcas] recovered [0] indices into cluster_state [2017-12-26 11:46:38,363][INFO ][node ] [Doctor Dorcas] started |
查看ES服务状态
curl -GET http://localhost:9200/
| {
"name" : "Doctor Dorcas", "cluster_name" : "elasticsearch", "cluster_uuid" : "RXsZqkc7RvKejoXO3pVbdg", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
安装认证插件
默认GoldenGate是通过shield组件访问ES 2.x，所以要有认证模块才能投递数据。
停止ES进程
bin/plugin install license
bin/plugin install shield
重新启动ES进程
创建OGG访问的用户，并指定为admin用户。
bin/shield/esusers useradd elastic –r damin
Enter new password: changeme
Retype new password: changeme
查看用户及角色
[oracle@ol73 shield]$ ./esusers list
elastic : admin,user
确认新建的用户有权限访问
此时匿名访问将报错
curl -GET http://localhost:9200/
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"shield\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"shield\" charset=\"UTF-8\""}},"status":401}
需要带上用户密码即可访问
curl -u elastic:changeme -GET http://localhost:9200/
| {
"name" : "Cardinal", "cluster_name" : "elasticsearch", "cluster_uuid" : "8DEhE0t0R_u9tN6ZtwagkA", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
OGG投递测试
OGG配置
设置环境变量
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server
使用示例自带的队列文件投递数据
GGSCI> add replicat res, exttrail AdapterExamples/trail/tr
进程内容，用系统自带的示例即可
| REPLICAT res
-- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat res, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/elasticsearch2x.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1000 MAP QASOURCE.*, TARGET QASOURCE.*; |
其中elasticsearch2x.props 内容如下
| gg.handlerlist=elasticsearch
gg.handler.elasticsearch.type=elasticsearch ## Handler properties for Elasticsearch 2.x gg.handler.elasticsearch.ServerAddressList=localhost:9300 gg.handler.elasticsearch.clientSettingsFile=client2x.properties gg.handler.elasticsearch.version=2.x #gg.handler.elasticsearch.bulkWrite=true gg.classpath=/u02/elastic/lib/*:/u02/elastic/plugins/shield/* goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm |
client2x.properties
cluster.name=elasticsearch
shield.user=elastic:changeme
启动ogg res
GGSCI (ol73) 16> start res
Sending START request to MANAGER ...
REPLICAT RES starting
查看状态
GGSCI (ol73) 28> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:10 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
First Record RBA 0
GGSCI (ol73) 29> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:07 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
2015-11-06 02:45:39.000000 RBA 5660
已经处理完示例队列文件，查看统计信息
GGSCI (ol73) 30> stats res, total
Sending STATS request to REPLICAT RES ...
Start of Statistics at 2017-12-27 16:37:57.
Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 1.00
Total deletes 0.00
Total discards 0.00
Total operations 6.00
Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 3.00
Total deletes 2.00
Total discards 0.00
Total operations 10.00
End of Statistics.
可以看到，数据有正常写入。
数据验证
通过ES Rest进行查询
curl -GET http://localhost:9200/_cat/indices?v
| health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open qasource_tcustmer 5 1 5 1 14.8kb 14.8kb yellow open qasource_tcustord 5 1 3 3 21.4kb 21.4kb |
可以看到有新建2个索引：qasource_tcustmer, qasource_tcustord.
查看索引的mapping信息
curl -XGET 'http://localhost:9200/_mapping?pretty=true'
|
{ "qasource_tcustord" : { "mappings" : { "TCUSTORD" : { "properties" : { "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ORDER_DATE" : { "type" : "date" }, "ORDER_ID" : { "type" : "float" }, "PRODUCT_AMOUNT" : { "type" : "long" }, "PRODUCT_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "PRODUCT_PRICE" : { "type" : "float" }, "TRANSACTION_ID" : { "type" : "float" } } } } }, "qasource_tcustmer" : { "mappings" : { "TCUSTMER" : { "properties" : { "CITY" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "NAME" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "STATE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } |
可以看到每个索引的各个字段属性
查询其中一张表的所有数据
$ curl -s -XGET 'http://localhost:9200/qasource_tcustord/_search?q=*&pretty=true'
{
"took" : 83,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1995-12-31 15:00:00_CAR_765",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 14000.0,
"ORDER_DATE" : "1995-12-31T15:00:00",
"ORDER_ID" : 765.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1996-01-01 00:00:00_TRUCK_333",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 25000.0,
"ORDER_DATE" : "1996-01-01T00:00:00",
"ORDER_ID" : 333.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 15,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "TRUCK"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "WILL_1994-09-30 15:33:00_CAR_144",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 16520.0,
"ORDER_DATE" : "1994-09-30T15:33:00",
"ORDER_ID" : 144.0,
"CUST_CODE" : "WILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
}
]
}
}
ES中可正常查询写入的数据，测试完成。
GoldenGate实时投递数据到大数据平台（4）- ElasticSearch 2.x的更多相关文章
- GoldenGate实时投递数据到大数据平台（2）- Cassandra
简介 GoldenGate是一款可以实时投递数据到大数据平台的软件,针对apache cassandra,经过简单配置,即可实现从关系型数据将增量数据实时投递到Cassandra,以下介绍配置过程. ...
- GoldenGate实时投递数据到大数据平台（5） - Kafka
Oracle GoldenGate是Oracle公司的实时数据复制软件,支持关系型数据库和多种大数据平台.从GoldenGate 12.2开始,GoldenGate支持直接投递数据到Kafka等平台, ...
- 大数据学习---大数据的学习【all】
大数据介绍 什么是大数据以及有什么特点 大数据:是指无法在一定时间内用常规软件工具对其内容进行抓取.管理和处理的数据集合. 大数据是一种方法论:“一切都被记录,一切都被数字化,从数据中寻找需求,寻找知 ...
- 转 开启“大数据”时代--大数据挑战与NoSQL数据库技术 iteye
一直觉得“大数据”这个名词离我很近,却又很遥远.最近不管是微博上,还是各种技术博客.论坛,碎碎念大数据概念的不胜枚举. 在我的理解里,从概念理解上来讲,大数据的目的在于更好的数据分析,否则如此大数据的 ...
- GoldenGate实时投递数据到大数据平台（6）– HDFS
GoldenGate可以实时将RDBMS的数据投递到HDFS中,在前面的文章中,已经配置过投递到kafka, mongodb等数据平台,本文通过OGG for bigdata的介质中自带的示例演示实时 ...
- GoldenGate实时投递数据到大数据平台（3）- Apache Flume
Apache Flume Flume NG是一个分布式.可靠.可用的系统,它能够将不同数据源的海量日志数据进行高效收集.聚合,最后存储到一个中心化数据存储系统中,方便进行数据分析.事实上flume也可 ...
- GoldenGate实时投递数据到大数据平台（7）– Apache Hbase
Apache Hbase安装及运行 安装hbase1.4,确保在这之前hadoop是正常运行的.设置相应的环境变量, export HADOOP_HOME=/u01/hadoop export HBA ...
- GoldenGate实时投递数据到大数据平台（1）-MongoDB
mongodb安装 安装 linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 进行安装. 也可以在windows上 ...
- [转载] 使用 Twitter Storm 处理实时的大数据
转载自http://www.ibm.com/developerworks/cn/opensource/os-twitterstorm/ 流式处理大数据简介 Storm 是一个开源的.大数据处理系统,与 ...
随机推荐
- socket 套接字总结
简单版 服务端 import socket import struct import json import os server_dir = r'E:\Moudule_1\socket练习\serve ...
- pip升级时报错--- No module named 'pip._internal'
一.问题: 之前python3.6是安装的pip版本为:pip=9.0.1,我按照提示升级报错,一直装不上pip18.0,于是直接在site-package目录下删掉了pi ...
- IE8“开发人员工具”（下）
浏览器模式 说白了,就是让用户选择当前页面用何种IE版本去渲染. 文本模式 说起“文本模式”这个名词,这又要回到渲染页面的3种模式了:诡异模式(Quirks mode,也有翻译为兼容模式.怪异模式的) ...
- C#编程基础
1..NET与C# A..NET 是 Microsoft XML Web services 平台.XML Web services 允许应用程序通过 Internet 进行通讯和共享数据,而不管所采用 ...
- 快学Scala 1
1. Scala解释器读到一个表达式,对它进行求值,将它打印出来,接着再继续读下一个表达式.这个过程被称作“读取-求值-打印-循环”,即REPL. 2. 从技术上来讲,scala程序并不是一个 ...
- BiLSTM学习
转自:https://blog.csdn.net/aliceyangxi1987/article/details/77094970 https://blog.csdn.net/jojozhangju/ ...
- [Git/GitHub] Tutorial 1. Git download and commit first project
1. Install at https://git-scm.com/downloads 2. Set up your name and email $ git config --global user ...
- [LeetCode] 492. Construct the Rectangle_Easy tag: Math
For a web developer, it is very important to know how to design a web page's size. So, given a speci ...
- Java Selenium - 元素操作 (二)
一篇概括了常用的元素定位方法,但是找到元素还是不够的,模拟鼠标的操作,完成各个功能点的自动操作才是关键. 下面是常见的页面元素操作会涉及到的方法,不是很全,比较复杂的后面单独拿出来做案例. 一, 输入 ...
- https://sweetalert2.github.io/
https://sweetalert2.github.io/