ORACLE GOLDEN GATE oracle同步数据至kafka

一.服务器信息

ip		软件版本	ogg版本	软件包	操作系统版本	OGG安装路径
10.1.50.52	源	oracle11.2.0.4	12.2.0.1.1	V100692-01.zip	centos6.9	/data/ogg
10.0.71.102	目标	kafka0.10	12.2.0.1.160823	123010_ggs_Adapters_Linux_x64.zip	centos7.3	/data/ogg

二.源端安装OGG

1.解压V100692-01.zip

unzip V100692-.zip

这个时候会看到fbo_ggs_Linux_x64_shiphome文件夹

2.我们采用静默安装的方式进行安装（注意要使用oracle用户来进行安装，oggcore.rsp配置需要根据实际需要进行修改，也可使用图形界面进行安装）

/usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent -responseFile /usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

三.源端OGG配置

1.开启主库归档日志、补充日志及force logging

alter database add supplemental log data;

alter database force logging;

alter system set enable_goldengate_replication=true scope=both;

2.创建ogg的管理用户

create user ogg identified by password account unlock;

grant connect,resource to ogg;

grant select any dictionary to ogg;

grant select any table to ogg;

grant execute on utl_file to ogg;

grant restricted session to ogg;

grant create table,create sequence to ogg;

alter table DCSDBA.YH_ITF_INVENTORY_TRANSACTION add supplemental log data (all) columns;--如果kafka需要接收到完整的修改信息就需要开启补全日志

3.登录ogg并创建目录

/data/ogg/ggsci

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:普通表格;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.5pt;
mso-bidi-font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-font-kerning:1.0pt;}

create subdirs

4.配置源端manager进程

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

edit params mgr

port 1357
autostart er *
autorestart er *

5.启动

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

manager并确认状态

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

start mgr

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

info all 查看MANAGER进程为running状态

6.配置加密

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

ENCRYPT PASSWORD password ENCRYPTKEY
DEFAULT

7.配置数据抽取进程

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

①加一个extract，名字随便取

add extract hnwmsprd,tranlog,begin now

②为这个ext指定他把远程文件放到远程机的哪个目录

ADD RMTTRAIL /data/ogg/dirdat/tt, EXTRACT hnwmsprd

③编辑配置文件

edit params hnwmsprd

extract hnwmsprd

setenv
(NLS_LANG="AMERICAN_AMERICA.UTF8")

setenv
(ORACLE_SID="hnwmsprd")

userid ogg,password AACAAAAAAAAAAAKALILFJIXIECMGKFAHRBJHGDGBNJLBOAUC,ENCRYPTKEY
DEFAULT

rmthost 10.0.71.102, mgrport 1357

rmttrail /data/ogg/dirdat/tt

discardfile /data/ogg/dirrpt/trail.dsc,append,megabytes
100

gettruncates

GETUPDATEBEFORES
NOCOMPRESSDELETES
NOCOMPRESSUPDATES

TRANLOGOPTIONS MINEFROMACTIVEDG

table DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

④启动进程

start hnwmsprd

四.OGG目标端搭建

目标端搭建只需解压软件即可

unzip 123010_ggs_Adapters_Linux_x64.zip

解压出来ggs_Adapters_Linux_x64.tar文件

将文件移动到/data/ogg目录下解压

mv ggs_Adapters_Linux_x64.tar /data/ogg

tar -xvf ggs_Adapters_Linux_x64.tar

五.OGG目标端配置

1.登录ogg并创建目录

/data/ogg/ggsci

create subdirs

2.配置源端manager进程

edit params mgr

port 1357
autostart er *
autorestart er *

3.启动manager并确认状态

start mgr

info all 查看MANAGER进程为running状态

4.添加复制进程

add replicat rkafka,exttrail dirdat/tt,begin now

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

edit params rkafka

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

REPLICAT rkafka

TARGETDB LIBFILE libggjava.so SET
property=dirprm/kafka.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP DCSDBA.YH_ITF_INVENTORY_TRANSACTION,
TARGET DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

5.修改kafka.props（注意：kafka.props要放在/data/ogg/dirprm路径下）

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

gg.handlerlist = kafkahandler

gg.handler.kafkahandler.type = kafka

gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties

gg.handler.kafkahandler.TopicName =hnwmsprd

gg.handler.kafkahandler.format =json

gg.handler.kafkahandler.SchemaTopicName=ogg

gg.handler.kafkahandler.BlockingSend
=true

gg.handler.kafkahandler.includeTokens=false

gg.handler.kafkahandler.mode =tx

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

gg.classpath=dirprm/:/usr/local/kafka/libs/*:

javawriter.bootoptions=-Xmx512m
-Xms32m -Djava.class.path=ggjava/ggjava.jar

以上配置文件根据实际配置进行修改

6.编辑custom_kafka_producer.properties文件（注意：custom_kafka_producer.properties要放在/data/ogg/dirprm路径下）

#bootstrap.servers=host:port
bootstrap.servers=10.0.71.102:9092
acks=1
#compression.type=gzip
reconnect.backoff.ms=1000

value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=102400
linger.ms=10000

7.启动复制进程

start

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

rkafka

六.验证

目标端执行

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hnwmsprd

源端更新DCSDBA.YH_ITF_INVENTORY_TRANSACTION表。

如果是在adg作为源端需要在源抽取进程上添加参数

TRANLOGOPTIONS MINEFROMACTIVEDG

并在主库开启ddl，可以参考下面这位大神的配置

http://czmmiao.iteye.com/blog/1609823

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE