zookeeper适用场景：配置文件同步
问题导读：
1.本文三个角色之间是什么关系？
2.三个角色的作用是什么？
3.如何代码实现这三个角色的作用？
在 zookeeper适用场景：zookeeper解决了哪些问题有关于分布式集群配置文件同步问题的描述，本文介绍如何把zk应用到配置文件分发的场景。
假设有三个角色
- trigger：发布最新的配置文件数据，发送指令和数据给zk_agent，实现是下面的trigger.py
- zk_agent：接收来自trigger.py触发的指令和数据，并且把数据更新到zk service上，从而触发zk_app来获取最新的配置数据，实现是下面的zk_agent.py
- zk_app：部署在每台worker上的注册监听zk中配置文件所在znode的变化，从而获取最新的配置文件，应用到worker中，实现是下面的zk_app.py
<ignore_js_op>
配置文件同步到:zk_agent实现逻辑：
* 初始化连接到zk service，首先竞选出master
* master create一个配置文件管理的PERSISTENT类型的znode，比如是/Applications/NginxConf，
* 启动一个线程，专门接收trigger发送的指令，一收到指令，就create一个"conf-"开头的SEQUENCE|PERSISTENT类型的节点，指定znode数据内容是从trigger收到的数据
* 第一次刚更新会创建节点/Applications/NginxConf/conf-0000000000,以后每次更新新的配置文件编号就会增大。
vim zk_agent.py
#!/usr/bin/env python2.7
# -*- coding: UTF-8 -*- # author : firefoxbug
# E-Mail : wanghuafire@gmail.com
# Blog : www.firefoxbug.net import logging
from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod import os
import sys
import threading
import signal
import time logging.basicConfig(
level = logging.DEBUG,
format = "[%(asctime)s] %(levelname)-8s %(message)s"
) log = logging class GJZookeeper(object): ZK_HOST = "localhost:2181"
ROOT = "/Roles"
WORKERS_PATH = join(ROOT, "workers")
MASTERS_NUM = 1
TIMEOUT = 10000 def __init__(self, verbose = True):
self.VERBOSE = verbose
self.masters = []
self.is_master = False
self.path = None self.APP_ROOT = "/Applications"
self.APP_CONF = join(self.APP_ROOT,"NginxConf") self.zk = ZKClient(self.ZK_HOST, timeout = self.TIMEOUT)
self.say("login zookeeper successfully!")
# init
self.create_roles_znode()
# register
self.register() def create_roles_znode(self):
"""
create the zookeeper node if not exist
|-Roles
|-workers
"""
nodes = (self.ROOT, self.WORKERS_PATH)
for node in nodes:
if not self.zk.exists(node):
try:
self.zk.create(node, "")
except:
pass @property
def is_slave(self):
return not self.is_master def register(self):
"""
register a node for this worker,znode type : EPHEMERAL | SEQUENCE
|-Roles
|-workers
|-worker000000000x ==>>master
|-worker000000000x+1 ==>>worker
....
"""
self.path = self.zk.create(self.WORKERS_PATH + "/worker", "1", flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE)
self.path = basename(self.path)
self.say("I'm %s" % self.path)
# check who is the master
self.get_master() def get_master(self):
"""
get children, and check who is the smallest child
"""
@watchmethod
def watcher(event):
self.say("child changed, try to get master again.")
self.get_master() children = self.zk.get_children(self.WORKERS_PATH, watcher)
children.sort()
self.say("%s's children: %s" % (self.WORKERS_PATH, children)) # check if I'm master
self.masters = children[:self.MASTERS_NUM]
if self.path in self.masters:
self.is_master = True
self.say("I've become master!")
self.create_app_znode()
else:
self.say("%s is masters, I'm slave" % self.masters) def create_app_znode(self):
"""
create the zookeeper node if not exist
|-Applications
|-NginxConf
"""
nodes = (self.APP_ROOT, self.APP_CONF)
for node in nodes:
if not self.zk.exists(node):
try:
self.say("Create znode [%s] ..."%(node))
self.zk.create(node, "")
except:
pass def create_conf_znode(self,data):
"""
create the zookeeper node's children if not exist,contents is conf data
|-Applications
|-NginxConf
|-item-000000000x => data
"""
self.child_node = join(self.APP_CONF,"conf-")
path = self.zk.create(self.child_node,data, flags=zookeeper.SEQUENCE|zookeeper.PERSISTENT)
self.say("create znode %s"%path) def say(self, msg):
"""
print messages to screen
"""
if self.VERBOSE:
if self.path:
log.info("[ %s(%s) ] %s" % (self.path, "master" if self.is_master else "slave", msg))
else:
log.info(msg) class Watcher:
def __init__(self):
""" Creates a child thread, which returns. The parent
thread waits for a KeyboardInterrupt and then kills
the child thread.
"""
self.child = os.fork()
if self.child == 0:
return
else:
self.watch() def watch(self):
try:
os.wait()
except KeyboardInterrupt:
print ' exit...'
self.kill()
sys.exit() def kill(self):
try:
os.kill(self.child, signal.SIGKILL)
except OSError:
pass def start_zk_worker():
"""
连接到zookeeper执行初始化
"""
gj_zookeeper = GJZookeeper()
th1 = threading.Thread(target = start_agent_worker, name = "thread_1", args = (gj_zookeeper,)) th1.start() th1.join() def start_agent_worker(gj_zookeeper):
"""
监听配置文件变更信息，解析指令
"""
import socket address = ('', 8877)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # s = socket.socket()
s.bind(address)
s.listen(5)
print "listen on port 8877 ..." while True:
ss, addr = s.accept()
print "receive connetcion from " ,addr
content = ""
while True:
try :
data = ss.recv(512)
if not data:
print "close connetcion " ,addr
ss.close()
break
content = content + data
print "receive message from %s : %s"%(addr[0],data)
except Exception,e:
print "receive error from %s : %s"%(addr[0],str(e))
ss.close()
break
parse_trigger_package(content,gj_zookeeper) s.close() def parse_trigger_package(data,gj_zookeeper):
try:
cmd = data.split('|')[0]
content = data.split('|')[1]
except Exception, e:
print "ERROR :",str(e)
return
if cmd == "ADD":
gj_zookeeper.create_conf_znode(content)
else:
pass def main():
Watcher()
start_zk_worker() if __name__ == "__main__":
main()
import time
time.sleep(20000)
配置文件接收应用:zk_appzk_app逻辑如下，它工作在每台worker上
- 初始化连接到zk service
- 获取所有/Applications/NginxConf的znode列表并且设置Watcher
- 找到列表中最大的znode节点，记录它的编号，然后getData获取它的数据，这里就拿到了最新的配置信息
- 每次又Watcher被触发，就获取列表中编号最大的节点，然后把上一次保存的编号和它比较，一旦又更新就重新获取数据，应用到worker，如果相同就不需要到zk上获取数据。
vim zk_app.py
#!/usr/bin/env python2.7
# -*- coding: UTF-8 -*- # author : firefoxbug
# E-Mail : wanghuafire@gmail.com
# Blog : www.firefoxbug.net import logging
from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod import os
import sys
import threading
import signal
import time logging.basicConfig(
level = logging.DEBUG,
format = "[%(asctime)s] %(levelname)-8s %(message)s"
) log = logging class GJZookeeper(object): ZK_HOST = "localhost:2181"
ROOT = "/Roles"
WORKERS_PATH = join(ROOT, "workers")
MASTERS_NUM = 1
TIMEOUT = 10000 def __init__(self, verbose = True):
self.VERBOSE = verbose
self.masters = []
self.is_master = False
self.path = None self.APP_ROOT = "/Applications"
self.APP_CONF = join(self.APP_ROOT,"NginxConf") self.zk = ZKClient(self.ZK_HOST, timeout = self.TIMEOUT)
self.say("login zookeeper successfully!")
# init
self.create_roles_znode()
# register
self.register() def create_roles_znode(self):
"""
create the zookeeper node if not exist
|-Roles
|-workers
"""
nodes = (self.ROOT, self.WORKERS_PATH)
for node in nodes:
if not self.zk.exists(node):
try:
self.zk.create(node, "")
except:
pass @property
def is_slave(self):
return not self.is_master def register(self):
"""
register a node for this worker,znode type : EPHEMERAL | SEQUENCE
|-Roles
|-workers
|-worker000000000x ==>>master
|-worker000000000x+1 ==>>worker
....
"""
self.path = self.zk.create(self.WORKERS_PATH + "/worker", "1", flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE)
self.path = basename(self.path)
self.say("I'm %s" % self.path)
# check who is the master
self.get_master() def get_master(self):
"""
get children, and check who is the smallest child
"""
@watchmethod
def watcher(event):
self.say("child changed, try to get master again.")
self.get_master() children = self.zk.get_children(self.WORKERS_PATH, watcher)
children.sort()
self.say("%s's children: %s" % (self.WORKERS_PATH, children)) # check if I'm master
self.masters = children[:self.MASTERS_NUM]
if self.path in self.masters:
self.is_master = True
self.say("I've become master!")
self.create_app_znode()
else:
self.say("%s is masters, I'm slave" % self.masters) def create_app_znode(self):
"""
create the zookeeper node if not exist
|-Applications
|-NginxConf
"""
nodes = (self.APP_ROOT, self.APP_CONF)
for node in nodes:
if not self.zk.exists(node):
try:
self.say("Create znode [%s] ..."%(node))
self.zk.create(node, "")
except:
pass def create_conf_znode(self,data):
"""
create the zookeeper node's children if not exist,contents is conf data
|-Applications
|-NginxConf
|-conf-000000000x => data
"""
self.child_node = join(self.APP_CONF,"conf-")
path = self.zk.create(self.child_node,data, flags=zookeeper.SEQUENCE|zookeeper.PERSISTENT)
self.say("create znode %s"%path) def say(self, msg):
"""
print messages to screen
"""
if self.VERBOSE:
if self.path:
log.info("[ %s(%s) ] %s" % (self.path, "master" if self.is_master else "slave", msg))
else:
log.info(msg) class Watcher:
def __init__(self):
""" Creates a child thread, which returns. The parent
thread waits for a KeyboardInterrupt and then kills
the child thread.
"""
self.child = os.fork()
if self.child == 0:
return
else:
self.watch() def watch(self):
try:
os.wait()
except KeyboardInterrupt:
print ' exit...'
self.kill()
sys.exit() def kill(self):
try:
os.kill(self.child, signal.SIGKILL)
except OSError:
pass def start_zk_worker():
"""
连接到zookeeper执行初始化
"""
gj_zookeeper = GJZookeeper()
th1 = threading.Thread(target = start_agent_worker, name = "thread_1", args = (gj_zookeeper,)) th1.start() th1.join() def start_agent_worker(gj_zookeeper):
"""
监听配置文件变更信息，解析指令
"""
import socket address = ('', 8877)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # s = socket.socket()
s.bind(address)
s.listen(5)
print "listen on port 8877 ..." while True:
ss, addr = s.accept()
print "receive connetcion from " ,addr
content = ""
while True:
try :
data = ss.recv(512)
if not data:
print "close connetcion " ,addr
ss.close()
break
content = content + data
print "receive message from %s : %s"%(addr[0],data)
except Exception,e:
print "receive error from %s : %s"%(addr[0],str(e))
ss.close()
break
parse_trigger_package(content,gj_zookeeper) s.close() def parse_trigger_package(data,gj_zookeeper):
try:
cmd = data.split('|')[0]
content = data.split('|')[1]
except Exception, e:
print "ERROR :",str(e)
return
if cmd == "ADD":
gj_zookeeper.create_conf_znode(content)
else:
pass def main():
Watcher()
start_zk_worker() if __name__ == "__main__":
main()
import time
time.sleep(20000)
配置文件发送:trigger实现的逻辑很简单，就是模拟给zk_agent发送数据包，格式如下
- 指令|数据
- ADD|helloworld
vim trigger.py
#!/usr/bin/python
import socket
import sys sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 8877))
str = sys.argv[1] + '|' + sys.argv[2]
sock.send(str)
sock.close()
运行
$ python zk_agent.py
$ python zk_app.py
$ python trigger.py
zookeeper适用场景：配置文件同步的更多相关文章
- zookeeper适用场景：分布式锁实现
问题导读:1.zookeeper如何实现分布式锁?2.什么是羊群效应?3.zookeeper如何释放锁? 在zookeeper应用场景有关于分布式集群配置文件同步问题的描述,设想一下如果有100台机器 ...
- zookeeper适用场景：zookeeper解决了哪些问题
问题导读:1.master挂机,传统做法备份必然是以前数据,该如何保证挂机数据与备份数据一致?2.分布式系统如何实现对同一资源的访问,保证数据的强一致性?3.集群中的worker挂了,传统做法是什么? ...
- zookeeper使用场景【转】
分布式网站架构后续:zookeeper技术浅析 Zookeeper是hadoop的一个子项目,虽然源自hadoop,但是我发现zookeeper脱离hadoop的范畴开发分布式框架的运用越来越多. ...
- 【分布式】Zookeeper应用场景
一.前言 在上一篇博客已经介绍了Zookeeper开源客户端的简单实用,本篇讲解Zookeeper的应用场景. 二.典型应用场景 Zookeeper是一个高可用的分布式数据管理和协调框架,并且能够很好 ...
- Zookeeper应用场景和ZAB协议
Zookeeper应用场景 数据发布/订阅(配置中心) 我们平常的开发过程中,经常会碰到这样的需求:系统中需要一些通用的配置信息,如一些运行时的开关.前端需要展示的通知信息.数据库配置信息等等.这些需 ...
- 如何排查sharepoint2010用户配置文件同步服务启动问题
用户配置文件同步服务与 Microsoft Forefront Identity Manager (FIM) 交互,以与外部系统(如目录服务和业务系统)同步配置文件信息.启用用户配置文件同步服务时,将 ...
- zookeeper适用场景：如何竞选Master及代码实现
问题导读:1.如何利用zookeeper保证集群Master可用性和唯一性?2.zookeeper竞选Master包含哪些过程?3.zookeeper竞选Master机制利用了zk哪些特性? 在zoo ...
- 解决sharepoint 2010 用户配置文件同步服务 正在启动
用户配置文件同步服务一直显示“正在启动”,而且无法停止,如下办法可以停止这个服务: 在sharepoint power shell 中执行下面的命令: Get-spserviceinstance 获取 ...
- Hbase 学习(一) hbase配置文件同步
最近在狂啃hadoop的书籍,这部<hbase:权威指南>就进入我的视野里面了,啃吧,因为是英文的书籍,有些个人理解不对的地方,欢迎各位拍砖. HDFS和Hbase配置同步 hbase的配 ...
随机推荐
- ASP.NET 取得 Uri 各项属性值
Uri uri = new Uri("http://www.yoercn.com/aboutus/idea.html");string Host = uri.Host; ...
- 与众不同 windows phone (41) - 8.0 相机和照片: 通过 AudioVideoCaptureDevice 捕获视频和音频
[源码下载] 与众不同 windows phone (41) - 8.0 相机和照片: 通过 AudioVideoCaptureDevice 捕获视频和音频 作者:webabcd 介绍与众不同 win ...
- Programming in Go (Golang) – Setting up a Mac OS X Development Environment
http://www.distilnetworks.com/setup-go-golang-ide-for-mac-os-x/#.V1Byrf50yM8 Programming in Go (Gola ...
- javaweb学习之Servlet开发(二)
javaweb学习总结(六)--Servlet开发(二) 一.ServletConfig讲解 1.1.配置Servlet初始化参数 在Servlet的配置文件web.xml中,可以使用一个或多个< ...
- log4j.xml 配置参数属性level使用心得
jdbc.sqlonly 只显示执行的sql语句.info级才可以显示,debug增加显示java源代码位置. jdbc.sqltiming 显示执行的sql语句以及语句执行时间, ...
- java获取class所在jar
在类库的开发过程中,有些时候为了self-contain的原因,我们希望所有的资源都打包在jar中,但是有些工具好像无法支持从classpasth直接获取比如velocity的模板合并,此时我们就知道 ...
- Vue表单
gitHub地址: https://github.com/lily1010/vue_learn/tree/master/lesson11 一 vue表单 实在是太简单了,直接来个例子 <!DOC ...
- MSCRM 2011 操作大全
CRM字段类型:货币:new Money(Decimal){SQL更新Money类型字段,需要同时更新_base字段,存在汇率差的时候值不同}查找:new EntityReference(object ...
- Convert part to feature command
Search 库主页 Related Links ArcObjects SDK for ArcGIS 10Microsoft Help System Documentation Convert par ...
- android 圆角按钮和按钮颜色
1. android 设置圆角按钮后,按下按钮后,还能改变按钮的颜色 <?xml version="1.0" encoding="UTF-8"?> ...