zookeeper适用场景：配置文件同步
问题导读：
1.本文三个角色之间是什么关系？
2.三个角色的作用是什么？
3.如何代码实现这三个角色的作用？
在 zookeeper适用场景：zookeeper解决了哪些问题有关于分布式集群配置文件同步问题的描述，本文介绍如何把zk应用到配置文件分发的场景。
假设有三个角色
- trigger：发布最新的配置文件数据，发送指令和数据给zk_agent，实现是下面的trigger.py
- zk_agent：接收来自trigger.py触发的指令和数据，并且把数据更新到zk service上，从而触发zk_app来获取最新的配置数据，实现是下面的zk_agent.py
- zk_app：部署在每台worker上的注册监听zk中配置文件所在znode的变化，从而获取最新的配置文件，应用到worker中，实现是下面的zk_app.py
<ignore_js_op>
配置文件同步到:zk_agent实现逻辑：
* 初始化连接到zk service，首先竞选出master
* master create一个配置文件管理的PERSISTENT类型的znode，比如是/Applications/NginxConf，
* 启动一个线程，专门接收trigger发送的指令，一收到指令，就create一个"conf-"开头的SEQUENCE|PERSISTENT类型的节点，指定znode数据内容是从trigger收到的数据
* 第一次刚更新会创建节点/Applications/NginxConf/conf-0000000000,以后每次更新新的配置文件编号就会增大。
vim zk_agent.py
#!/usr/bin/env python2.7
# -*- coding: UTF-8 -*- # author : firefoxbug
# E-Mail : wanghuafire@gmail.com
# Blog : www.firefoxbug.net import logging
from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod import os
import sys
import threading
import signal
import time logging.basicConfig(
level = logging.DEBUG,
format = "[%(asctime)s] %(levelname)-8s %(message)s"
) log = logging class GJZookeeper(object): ZK_HOST = "localhost:2181"
ROOT = "/Roles"
WORKERS_PATH = join(ROOT, "workers")
MASTERS_NUM = 1
TIMEOUT = 10000 def __init__(self, verbose = True):
self.VERBOSE = verbose
self.masters = []
self.is_master = False
self.path = None self.APP_ROOT = "/Applications"
self.APP_CONF = join(self.APP_ROOT,"NginxConf") self.zk = ZKClient(self.ZK_HOST, timeout = self.TIMEOUT)
self.say("login zookeeper successfully!")
# init
self.create_roles_znode()
# register
self.register() def create_roles_znode(self):
"""
create the zookeeper node if not exist
|-Roles
|-workers
"""
nodes = (self.ROOT, self.WORKERS_PATH)
for node in nodes:
if not self.zk.exists(node):
try:
self.zk.create(node, "")
except:
pass @property
def is_slave(self):
return not self.is_master def register(self):
"""
register a node for this worker,znode type : EPHEMERAL | SEQUENCE
|-Roles
|-workers
|-worker000000000x ==>>master
|-worker000000000x+1 ==>>worker
....
"""
self.path = self.zk.create(self.WORKERS_PATH + "/worker", "1", flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE)
self.path = basename(self.path)
self.say("I'm %s" % self.path)
# check who is the master
self.get_master() def get_master(self):
"""
get children, and check who is the smallest child
"""
@watchmethod
def watcher(event):
self.say("child changed, try to get master again.")
self.get_master() children = self.zk.get_children(self.WORKERS_PATH, watcher)
children.sort()
self.say("%s's children: %s" % (self.WORKERS_PATH, children)) # check if I'm master
self.masters = children[:self.MASTERS_NUM]
if self.path in self.masters:
self.is_master = True
self.say("I've become master!")
self.create_app_znode()
else:
self.say("%s is masters, I'm slave" % self.masters) def create_app_znode(self):
"""
create the zookeeper node if not exist
|-Applications
|-NginxConf
"""
nodes = (self.APP_ROOT, self.APP_CONF)
for node in nodes:
if not self.zk.exists(node):
try:
self.say("Create znode [%s] ..."%(node))
self.zk.create(node, "")
except:
pass def create_conf_znode(self,data):
"""
create the zookeeper node's children if not exist,contents is conf data
|-Applications
|-NginxConf
|-item-000000000x => data
"""
self.child_node = join(self.APP_CONF,"conf-")
path = self.zk.create(self.child_node,data, flags=zookeeper.SEQUENCE|zookeeper.PERSISTENT)
self.say("create znode %s"%path) def say(self, msg):
"""
print messages to screen
"""
if self.VERBOSE:
if self.path:
log.info("[ %s(%s) ] %s" % (self.path, "master" if self.is_master else "slave", msg))
else:
log.info(msg) class Watcher:
def __init__(self):
""" Creates a child thread, which returns. The parent
thread waits for a KeyboardInterrupt and then kills
the child thread.
"""
self.child = os.fork()
if self.child == 0:
return
else:
self.watch() def watch(self):
try:
os.wait()
except KeyboardInterrupt:
print ' exit...'
self.kill()
sys.exit() def kill(self):
try:
os.kill(self.child, signal.SIGKILL)
except OSError:
pass def start_zk_worker():
"""
连接到zookeeper执行初始化
"""
gj_zookeeper = GJZookeeper()
th1 = threading.Thread(target = start_agent_worker, name = "thread_1", args = (gj_zookeeper,)) th1.start() th1.join() def start_agent_worker(gj_zookeeper):
"""
监听配置文件变更信息，解析指令
"""
import socket address = ('', 8877)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # s = socket.socket()
s.bind(address)
s.listen(5)
print "listen on port 8877 ..." while True:
ss, addr = s.accept()
print "receive connetcion from " ,addr
content = ""
while True:
try :
data = ss.recv(512)
if not data:
print "close connetcion " ,addr
ss.close()
break
content = content + data
print "receive message from %s : %s"%(addr[0],data)
except Exception,e:
print "receive error from %s : %s"%(addr[0],str(e))
ss.close()
break
parse_trigger_package(content,gj_zookeeper) s.close() def parse_trigger_package(data,gj_zookeeper):
try:
cmd = data.split('|')[0]
content = data.split('|')[1]
except Exception, e:
print "ERROR :",str(e)
return
if cmd == "ADD":
gj_zookeeper.create_conf_znode(content)
else:
pass def main():
Watcher()
start_zk_worker() if __name__ == "__main__":
main()
import time
time.sleep(20000)
配置文件接收应用:zk_appzk_app逻辑如下，它工作在每台worker上
- 初始化连接到zk service
- 获取所有/Applications/NginxConf的znode列表并且设置Watcher
- 找到列表中最大的znode节点，记录它的编号，然后getData获取它的数据，这里就拿到了最新的配置信息
- 每次又Watcher被触发，就获取列表中编号最大的节点，然后把上一次保存的编号和它比较，一旦又更新就重新获取数据，应用到worker，如果相同就不需要到zk上获取数据。
vim zk_app.py
#!/usr/bin/env python2.7
# -*- coding: UTF-8 -*- # author : firefoxbug
# E-Mail : wanghuafire@gmail.com
# Blog : www.firefoxbug.net import logging
from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod import os
import sys
import threading
import signal
import time logging.basicConfig(
level = logging.DEBUG,
format = "[%(asctime)s] %(levelname)-8s %(message)s"
) log = logging class GJZookeeper(object): ZK_HOST = "localhost:2181"
ROOT = "/Roles"
WORKERS_PATH = join(ROOT, "workers")
MASTERS_NUM = 1
TIMEOUT = 10000 def __init__(self, verbose = True):
self.VERBOSE = verbose
self.masters = []
self.is_master = False
self.path = None self.APP_ROOT = "/Applications"
self.APP_CONF = join(self.APP_ROOT,"NginxConf") self.zk = ZKClient(self.ZK_HOST, timeout = self.TIMEOUT)
self.say("login zookeeper successfully!")
# init
self.create_roles_znode()
# register
self.register() def create_roles_znode(self):
"""
create the zookeeper node if not exist
|-Roles
|-workers
"""
nodes = (self.ROOT, self.WORKERS_PATH)
for node in nodes:
if not self.zk.exists(node):
try:
self.zk.create(node, "")
except:
pass @property
def is_slave(self):
return not self.is_master def register(self):
"""
register a node for this worker,znode type : EPHEMERAL | SEQUENCE
|-Roles
|-workers
|-worker000000000x ==>>master
|-worker000000000x+1 ==>>worker
....
"""
self.path = self.zk.create(self.WORKERS_PATH + "/worker", "1", flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE)
self.path = basename(self.path)
self.say("I'm %s" % self.path)
# check who is the master
self.get_master() def get_master(self):
"""
get children, and check who is the smallest child
"""
@watchmethod
def watcher(event):
self.say("child changed, try to get master again.")
self.get_master() children = self.zk.get_children(self.WORKERS_PATH, watcher)
children.sort()
self.say("%s's children: %s" % (self.WORKERS_PATH, children)) # check if I'm master
self.masters = children[:self.MASTERS_NUM]
if self.path in self.masters:
self.is_master = True
self.say("I've become master!")
self.create_app_znode()
else:
self.say("%s is masters, I'm slave" % self.masters) def create_app_znode(self):
"""
create the zookeeper node if not exist
|-Applications
|-NginxConf
"""
nodes = (self.APP_ROOT, self.APP_CONF)
for node in nodes:
if not self.zk.exists(node):
try:
self.say("Create znode [%s] ..."%(node))
self.zk.create(node, "")
except:
pass def create_conf_znode(self,data):
"""
create the zookeeper node's children if not exist,contents is conf data
|-Applications
|-NginxConf
|-conf-000000000x => data
"""
self.child_node = join(self.APP_CONF,"conf-")
path = self.zk.create(self.child_node,data, flags=zookeeper.SEQUENCE|zookeeper.PERSISTENT)
self.say("create znode %s"%path) def say(self, msg):
"""
print messages to screen
"""
if self.VERBOSE:
if self.path:
log.info("[ %s(%s) ] %s" % (self.path, "master" if self.is_master else "slave", msg))
else:
log.info(msg) class Watcher:
def __init__(self):
""" Creates a child thread, which returns. The parent
thread waits for a KeyboardInterrupt and then kills
the child thread.
"""
self.child = os.fork()
if self.child == 0:
return
else:
self.watch() def watch(self):
try:
os.wait()
except KeyboardInterrupt:
print ' exit...'
self.kill()
sys.exit() def kill(self):
try:
os.kill(self.child, signal.SIGKILL)
except OSError:
pass def start_zk_worker():
"""
连接到zookeeper执行初始化
"""
gj_zookeeper = GJZookeeper()
th1 = threading.Thread(target = start_agent_worker, name = "thread_1", args = (gj_zookeeper,)) th1.start() th1.join() def start_agent_worker(gj_zookeeper):
"""
监听配置文件变更信息，解析指令
"""
import socket address = ('', 8877)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # s = socket.socket()
s.bind(address)
s.listen(5)
print "listen on port 8877 ..." while True:
ss, addr = s.accept()
print "receive connetcion from " ,addr
content = ""
while True:
try :
data = ss.recv(512)
if not data:
print "close connetcion " ,addr
ss.close()
break
content = content + data
print "receive message from %s : %s"%(addr[0],data)
except Exception,e:
print "receive error from %s : %s"%(addr[0],str(e))
ss.close()
break
parse_trigger_package(content,gj_zookeeper) s.close() def parse_trigger_package(data,gj_zookeeper):
try:
cmd = data.split('|')[0]
content = data.split('|')[1]
except Exception, e:
print "ERROR :",str(e)
return
if cmd == "ADD":
gj_zookeeper.create_conf_znode(content)
else:
pass def main():
Watcher()
start_zk_worker() if __name__ == "__main__":
main()
import time
time.sleep(20000)
配置文件发送:trigger实现的逻辑很简单，就是模拟给zk_agent发送数据包，格式如下
- 指令|数据
- ADD|helloworld
vim trigger.py
#!/usr/bin/python
import socket
import sys sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 8877))
str = sys.argv[1] + '|' + sys.argv[2]
sock.send(str)
sock.close()
运行
$ python zk_agent.py
$ python zk_app.py
$ python trigger.py
zookeeper适用场景：配置文件同步的更多相关文章
- zookeeper适用场景：分布式锁实现
问题导读:1.zookeeper如何实现分布式锁?2.什么是羊群效应?3.zookeeper如何释放锁? 在zookeeper应用场景有关于分布式集群配置文件同步问题的描述,设想一下如果有100台机器 ...
- zookeeper适用场景：zookeeper解决了哪些问题
问题导读:1.master挂机,传统做法备份必然是以前数据,该如何保证挂机数据与备份数据一致?2.分布式系统如何实现对同一资源的访问,保证数据的强一致性?3.集群中的worker挂了,传统做法是什么? ...
- zookeeper使用场景【转】
分布式网站架构后续:zookeeper技术浅析 Zookeeper是hadoop的一个子项目,虽然源自hadoop,但是我发现zookeeper脱离hadoop的范畴开发分布式框架的运用越来越多. ...
- 【分布式】Zookeeper应用场景
一.前言 在上一篇博客已经介绍了Zookeeper开源客户端的简单实用,本篇讲解Zookeeper的应用场景. 二.典型应用场景 Zookeeper是一个高可用的分布式数据管理和协调框架,并且能够很好 ...
- Zookeeper应用场景和ZAB协议
Zookeeper应用场景 数据发布/订阅(配置中心) 我们平常的开发过程中,经常会碰到这样的需求:系统中需要一些通用的配置信息,如一些运行时的开关.前端需要展示的通知信息.数据库配置信息等等.这些需 ...
- 如何排查sharepoint2010用户配置文件同步服务启动问题
用户配置文件同步服务与 Microsoft Forefront Identity Manager (FIM) 交互,以与外部系统(如目录服务和业务系统)同步配置文件信息.启用用户配置文件同步服务时,将 ...
- zookeeper适用场景：如何竞选Master及代码实现
问题导读:1.如何利用zookeeper保证集群Master可用性和唯一性?2.zookeeper竞选Master包含哪些过程?3.zookeeper竞选Master机制利用了zk哪些特性? 在zoo ...
- 解决sharepoint 2010 用户配置文件同步服务 正在启动
用户配置文件同步服务一直显示“正在启动”,而且无法停止,如下办法可以停止这个服务: 在sharepoint power shell 中执行下面的命令: Get-spserviceinstance 获取 ...
- Hbase 学习(一) hbase配置文件同步
最近在狂啃hadoop的书籍,这部<hbase:权威指南>就进入我的视野里面了,啃吧,因为是英文的书籍,有些个人理解不对的地方,欢迎各位拍砖. HDFS和Hbase配置同步 hbase的配 ...
随机推荐
- 使用win2d实现萤火虫粒子效果
这几天我在做游戏的标题画面,需要实现随机飞舞的萤火虫.萤火虫会闪烁黄绿色的光,远近不同,并且飞出屏幕边界不久会自动飞回来. 我前一阵子用win2d写了个简单的游戏引擎(现在还是alpha阶段),用它可 ...
- 重新想象 Windows 8.1 Store Apps (90) - 通信的新特性: 通过 HttpBaseProtocolFilter 实现 http 请求的缓存控制，以及 cookie 读写; 自定义 HttpFilter; 其他
[源码下载] 重新想象 Windows 8.1 Store Apps (90) - 通信的新特性: 通过 HttpBaseProtocolFilter 实现 http 请求的缓存控制,以及 cooki ...
- MySQL备份与恢复
1．通过mysqldump命令进行数据备份 mysql数据库的备份采用直接打包数据库文件夹(注:InnoDB引擎不支持直接拷贝数据文件备份数据库,由于使用InnoDB引擎的库或表是不能使用此种方式): ...
- HDU 1069---背包---Monkey and Banana
HDU 1069 Description A group of researchers are designing an experiment to test the IQ of a monkey. ...
- JavaWebSession
一.Session简单介绍 在WEB开发中,服务器可以为每个用户浏览器创建一个会话对象(session对象),注意:一个浏览器独占一个session对象(默认情况下).因此,在需要保存用户数据时,服务 ...
- mysql zip install
1.Question Description: 1.1 version: mysql-5.7.11-64 1.2 form: zip file 1.3 >mysqld --install (su ...
- 机器学习实战 - 读书笔记(07) - 利用AdaBoost元算法提高分类性能
前言 最近在看Peter Harrington写的"机器学习实战",这是我的学习笔记,这次是第7章 - 利用AdaBoost元算法提高分类性能. 核心思想 在使用某个特定的算法是, ...
- linux服务器如何设置目录权限，让开发只能在测试目录下开发，不在线上目录上开发
当一台服务器上,既有测试环境,也有生成的环境,开发需要在线上测试,如果开发生产环境的权限,那开发容易误操作 需求如下: (1)生产环境的代码,必须有专用的账号登陆进行管理 (2)开发测试环境的代码,开 ...
- java 多线程断点下载demo
源码链接 import java.io.BufferedReader; import java.io.File; import java.io.FileInputStream; import java ...
- LCS修改版（Longest Common Subsequence 最长公共子序列）
题目描述 作为一名情报局特工,Nova君(2号)有着特殊的传达情报的技巧.为了避免被窃取情报,每次传达时,他都会发出两句旁人看来意义不明话,实际上暗号已经暗含其中.解密的方法很简单,分别从两句话里删掉 ...