Python 消息队列snakemq总结

最近学习消息总线zeromq,在网上搜了python实现的消息总线模块,意外发现有个消息队列snakemq,于是拿来研究一下,感觉还是很不错的,入手简单使用也简单(比ice强多了),就是资料太少了,只能自己抠。

一、关于snakemq的官方介绍

1纯python实现,跨平台

2自动重连接

3可靠发送--可配置的消息方式与消息超时方式

4持久化/临时 两种队列

5支持异步 -- poll()

6symmetrical -- 单个TCP连接可用于双工通讯

7多数据库支持 -- SQLite、MongoDB……

8brokerless - 类似ZeroMQ的实现原理

9扩展模块:RPC, bandwidth throttling

以上都是官话,需要自己验证,动手封装了一下,感觉萌萌哒。

二、几个主要问题说明

1支持自动重连,不需要自己动手写心跳逻辑,你只需要关注发送和接收就行

2支持数据持久化,如果开始持久化,在重连之后会自动发送数据。

3数据的接收,snakemq通过提供回调实现,你只需要写个接收方法添加到回调列表里去。

4数据的发送,在此发送的都是bytes类型(二进制),因此需要转换。我在程序中测试的都是文本字符串,使用str.encode(‘utf-8’)转换成bytes,接收时再转换回来。

5术语解释,Connector:类似于socket的TcpClient,Lisenter:类似于socket的TcpServer,每个connector或者listener都一个一个ident标识,发送和接收数据时就知道是谁的数据了。

6使用sqlite持久化时,需要修改源码,sqlite3.connect(filename,check_same_thread = False),用于解决多线程访问sqlite的问题。(会不会死锁?)

7启动持久化时,如果重新连上,则会自动发送,保证可靠。
8
为了封装的需要,数据接收以后,我通过callback方式传送出去。

 

三、代码

说明代码中使用了自定义的日志模块

from common import nxlogger

import snakemqlogger as logger

可替换成logging的。

 

回调类(callbacks.py):

 

# -*- coding:utf-8 -*-

#*********************************************************#

# @@FileName: callbacks.py

# @@Author:
Shelwinnee<Shelwinnee@163.com>

# @@Create Date: 2016-04-14 22:24:14

# @@Modify Date: 2016-04-14 22:24:34

# @@Description: simple synchronized
callbacks helper

#*********************************************************#

 

'''synchronized callback'''

class Callback(object):

   
def __init__(self):

       
self.callbacks = []

 

   
def add(self, func):

       
self.callbacks.append(func)

 

   
def remove(self, func):

       
self.callbacks.remove(func)

 

   
def __call__(self, *args, **kwargs):

       
for callback in self.callbacks:

            callback(*args, **kwargs)

 

Connector类(snakemqConnector.py):

# -*- coding:utf-8 -*-

#*********************************************************#

# @@FileName: snakemqConnector.py

# @@Author:
Shelwinnee<Shelwinnee@163.com>

# @@Create Date: 2016-04-14 22:24:14

# @@Modify Date: 2016-04-14 22:24:34

# @@Description: 

# note:if message persistent is
necessary, please make sure you know the following instruction meaning,

# 'sqlite object created in a thread can
only be used in that same thread'

# this problem's solution: modify the
connection method of sqlite3, open file ('snakemq\storage\sqlite.py') ,and
modify the following code:

# self.conn =
sqlite3.connect(filename,check_same_thread = False)

#*********************************************************#

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

from snakemq.storage.sqlite import
SqliteQueuesStorage

from snakemq.message import
FLAG_PERSISTENT

from common.callbacks import Callback

 

from common import nxlogger

import snakemqlogger as logger

 

class
SnakemqConnector(threading.Thread):

         def
__init__(self, snakemqident = None, remoteIp = "localhost",
remotePort = 9090, persistent = False):

                  super(SnakemqConnector,self).__init__()

                  self.messaging = None

                  self.link = None

                  self.snakemqident = snakemqident

                  self.pktr = None

                  self.remoteIp = remoteIp

                  self.remotePort = remotePort

                  self.persistent = persistent

                  self.on_recv = Callback()

                  self._initConnector()

 

         def
run(self):

                  logger.info("connector
start...")

                 

                  if self.link != None:

                           self.link.loop()

 

                  logger.info("connector
end...")

        

         def
terminate(self):

                  logger.info("connetor
terminating...")

                  if self.link != None:

                           self.link.stop()

                           self.link.cleanup()

                  logger.info("connetor
terminated")

 

         def
on_recv_message(self, conn, ident, message):

                  try:

                           self.on_recv(ident,
message.data.decode('utf-8'))#dispatch received data

                  except Exception as e:

                           logger.error("connector
recv:{0}".format(e))

                           print(e)

 

         '''send
message to dest host named destIdent'''

         def
sendMsg(self, destIdent, byteseq):

                  msg = None

                  if self.persistent:

                           msg
= snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

                  else:

                           msg
= snakemq.message.Message(byteseq, ttl=60)

                  if self.messaging == None:

                           logger.error("connector:messaging
is not initialized, send message failed")

                           return

                  self.messaging.send_message(destIdent,
msg)

 

         '''

        

         '''

         def
_initConnector(self):

                  try:

                           self.link
= snakemq.link.Link()

                           self.link.add_connector((self.remoteIp,
self.remotePort))

 

                           self.pktr
= snakemq.packeter.Packeter(self.link)

 

                           if
self.persistent:

                                    storage =
SqliteQueuesStorage("SnakemqStorage.db")

                                    self.messaging = snakemq.messaging.Messaging(self.snakemqident,
"", self.pktr, storage)

                           else:

                                    self.messaging =
snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

                          

                           self.messaging.on_message_recv.add(self.on_recv_message)

                          

                  except Exception as e:

                           logger.error("connector:{0}".format(e))

                  finally:

                           logger.info("connector[{0}]
loop ended...".format(self.snakemqident))

 

Listener类(snakemqListener.py):

# -*- coding:utf-8 -*-

#*********************************************************#

# @@FileName: snakemqListener.py

# @@Author:
Shelwinnee<Shelwinnee@163.com>

# @@Create Date: 2016-04-14 22:24:14

# @@Modify Date: 2016-04-14 22:24:34

# @@Description: 

# note:if message persistent is
necessary, please make sure you know the following instruction meaning,

# 'sqlite object created in a thread can
only be used in that same thread'

# this problem's solution: modify the
connection method of sqlite3, open file ('snakemq\storage\sqlite.py') ,and
modify the following code:

# self.conn =
sqlite3.connect(filename,check_same_thread = False)

#*********************************************************#

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

 

from common import nxlogger

import snakemqlogger as logger

 

from common.callbacks import Callback

 

class SnakemqListener(threading.Thread):

         def
__init__(self, snakemqident = None, ip = "localhost", port = 9090,
persistent = False):

                  super(SnakemqListener,self).__init__()

                  self.messaging = None

                  self.link = None

                  self.pktr = None

                  self.snakemqident = snakemqident

                  self.ip = ip;

                  self.port = port

                  self.connectors = {}

                  self.on_recv = Callback()

                  self.persistent = persistent

                  self._initlistener()

 

         '''

         thread
run

         '''

         def
run(self):

                  logger.info("listener
start...")

                 

                  if self.link != None:

                           self.link.loop()

 

                  logger.info("listener
end...")

 

         '''

         terminate
snakemq listener thread

         '''

         def
terminate(self):

                  logger.info("listener
terminating...")

                  if self.link != None:

                           self.link.stop()

                           self.link.cleanup()

                  logger.info("listener
terminated")

 

         '''

         receive
message from host named ident

         '''

         def
on_recv_message(self, conn, ident, message):

                  try:

                           self.on_recv(ident,
message.data.decode('utf-8'))#dispatch received data

                           self.sendMsg('bob','hello,{0}'.format(ident).encode('utf-8'))

                  except Exception as e:

                           logger.error("listener
recv:{0}".format(e))

                           print(e)

 

         def
on_drop_message(self, ident, message):

                  print("message dropped",
ident, message)

                  logger.debug("listener:message
dropped,ident:{0},message:{1}".format(ident, message))

 

         '''client
connect'''

         def
on_connect(self, ident):

                  logger.debug("listener:{0}
connected".format(ident))

                  self.connectors[ident] = ident

                  self.sendMsg(ident,
"hello".encode('utf-8'))

 

         '''client
disconnect'''

         def
on_disconnect(self, ident):

                  logger.debug("listener:{0}
disconnected".format(ident))

                  if ident in self.connectors:

                           self.connectors.pop(ident)

 

         '''

         listen
start loop

         '''

         def
_initlistener(self):

                  try:

                           self.link
= snakemq.link.Link()

                           self.link.add_listener((self.ip,
self.port))

 

                           self.pktr
= snakemq.packeter.Packeter(self.link)

                           self.pktr.on_connect.add(self.on_connect)

                           self.pktr.on_disconnect.add(self.on_disconnect)

 

                           if
self.persistent:

                                    storage =
SqliteQueuesStorage("SnakemqStorage.db")

                                    self.messaging =
snakemq.messaging.Messaging(self.snakemqident, "", self.pktr,
storage)

                           else:

                                    self.messaging =
snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

                          

                           self.messaging.on_message_recv.add(self.on_recv_message)

                           self.messaging.on_message_drop.add(self.on_drop_message)

 

                  except Exception as e:

                           logger.error("listener:{0}".format(e))

                  finally:

                           logger.info("listener:loop
ended...")

 

         '''send
message to dest host named destIdent'''

         def
sendMsg(self, destIdent, byteseq):

                  msg = None

                  if self.persistent:

                           msg
= snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

                  else:

                           msg
= snakemq.message.Message(byteseq, ttl=60)

                  if self.messaging == None:

                           logger.error("listener:messaging
is not initialized, send message failed")

                           return

                  self.messaging.send_message(destIdent,
msg)

 

测试代码connector(testSnakeConnector.py):

读取本地一个1M的文件,然后发送给listener,然后listener发回一个hello的信息。

from netComm.snakemq import
snakemqConnector

import time

import sys

import os

 

def received(ident, data):

         print(data)

 

if __name__ == "__main__":

         bob
= snakemqConnector.SnakemqConnector('bob',"10.16.5.45",4002,True)

         bob.on_recv.add(received)

         bob.start()

         try:

                  with
open("testfile.txt",encoding='utf-8') as f:

                           txt
= f.read()

                           for
i in range(100):

                                    bob.sendMsg("niess",txt.encode('utf-8'))

                                    time.sleep(0.1)

         except
Exception as e:

                  print(e)

         time.sleep(5)

         bob.terminate()     

 

测试代码listener(testSnakeListener.py):

from netComm.snakemq import
snakemqListener

import time

 

def received(ident, data):

         filename

"log/recFile{0}.txt".format(time.strftime('%S',time.localtime()))

         file
= open(filename,'w')

         file.writelines(data)

         file.close()

 

if __name__ == "__main__":

         niess
=
snakemqListener.SnakemqListener("niess","10.16.5.45",4002)

         niess.on_recv.add(received)

         niess.start()

         print("niess
start...")

         time.sleep(60)

         niess.terminate()   

         print("niess
end...")

 

四、测试结果

 应该与其它消息类组件对比后出结论。应付一个小项目绝对杠杠滴。

python消息队列snakemq使用总结的更多相关文章

  1. Python消息队列工具 Python-rq 中文教程

    原创文章,作者:Damon付,如若转载,请注明出处:<Python消息队列工具 Python-rq 中文教程>http://www.tiangr.com/python-xiao-xi-du ...

  2. Python消息队列(RabbitMQ)

    RabbitMQ 即一个消息队列,主要是用来实现应用程序的异步和解耦,同时也能起到消息缓冲,消息分发的作用.可维护多个队列,可实现消息的一对一和广播等方式发送 RabbitMQ是一个开源的AMQP实现 ...

  3. python - 消息队列

    消息队列分类 1.先进先出 2.后进先出 3.优先级队列 4.双向队列 1.先进先出 import queue q = queue.Queue(2) #队列最大长度 q.put(11) q.put(2 ...

  4. python消息队列Queue

    实例1:消息队列Queue,不要将文件命名为"queue.py",否则会报异常"ImportError: cannot import name 'Queue'" ...

  5. python 消息队列-rabbitMQ 和 redis介绍使用

    1.rabbitMQ 与ptyhon 进程queue 区别.进程queue 主要用户Python父子进程之间或者统一进程不同子进程.rabbit可以用户不同语言之前的相互交流,socket可以实现同样 ...

  6. python之消息队列

    引言 你是否遇到过两个(多个)系统间需要通过定时任务来同步某些数据?你是否在为异构系统的不同进程间相互调用.通讯的问题而苦恼.挣扎?如果是,那么恭喜你,消息服务让你可以很轻松地解决这些问题.消息服务擅 ...

  7. Python进阶【第二篇】多线程、消息队列queue

    1.Python多线程.多进程 目的提高并发 1.一个应用程序,可以有多进程和多线程 2.默认:单进程,单线程 3.单进程,多线程 IO操作,不占用CPU python的多线程:IO操作,多线程提供并 ...

  8. python 操作消息队列

    图示 其中P指producer,即生产者:C指consumer,即消费者.中间的红色表示消息队列,实例中表现为HELLO队列. 往队列里插入数据前,查看消息队列 $sudo rabbitmqctl l ...

  9. python【第十一篇】消息队列RabbitMQ、缓存数据库Redis

    大纲 1.RabbitMQ 2.Redis 1.RabbitMQ消息队列 1.1 RabbitMQ简介 AMQP,即Advanced Message Queuing Protocol,高级消息队列协议 ...

随机推荐

  1. MINA学习之IoService

    从上一篇文章中知道,IoService出于MINA体系中的底层.IoService将会帮你维护网络交互,接受消息,发送消息,管理Sessions,管理连接Connections等等. IoServic ...

  2. Java NIO使用及原理分析(二)

    在第一篇中,我们介绍了NIO中的两个核心对象:缓冲区和通道,在谈到缓冲区时,我们说缓冲区对象本质上是一个数组,但它其实是一个特殊的数组,缓冲区对象内置了一些机制,能够跟踪和记录缓冲区的状态变化情况,如 ...

  3. BA的广度和深度

    http://insights.thoughtworkers.org/ba-capability-and-development-path-in-thoughtworks/ 作者:ThoughtWor ...

  4. Jafka来源分析——文章

    Kafka它是一个分布式消息中间件,我们可以大致分为三个部分:Producer.Broker和Consumer.当中,Producer负责产生消息并负责将消息发送给Kafka:Broker能够简单的理 ...

  5. oracle17 视图

    oracle的视图 介绍 视图是一个虚拟表是一个表,其内容由查询定义,同真实的表一样,视图包含一系列带有名称的列和行数据.但是,视图并不在数据库中以存储的数据值集形式存在(视图不实际存在).行和列数据 ...

  6. LabVIEW设计模式系列——case结构模拟事件结构

    标准:1.所有按钮的机械动作必须都用释放时触发或者单击时触发,这是为了保证仅仅触发一次动作. 标准:1.使用简单的case结构模拟事件结构.

  7. RedHat7搭建PHP开发环境(Zend Studio)

    下载Zend Studio # wget http://downloads.zend.com/studio-eclipse/13.0.1/ZendStudio-13.0.1-linux.gtk.x86 ...

  8. javascript中的call()和apply应用

    在javascript开发过程中,如果有看过几个javascirpt代码库,就会发现经常使用到call()和apply()函数,call()和aplly()结合javascript允许传递函数名,这种 ...

  9. (转)一个form表单实现提交多个action

    方法一: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4 ...

  10. jquery刷新iframe页面的方法

    1,reload 方法,该方法强迫浏览器刷新当前页面. 语法:location.reload([bForceGet]) 参数: bForceGet, 可选参数, 默认为 false,从客户端缓存里取当 ...