EPH接收Event Hub Message
简介:
使用Python SDK,基于EPH方式接收Azure Event Hub中存储的message,EventProcessorHost()中使用Azure Storage存储offerset等信息。目前版本的SDK对中国区的支持还不是太好,使用起来需要结合中国区的特点指定具体的endpoint。
示例程序:
import asyncio
from azure.eventprocessorhost import AbstractEventProcessor, EPHOptions
from azure.eventprocessorhost import AzureStorageCheckpointLeaseManager
from azure.eventprocessorhost import EventHubConfig
from azure.eventprocessorhost import EventProcessorHost
from azure.storage.blob import BlockBlobService
class EventProcessor(AbstractEventProcessor):
"""
Example Implmentation of AbstractEventProcessor
"""
def __init__(self, params=None):
"""
Init Event processor
"""
self._msg_counter = 0
async def open_async(self, context):
"""
Called by processor host to initialize the event processor.
"""
print(context.partition_id)
async def close_async(self, context, reason):
"""
Called by processor host to indicate that the event processor is being stopped.
(Params) Context:Information about the partition
"""
print("Reason: " + reason)
print("context.partition_id: " + context.partition_id)
print("context.offset: " + context.offset)
print("context.sequence_number: {}".format(context.sequence_number))
async def process_events_async(self, context, messages):
"""
Called by the processor host when a batch of events has arrived.
This is where the real work of the event processor is done.
(Params) Context: Information about the partition, Messages: The events to be processed.
"""
print(context.partition_id)
print(messages)
print("-----")
await context.checkpoint_async()
async def process_error_async(self, context, error):
"""
Called when the underlying client experiences an error while receiving.
EventProcessorHost will take care of recovering from the error and
continuing to pump messages,so no action is required from
(Params) Context: Information about the partition, Error: The error that occured.
"""
print(repr(error))
# 注意如果消息较多,请增加sleep的时间,防止程序提前终止
async def wait_and_close(host):
"""
Run EventProcessorHost for 6 minutes then shutdown.
"""
await asyncio.sleep(360)
await host.close_async()
try:
loop = asyncio.get_event_loop()
# Storage Account Credentials
STORAGE_ACCOUNT_NAME = "yuvmtestdiag489"
STORAGE_KEY = "OgNFVB93P2oWCKcJaNOt4kuW0lf911D11v6NiHcDC6R0SdiqUJYlP8UdW7U4p0UzSnreII+7BnS9Cy1VsMSqGg=="
LEASE_CONTAINER_NAME = "leases5"
# Eventhub config and storage manager
EH_CONFIG = EventHubConfig('cznamespace', 'yutaoeventhubtest', 'RootManageSharedAccessKey',
'wnwcU+VjKLGgNUwiecwAyHb9gAVPDIC+RvXu2JcDINc=', consumer_group="$default")
# 注意务必对key进行urlencode编码
ADDRESS = ("amqps://"
"RootManageSharedAccessKey"
":"
"wnwcU%2bVjKLGgNUwiecwAyHb9gAVPDIC%211vXu2JcDINc%3d"
"@"
"cznamespace.servicebus.chinacloudapi.cn"
"/"
"yutaoeventhubtest")
EH_CONFIG.client_address = ADDRESS
STORAGE_MANAGER = AzureStorageCheckpointLeaseManager(STORAGE_ACCOUNT_NAME,
STORAGE_KEY,
LEASE_CONTAINER_NAME,
storage_blob_prefix="blob.core.chinacloudapi.cn")
eph_options = EPHOptions()
#Event loop and host
LOOP = asyncio.get_event_loop()
HOST = EventProcessorHost(EventProcessor, EH_CONFIG, STORAGE_MANAGER, eph_options=eph_options,
ep_params=["param1", "param2"], loop=LOOP)
# 重新设置一遍防止SDK擦除了之前的配置
STORAGE_MANAGER.storage_client = BlockBlobService(account_name=STORAGE_ACCOUNT_NAME,
account_key=STORAGE_KEY,
endpoint_suffix="core.chinacloudapi.cn")
tasks = asyncio.gather(
HOST.open_async(),
wait_and_close(HOST))
loop.run_until_complete(tasks)
except KeyboardInterrupt:
# Canceling pending tasks and stopping the loop
for task in asyncio.Task.all_tasks():
task.cancel()
loop.run_forever()
tasks.exception()
finally:
loop.stop()
注意: 连接信息已经失效,仅供格式参考。
测试结果:
1
[<azure.eventhub.EventData object at 0x00000223AE556EF0>, <azure.eventhub.EventData object at 0x00000223AE556F98>, <azure.eventhub.EventData object at 0x00000223AE4DBA20>, <azure.eventhub.EventData object at 0x00000223AE4DB400>, <azure.eventhub.EventData object at 0x00000223AE4DBC88>, <azure.eventhub.EventData object at 0x00000223AE4DBAC8>, <azure.eventhub.EventData object at 0x00000223AE4FD7F0>, <azure.eventhub.EventData object at 0x00000223AE4FD6D8>, <azure.eventhub.EventData object at 0x00000223AE4FDBE0>, <azure.eventhub.EventData object at 0x00000223AE4FD048>]
-----
1
[<azure.eventhub.EventData object at 0x00000223AE4FD278>, <azure.eventhub.EventData object at 0x00000223AE4FDB70>, <azure.eventhub.EventData object at 0x00000223AE4FD668>, <azure.eventhub.EventData object at 0x00000223AE4FDAC8>, <azure.eventhub.EventData object at 0x00000223AE4FDB38>, <azure.eventhub.EventData object at 0x00000223AE4FDF60>, <azure.eventhub.EventData object at 0x00000223AE57AB38>, <azure.eventhub.EventData object at 0x00000223AE57A470>, <azure.eventhub.EventData object at 0x00000223AE57A748>]
-----
Reason: Shutdown
context.partition_id: 0
context.offset: 978600
context.sequence_number: 17559
Reason: Shutdown
context.partition_id: 1
context.offset: 978544
context.sequence_number: 17558
Process finished with exit code 0
参考示例:
azure-event-hubs-python/examples/eph.py
eph.py stop when the message not all received
EPH接收Event Hub Message的更多相关文章
- Azure Event Hub 技术研究系列3-Event Hub接收事件
上篇博文中,我们通过编程的方式介绍了如何将事件消息发送到Azure Event Hub: Azure Event Hub 技术研究系列2-发送事件到Event Hub 本篇文章中,我们继续:从Even ...
- Event Hub小白入门指南
Event Hub事件中心 本文的目的在于用最白的大白话,让你从“完全不懂”开始,理解什么是分布式大数据流平台Event Hub,并且理解它的关键概念,并且初步理解其收发数据API. 定义,Event ...
- 【Azure 事件中心】Azure Event Hub 新功能尝试 -- 异地灾难恢复 (Geo-Disaster Recovery)
问题描述 关于Event Hub(事件中心)的灾备方案,大多数就是新建另外一个备用的Event Hub,当主Event Hub出现不可用的情况时,就需要切换到备Event Hub上. 而在切换的过程中 ...
- 【Azure 事件中心】 org.slf4j.Logger 收集 Event Hub SDK(Java) 输出日志并以文件形式保存
问题描述 在使用Azure Event Hub的SDK时候,常规情况下,发现示例代码中并没有SDK内部的日志输出.因为在Java项目中,没有添加 SLF4J 依赖,已致于在启动时候有如下提示: SLF ...
- Azure Event Hub 技术研究系列2-发送事件到Event Hub
上篇博文中,我们介绍了Azure Event Hub的一些基本概念和架构: Azure Event Hub 技术研究系列1-Event Hub入门篇 本篇文章中,我们继续深入研究,了解Azure Ev ...
- Azure IoT Hub和Event Hub相关的技术系列-索引篇
Azure IoT Hub和Event Hub相关的技术系列,最近已经整理了不少了,统一做一个索引链接,置顶. Azure IoT 技术研究系列1-入门篇 Azure IoT 技术研究系列2-设备注册 ...
- Azure Event hub usage
1. create event hub on azure watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/40 ...
- 【事件中心 Azure Event Hub】Event Hub Java SDK的消费端出现不消费某一个分区中数据的情况,出现IdleTimerExpired错误消息记录
问题情形 使用Java SDK编写的Event Hub消费端应用,随机性遇见了某个分区没有消费消息的情况,在检查日志时候,有发现IdelTimeExpired的错误记录.在重启应用后,连接EventH ...
- 【事件中心 Azure Event Hub】关于EventHub中出现Error时候的一些问题(偶发错误,EventHub后台升级,用户端错误,Retry机制的重要性)
请问对偶发的定义是多少频率? 针对偶发的定义,主要是看发生的时间非常短,次数极少(如 10次以内),并且发生的时候EventHub其他分区或其他连接都是正常接收和发送数据.所以对于频率是没有明确的定义 ...
随机推荐
- pyinstaller对多进程程序的打包
在使用python的第三方库pyinstaller对多进程程序进行打包时,程序不能正常的运行,但是后台却有多个进程一直在使用资源. 解决方法很简单,在if __name__ == '__main__' ...
- Dfs【p4906】小奔关闹钟
Background 由于今天是星期一,闹钟准时响了,由于小奔太困了,所以她想关停闹钟. Description 可是,他的闹钟电路太复杂了,有很多个开关,每个开关都连着其他开关,其他开关又连着更多的 ...
- ASP.NET Core 2.2 基础知识(十二) 发送 HTTP 请求
可以注册 IHttpClientFactory 并将其用于配置和创建应用中的 HttpClient 实例. 这能带来以下好处: 提供一个中心位置,用于命名和配置逻辑 HttpClient 实例. 例如 ...
- ant脚本调用.bat文件
build.xml内容如下: <project name="example" default="test"> <target name=&qu ...
- 【动态规划】bzoj1638 [Usaco2007 Mar]Cow Traffic 奶牛交通
设f[u]为从度数0到u的路径条数,f2[u]为从u到n的路径条数. ans=max{f[x[i]]*f2[y[i]]}(1<=i<=m). #include<cstdio> ...
- 【状压DP】旅行商问题
给定一张带权有向图,要求从顶点0出发,经过每个结点恰好一次后再返回0,求边权和的最小值. 2<=n<=15 0<=d(i,j)<=1000 样例 5 80 1 30 3 41 ...
- 【树链剖分】【树状数组】【最近公共祖先】【块状树】bzoj3631 [JLOI2014]松鼠的新家
裸题,树状数组区间修改+单点查询.当然要稍微讨论一下链的左右端点是否修改的情况咯. #include<cstdio> #include<algorithm> #include& ...
- 【线段树】bzoj3747 [POI2015]Kinoman
题解:http://www.cnblogs.com/zyfzyf/p/4105184.html 一.下传标记写法 #include<cstdio> #include<algorith ...
- springmvc与前端数据交互实例
一.从页面接收参数 Spring MVC接收请求提交的参数值的几种方法: 使用HttpServletRequest获取. @RequestMapping("/login.do" ...
- TabHost
(一) 知识点:id使用系统自带 1.效果图: 2.布局 activity_main.xml <?xml version="1.0" encoding="utf-8 ...