让你的saga更具有可伸缩性(Scaling NServiceBus Sagas)
https://lostechies.com/jimmybogard/2013/03/26/scaling-nservicebus-sagas/
当我们使用NServiceBus sagas (process managers)的时候,特别是在一个存在大量消息的情况下,我们常常会碰到下面两个问题:
- 死锁
- 资源不足
这是因为saga的设计师:
- 一个saga对应一个saga实体(导致死锁)
- 一个saga处理的所有的消息全部传递到同一个终点(endpoint )(导致性能瓶颈)
每个问题对应各自的解决方式,但是和之前一样,我们也会用真实世界中的例子来观察我们的问题。这些问题都被记录在企业集成模式()中,所以你如果遇到这些问题不必要惊讶。
死锁
如果我们的saga的并行需求突然激增就很容易产生死锁。回到我们的麦当劳例子,我们发布了一个消息然后等待所有的工作台报告:
When looking at NServiceBus sagas (process managers), especially at high volume of messages, we often run into two main problems:
- Deadlocks
- Starvation
This is because of the fundamental (default) design of sagas is that:
- A single saga shares a single saga entity (causing deadlocks)
- All messages handled by a saga are delivered to the same endpoint (causing a bottleneck)
Two very different problems, with very different solutions. But, as per usual, we’ll look at how we might handle these situations in the real life to see how our virtual world should behave. Both of these problems (amongst others) are called out in the Enterprise Integration Patterns book, so it shouldn’t be a surprise if you run into these issues. Not as clear, however, is how to deal with them.
Deadlocks
Deadlocks in sagas are pretty easy to run into if we start pumping up the concurrency on our sagas. Back in our McDonald’s example, we published a message out and waited for all of the stations to report back:
In our restaurant, there’s only room for one person to examine a single tray at a time. When someone finishes a food item, they have to go look at all the trays. But what if someone is putting food on my tray? Do I push them out of the way?
Probably not, I need to wait for them to finish. It turns out that this is what happens in our sagas – we can only really allow one person at a time to affect a saga instance. The way NServiceBus can handle this by default is by adjusting the isolation level for Serializable transactions. This ensures that only one person looks at a tray at any given time.
What we don’t want to have happen is the fries person look at the tray, see a missing sandwich, and decide that the order isn’t done. The sandwich person does the same thing in reverse – look at the tray, see missing fries, and decide the order isn’t done.
It turns out that our Serializable isolation level has unintended consequences – we might lock more trays (or ALL of the trays) unintentionally. If there are more than one message that can start our saga, then effectively every employee has to look at all the trays to see if our tray is already on the counter, or we’re the first one and need to put a new one out. But if someone is also doing the same thing – multiple trays for the same order might go out!
We don’t want to block the entire counter for our order, so we can adjust our scheme slightly. If we implement a simple versioning scheme, we can simply have the employees look at a version number on the receipt, make their changes, and bump the version number with a little tally mark. If the tally mark has gone up since we last looked at the tray, we’ll just re-do our work – something has changed, so let’s reevaluate our decisions.
To get around the problem of too many trays for the same order, we can rely on a third party to ensure we don’t have too many trays. If when anyone places a tray down, the supervisor enforces this rule that only one tray per order can exist, they’ll ask us to put our tray back up and redo our work.
Two things we want to do then:
- Introduce a unique constraint on the correlation identifier to prevent duplicates
- Use optimistic concurrency to allow us to isolate just our own tray (and not affect anyone else)
This is a rather straightforward fix – and in fact, NServiceBus defaults to using optimistic concurrency in 4.0 (now in beta). This was a rather easy fix, but what about our other problem of the bottleneck?
Starvation and bottlenecks
Back in my early days of school, I worked at a fast food restaurant during breaks (surprise, surprise). I worked the graveyard shift of a 24-hour burger joint, and it was just me and one other person manning the entire place. From 10PM to 6AM, just two of us.
One thing that I found pretty early was that we had quite a few “regulars”. These were folks that every morning, came in and ordered the same thing at around the same time. Because it was quite early, they could rely on little to no contention for resources, and the queue was more or less empty.
Most of the time.
But every once in a while, we would get someone ordering food for a large group of people. We didn’t do catering, but we did sell breakfast tacos. Handling one or two at a time wasn’t a problem, but someone coming in and ordering 50 tacos – that’s a problem. Our process looked something like this:
The problem was, because there were only two of us, most messages flowed through exactly one channel. In NServiceBus Sagas, this is what happens as well – all messages for a saga are delivered to a single endpoint. We might have the situation like this:
Saga Queue
Prep Taco – #1
Prep Taco – #1
Prep Taco – #1
Prep Taco – #1
Prep Taco – #1
Prep Taco – #1
Prep Taco – #1
Prep Pancakes – #2
My cook has dozens of tacos to go through before getting to that second order, and because that person packs all the food and preps all the food, all items from the first order must be both prepped and packed before order #2 is complete:
Saga Queue
Pack Taco – #1
Pack Taco – #1
Pack Taco – #1
Pack Taco – #1
Pack Taco – #1
Pack Taco – #1
Pack Taco – #1
Pack Pancakes – #2
That’s not a huge problem, except we have both packing and prepping done by a single person. Another order comes in, the prepping for that one interferes with the others. Additionally, items are sitting in the saga queue waiting to get packed while prepping finishes.
In many NServiceBus sagas, the sagas themselves don’t perform the actual work. They’re instead the controllers/coordinators, making decisions about next steps. But all the messages are delivered to the exact same queue, effectively creating a bottleneck. I’m waiting for all prepping to be done before moving on to any packing. The time it takes to this entire set of orders is basically the time it takes to prep all items plus the time it takes to pack all items.
The problem is our saga is modeled rather strangely. In the real world, long-running business processes, split into multiple activities/steps, are performed many times by different people. Effectively, different channels/endpoints/queues. But our saga shoved everyone back into one queue!
What we did in our taco tragedy above was that I came off the line from taking orders and performed the prep stage. For N items, we took a total time of N+1 or so. We had two queues going for the one saga:
Prep Queue
Prep Taco – #1
Prep Pancakes – #2
Pack Queue
Pack Taco – #1
Pack Taco – #1
The total number of items in the Pack queue was always small – packing was easy and quick. By splitting the saga handling into separate queues, we ensured that an overload in one type of message didn’t choke out all the others. This isn’t the default way of managing long-running processes in NServiceBus though, we have to set this up ourselves.
When dealing with the Process Manager pattern, it’s important to keep in mind what this pattern brings to the table. We’re able to support complex message flow, but at the price of potential bottlenecks and deadlocks.
Next time – a better pattern for linear processes that doesn’t bring in deadlocks and bottlenecks, the routing slip pattern.
让你的saga更具有可伸缩性(Scaling NServiceBus Sagas)的更多相关文章
- 【NServiceBus】什么是Saga,Saga能做什么
前言 Saga单词翻译过来是指尤指古代挪威或冰岛讲述冒险经历和英雄业绩的长篇故事,对,这里强调长篇故事.许多系统都存在长时间运行的业务流程,NServiceBus使用基于事件驱动的 ...
- Saga的实现模式——观察者(Saga implementation patterns – Observer)
https://lostechies.com/jimmybogard/2013/03/11/saga-implementation-patterns-observer/ 侵删. NServiceBus ...
- NServiceBus+Saga开发分布式应用
前言 当你在处理异步消息时,每个单独的消息处理程序都是一个单独的handler,每个handler之间互不影响.这时如果一个消息依赖另一个消息的状态呢? 这时业务逻辑怎么处理? ...
- Python cPickle模块
新博客地址:http://gorthon.sinaapp.com/ 持久性就是指保持对象,甚至在多次执行同一程序之间也保持对象.通过本文,您会对 Python对象的各种持久性机制(从关系数据库到 Py ...
- ]Java 5|6 并发包介绍
ava.util.concurrent 包含许多线程安全.测试良好.高性能的并发构建块.不客气地说,创建 java.util.concurrent 的目的就是要实现 Collection 框架对数据结 ...
- Java多线程与并发面试题
1,什么是线程? 线程是操作系统能够进行运算调度的最小单位,它被包含在进程之中,是进程中的实际运作单位.程序员可以通过它进行多处理器编程,你可以使用多线程对运算密集型任务提速.比如,如果一个线程完成一 ...
- Java面试题之基础篇概览
Java面试题之基础篇概览 1.一个“.java”源文件中是否可以包含多个类(不是内部类)?有什么限制? 可以有多个类,但只能有一个public的类,且public的类名必须与文件名相一致. 2.Ja ...
- [ Java面试题 ]并发篇
1.Java内存模型是什么? Java内存模型规定和指引Java程序在不同的内存架构.CPU和操作系统间有确定性地行为.它在多线程的情况下尤其重要.Java内存模型对一个线程所做的变动能被其它线程可见 ...
- Java多线程面试题整理
部分一:多线程部分: 1) 什么是线程? 线程是操作系统能够进行运算调度的最小单位,它被包含在进程之中,是进程中的实际运作单位.程序员可以通过它进行多处理器编程,你可以使用多线程对运算密集型任务提速. ...
随机推荐
- 单选按钮radio与文字如何对齐?
布局如下: <input type="radio" value="立即发送" name="a_1">立即发送 <input ...
- nginx重要配置项简要说明
1.重要配置项 以下是一个完整的nginx配置信息. #================================以下是全局配置项 #指定运行nginx的用户和用户组,默认情况下该选项关闭(关闭 ...
- POJ 3070 + 51Nod 1242 大斐波那契数取余
POJ 3070 #include "iostream" #include "cstdio" using namespace std; class matrix ...
- vivo面试学习3(git和svn的区别)
git和svn有什么区别? svn: 系统特点: 1).集中式版本控制系统(存在一个中央版本库,所有开发人员所使用的代码都是来源于版本库,提交代码也是这个中央版本库) 2).企业内部并行集中开发 3) ...
- mongodb的数据库操作
1.创建数据库 语法 MongoDB 创建数据库的语法格式如下: use DATABASE_NAME 如果数据库不存在,则创建数据库,否则切换到指定数据库. 1.创建数据库 > show dbs ...
- Method and apparatus for providing total and partial store ordering for a memory in multi-processor system
An improved memory model and implementation is disclosed. The memory model includes a Total Store Or ...
- android 调试 native 程序的方法
一.背景 首先说需求,这个需求非常常见,就是android上需要的一个功能,linux已经有开源代码而且非常稳定,希望能直接porting过去使用,这个程序是pure c 的代码,也就是说,跟andr ...
- pool.map的第二个参数想传入多个咋整?
from functools import partial from multiprocessing import Pool as ThreadPool pageurls=[] if maxpage: ...
- maven多模块项目执行 deploy 时 忽略某些model (忽略war包)
maven deploy 时,通常需要忽略生成war的model,简单调整一下配置即可: <plugins> <plugin> <groupId>org.apach ...
- Linux创建swap分区(用文件作为Swap分区)
1.创建要作为swap分区的文件:增加1GB大小的交换分区,则命令写法如下,其中的count等于想要的块的数量(bs*count=文件大小). dd if=/dev/zero of=/root/swa ...