twisted task.cpperator
twisted task.cpperator
1. twisted task.cpperator
1.1. 简介-cooperator
官方文档:
https://twistedmatrix.com/documents/current/api/twisted.internet.task.Cooperator.html#coiterate
Cooperative task scheduler.
A cooperative task is an iterator where each iteration represents an atomic unit of work. When the iterator yields, it allows the Cooperator to decide which of its tasks to execute next. If the iterator yields a defer.Deferred then work will pause until the defer.Deferred fires and completes its callback chain.
一个协作任务是迭代器,每个迭代元素是一个原子性的操作。当迭代器产出/抛出时,它允许Cooperator决定是否进行下一步。如果迭代器抛出一个Deferred,它会暂停直到Deferred对象fires(触发?)并且跑完了它的回调链。
When a Cooperator has more than one task, it distributes work between all tasks.
如果cooperator对象有多个任务,它会分配执行权到所有任务(简单说就是并发)。
There are two ways to add tasks to a Cooperator, cooperate and coiterate. cooperate is the more useful of the two, as it returns a CooperativeTask, which can be paused, resumed and waited on. coiterate has the same effect, but returns only a defer.Deferred that fires when the task is done.
有两种方式添加任务,cooperate and coiterate,前者功能较多,支持暂停,恢复,等待;后者效用相同,但不能控制执行过程。
Cooperator can be used for many things, including but not limited to:
running one or more computationally intensive tasks without blocking
limiting parallelism by running a subset of the total tasks simultaneously
doing one thing, waiting for a Deferred to fire, doing the next thing, repeat (i.e. serializing a sequence of asynchronous tasks)
Multiple Cooperators do not cooperate with each other, so for most cases you should use the global cooperator.
1.2. 代码解读
前置技能
_EPSILON = 0.00000001
def _defaultScheduler(x):
from twisted.internet
import reactor
return reactor.callLater(_EPSILON,
x)
下文提到的scheduler都是指上述方法,作用是向reactor注册任务
class _Timer(object):
MAX_SLICE = 0.01
def __init__(self):
self.end =
time.time() + self.MAX_SLICE
def __call__(self):
return time.time()
>= self.end
功能:判断延时
主体函数:
class Cooperator(object):
def __init__(self,
terminationPredicateFactory=_Timer,
scheduler=_defaultScheduler,
started=True):
self._tasks = []
self._metarator = iter(())
self._terminationPredicateFactory = terminationPredicateFactory
self._scheduler = scheduler
self._delayedCall = None
self._stopped = False
self._started = started def coiterate(self, iterator, doneDeferred=None):
if doneDeferred is None:
doneDeferred = defer.Deferred()
CooperativeTask(iterator, self).whenDone().chainDeferred(doneDeferred)
return doneDeferred def cooperate(self, iterator):
return CooperativeTask(iterator, self)
最重要的两个方法。
1.2.1. cooperate
先看一下def cooperate(self, iterator):
class CooperativeTask(object):
def __init__(self,
iterator, cooperator):
"""
A private constructor: to create
a new L{CooperativeTask}, see
L{Cooperator.cooperate}.
"""
self._iterator
= iterator
self._cooperator
= cooperator
self._deferreds
= []
self._pauseCount
= 0
self._completionState
= None
self._completionResult
= None
cooperator._addTask(self)
回到cooperator._addTask()
def _addTask(self, task):
"""
Add a L{CooperativeTask} object to
this L{Cooperator}.
"""
if self._stopped:
self._tasks.append(task) # XXX silly, I know, but _completeWith
# does the inverse
task._completeWith(SchedulerStopped(),
Failure(SchedulerStopped()))
else:
self._tasks.append(task)
self._reschedule()
_mustScheduleOnStart = False
def _reschedule(self):
if not self._started:
self._mustScheduleOnStart = True
return
if self._delayedCall is None
and self._tasks:
self._delayedCall =
self._scheduler(self._tick)
通过self._scheduler(self._tick)注册了一个调用,实例化到这里就结束了。
这也是twisted的一个特点,经常是向循环注册一个方法,后续执行从循环开始。
待reactor开始之后,会调用_tick
def _tasksWhileNotStopped(self):
"""
Yield all L{CooperativeTask} objects
in a loop as long as this
L{Cooperator}'s termination condition
has not been met.
"""
terminator =
self._terminationPredicateFactory()
while self._tasks:
for t in self._metarator:
yield t
if terminator():
return
self._metarator = iter(self._tasks)
def _tick(self):
"""
Run one scheduler tick.
"""
self._delayedCall = None
for taskObj in self._tasksWhileNotStopped():
taskObj._oneWorkUnit()
self._reschedule()
terminator()是用于执行时间片长度的判断,用于保证在多个task之间按时间片切换,定制时间片长可以重写_timer并在创建cooperator时指定。
返回的t实际是一个class CooperativeTask(object)实例
def _oneWorkUnit(self):
"""
Perform one unit of work for this
task, retrieving one item from its
iterator, stopping if there are no
further items in the iterator, and
pausing if the result was a
L{defer.Deferred}.
"""
try:
result = next(self._iterator)
except StopIteration:
self._completeWith(TaskDone(),
self._iterator)
except:
self._completeWith(TaskFailed(),
Failure())
else:
if isinstance(result,
defer.Deferred):
self.pause()
def failLater(f):
self._completeWith(TaskFailed(),
f)
result.addCallbacks(lambda result:
self.resume(),
failLater)
result = next(self._iterator)
这里就是传入的iter对象了
if isinstance(result, defer.Deferred):
注意,如果result不是延迟调用,则直接执行,然后循环;
如果它是deferred,执行self.pause(),在其中会把self从_task队列中删除。
同时为result添加callback,在回调中执行self.resume()
def resume(self):
"""
Resume processing of a paused
L{CooperativeTask}.
@raise NotPaused: if this
L{CooperativeTask} is not paused.
"""
if self._pauseCount
== 0:
raise NotPaused()
self._pauseCount -= 1
if self._pauseCount
== 0 and self._completionState is None:
self._cooperator._addTask(self)
在resume中重新执行_addTask添加任务,注册tick调用。
通过上述过程决定了一个deferred只能在执行-暂停-重新注册任务-向reactor注册任务-调用-执行的过程中循环,也就是执行流程只能一一执行。
1.2.2.
coiterate
coiterate解读
def coiterate(self,
iterator, doneDeferred=None):
if doneDeferred is None:
doneDeferred =
defer.Deferred()
CooperativeTask(iterator,
self).whenDone().chainDeferred(doneDeferred)
return doneDeferred
与cooperate()不同的是,它返回一个deferred对象。
def whenDone(self):
d = defer.Deferred()
if self._completionState
is None:
self._deferreds.append(d)
else:
d.callback(self._completionResult)
return d
整个过程实质就是在CooperativeTask的_deferreds列表中保存deferred对象,并为这个对象添加一个回调指向doneDeferred的回调,然后返回doneDeferred
结果就是当iterator结束或异常时都会调用CooperativeTask的_completeWith
def _completeWith(self, completionState,
deferredResult):
self._completionState
= completionState
self._completionResult =
deferredResult
if not self._pauseCount:
self._cooperator._removeTask(self)
for d in self._deferreds:
d.callback(deferredResult)
调用_completeWith就标志着迭代结束或异常,做了两个操作,清除task,调用self._deferreds中每个对象的回调。
这一操作会触发doneDeferred的回调;至此所有处理流程结束,cooperator执行操作,返回延迟调用对象,触发回调。
twisted task.cpperator的更多相关文章
- twisted的task之cooperator和scrapy的parallel()函数
def handle_spider_output(self, result, request, response, spider): if not result: return defer_succe ...
- Python自动化运维之15、网络编程之socket、socketserver、select、twisted
一.TCP/IP相关知识 TCP/UDP提供进程地址,两个协议互不干扰的独自的协议 TCP :Transmission Control Protocol 传输控制协议,面向连接的协议,通信 ...
- twisted(1)--何为异步
早就想写一篇文章,整体介绍python的2个异步库,twisted和tornado.我们在开发python的tcpserver时候,通常只会用3个库,twisted.tornado和gevent,其中 ...
- twisted(3)--再谈twisted
上一章,我们直接写了一个小例子来从整体讲述twisted运行的大致过程,今天我们首先深入一些概念,在逐渐明白这些概念以后,我们会修改昨天写的例子. 先看下面一张图: 这个系列的第一篇文章,我们已经为大 ...
- twisted(2)--聊天系统
我们今天要做一个聊天系统,这样可以和我们之前flask api那系列文章结合起来:其次,聊天系统最能代表tcpserver,以后可以套用各种模型,比如我们公司做的物联网,其实就是把聊天系统简化一下. ...
- twisted的一些代码
之前用swoole(1.7.19)写的一段程序在数据量大的时候存在内存泄漏,改为twisted(15.4)实现,自测无误,记录如下(两者cpu占用率90%时吞吐rps能从120提升到1000). #! ...
- twisted学习笔记 No.2 WebServer
原创博文,转载请注明出处. 当服务器接收到一个客户端请求后,会创建一个请求对象并传递到资源系统,资源系统会根据请求路径分发到相应的资源对象,资源被要求渲染自身并返回结果到客户端. 解析HTTP Req ...
- 理解Twisted与非阻塞编程
先来看一段代码: # ~*~ Twisted - A Python tale ~*~ from time import sleep # Hello, I'm a developer and I mai ...
- 转载 twisted(1)--何为异步
Reference: http://www.cnblogs.com/yueerwanwan0204/p/5589860.html 早就想写一篇文章,整体介绍python的2个异步库,twisted和t ...
随机推荐
- Day4 注解 泛型
注解是什么 注释是绑定到程序源代码元素的元数据,对它们运行的代码的操作没有影响. https://blog.csdn.net/SDDDLLL/article/details/93509699 他们 ...
- winform学习(9)无边框拖动窗体
去除边框 选中窗体,在属性中找到FormBorderStyle,设置为None 实现无边框拖动 [DllImport("user32.dll")] public st ...
- Nearest Interesting Number
Polycarp knows that if the sum of the digits of a number is divisible by 33, then the number itself ...
- SparkSQL个人记录
SparkSQL将RDD封装成一个DataFrame对象,这个对象类似于关系型数据库中的表. 一.SparkSQL入门 1.创建DataFrame 相当于数据库中的一张表,它是一个只读的表,不能在运算 ...
- 剑指offer 面试题35.复杂链表的复制
时间O(N),空间O(N) /* struct RandomListNode { int label; struct RandomListNode *next, *random; RandomList ...
- thinkphp新的心得
1,代码的高度复用 1,js的删除,弹出框都可以用嵌入的js,不用每个页面都写 2,控制器的初始化函数可以实现配置某些变量,比如admin模块的topic控制器的初始化函数可以 $this->o ...
- sublime 神一样的插件
专属配置 // 主题 "theme": "Boxy Tomorrow.sublime-theme", "theme_grid_border_size_ ...
- MongoDB-2 安装与配置
安装建议 MongoDB 可以在mac/win/linux上安装,我个人建议在linux上安装会更好,这样测试起来更方便.一般来说本地只装mysql或者MariaDB,其余的中间件一律放linux,让 ...
- rancher 方式创建nfs-client 存储类流程
rancher 方式创建nfs-client 存储类流程 待办 https://www.iamle.com/archives/2514.html
- Jmeter注册100个账户的三个方法
Jmeter注册账户比如注册成千上万个账户,如何快速实现呢? 三种方法分别举例注册5个账户 1)添加CSV data config_txt 2)添加CSV data config_csv 3)函数助手 ...