twisted task.cpperator

twisted task.cpperator

1.      twisted task.cpperator

1.1.    简介-cooperator

官方文档：

https://twistedmatrix.com/documents/current/api/twisted.internet.task.Cooperator.html#coiterate

Cooperative task scheduler.

A cooperative task is an iterator where each iteration represents an atomic unit of work. When the iterator yields, it allows the Cooperator to decide which of its tasks to execute next. If the iterator yields a defer.Deferred then work will pause until the defer.Deferred fires and completes its callback chain.

一个协作任务是迭代器，每个迭代元素是一个原子性的操作。当迭代器产出/抛出时，它允许Cooperator决定是否进行下一步。如果迭代器抛出一个Deferred，它会暂停直到Deferred对象fires(触发？)并且跑完了它的回调链。

When a Cooperator has more than one task, it distributes work between all tasks.

如果cooperator对象有多个任务，它会分配执行权到所有任务（简单说就是并发）。

There are two ways to add tasks to a Cooperator, cooperate and coiterate. cooperate is the more useful of the two, as it returns a CooperativeTask, which can be paused, resumed and waited on. coiterate has the same effect, but returns only a defer.Deferred that fires when the task is done.

有两种方式添加任务，cooperate and coiterate，前者功能较多，支持暂停，恢复，等待；后者效用相同，但不能控制执行过程。

Cooperator can be used for many things, including but not limited to:

running one or more computationally intensive tasks without blocking

limiting parallelism by running a subset of the total tasks simultaneously

doing one thing, waiting for a Deferred to fire, doing the next thing, repeat (i.e. serializing a sequence of asynchronous tasks)

Multiple Cooperators do not cooperate with each other, so for most cases you should use the global cooperator.

1.2.    代码解读

前置技能

_EPSILON = 0.00000001
def _defaultScheduler(x):
    from twisted.internet
import reactor
    return reactor.callLater(_EPSILON,
x)

下文提到的scheduler都是指上述方法，作用是向reactor注册任务

class _Timer(object):
    MAX_SLICE = 0.01
    def __init__(self):
        self.end =
time.time() + self.MAX_SLICE

def __call__(self):
        return time.time()
>= self.end

功能：判断延时

主体函数：

class Cooperator(object):
    def __init__(self,
                 terminationPredicateFactory=_Timer,
                 scheduler=_defaultScheduler,
                 started=True):
        self._tasks = []
        self._metarator = iter(())
        self._terminationPredicateFactory = terminationPredicateFactory
        self._scheduler = scheduler
        self._delayedCall = None
        self._stopped = False
        self._started = started

    def coiterate(self, iterator, doneDeferred=None):
        if doneDeferred is None:
            doneDeferred = defer.Deferred()
        CooperativeTask(iterator, self).whenDone().chainDeferred(doneDeferred)
        return doneDeferred


    def cooperate(self, iterator):
        return CooperativeTask(iterator, self)

最重要的两个方法。

1.2.1.   cooperate

先看一下def cooperate(self, iterator)：

class CooperativeTask(object):
    def __init__(self,
iterator, cooperator):
        """
        A private constructor: to create
a new L{CooperativeTask}, see
        L{Cooperator.cooperate}.
        """
        self._iterator
= iterator
        self._cooperator
= cooperator
        self._deferreds
= []
        self._pauseCount
= 0
        self._completionState
= None
        self._completionResult
= None
        cooperator._addTask(self)

回到cooperator._addTask()

def _addTask(self, task):
    """
    Add a L{CooperativeTask} object to
this L{Cooperator}.
    """
    if self._stopped:
        self._tasks.append(task)  # XXX silly, I know, but _completeWith
        # does the inverse
        task._completeWith(SchedulerStopped(),
Failure(SchedulerStopped()))
    else:
        self._tasks.append(task)
        self._reschedule()

_mustScheduleOnStart = False
def _reschedule(self):
    if not self._started:
        self._mustScheduleOnStart = True
        return
    if self._delayedCall is None
and self._tasks:
        self._delayedCall =
self._scheduler(self._tick)

通过self._scheduler(self._tick)注册了一个调用,实例化到这里就结束了。

这也是twisted的一个特点，经常是向循环注册一个方法，后续执行从循环开始。

待reactor开始之后，会调用_tick

def _tasksWhileNotStopped(self):
    """
    Yield all L{CooperativeTask} objects
in a loop as long as this
    L{Cooperator}'s termination condition
has not been met.
    """
    terminator =
self._terminationPredicateFactory()
    while self._tasks:
        for t in self._metarator:
            yield t
            if terminator():
                return
        self._metarator = iter(self._tasks)

def _tick(self):
    """
    Run one scheduler tick.
    """
    self._delayedCall = None
    for taskObj in self._tasksWhileNotStopped():
        taskObj._oneWorkUnit()
    self._reschedule()

terminator()是用于执行时间片长度的判断，用于保证在多个task之间按时间片切换，定制时间片长可以重写_timer并在创建cooperator时指定。

返回的t实际是一个class CooperativeTask(object)实例

def _oneWorkUnit(self):
    """
    Perform one unit of work for this
task, retrieving one item from its
    iterator, stopping if there are no
further items in the iterator, and
    pausing if the result was a
L{defer.Deferred}.
    """
    try:
        result = next(self._iterator)
    except StopIteration:
        self._completeWith(TaskDone(),
self._iterator)
    except:
        self._completeWith(TaskFailed(),
Failure())
    else:
        if isinstance(result,
defer.Deferred):
            self.pause()

def failLater(f):
                self._completeWith(TaskFailed(),
f)
            result.addCallbacks(lambda result:
self.resume(),
                               
failLater)

result = next(self._iterator)

这里就是传入的iter对象了

if isinstance(result, defer.Deferred):

注意，如果result不是延迟调用，则直接执行，然后循环；

如果它是deferred，执行self.pause()，在其中会把self从_task队列中删除。

同时为result添加callback，在回调中执行self.resume()

def resume(self):
    """
    Resume processing of a paused
L{CooperativeTask}.
    @raise NotPaused: if this
L{CooperativeTask} is not paused.
    """
    if self._pauseCount
== 0:
        raise NotPaused()
    self._pauseCount -= 1
    if self._pauseCount
== 0 and self._completionState is None:
        self._cooperator._addTask(self)

在resume中重新执行_addTask添加任务，注册tick调用。

通过上述过程决定了一个deferred只能在执行-暂停-重新注册任务-向reactor注册任务-调用-执行的过程中循环，也就是执行流程只能一一执行。

1.2.2.  
coiterate

coiterate解读

    def coiterate(self,
iterator, doneDeferred=None):
        if doneDeferred is None:
            doneDeferred =
defer.Deferred()
        CooperativeTask(iterator,
self).whenDone().chainDeferred(doneDeferred)
        return doneDeferred

与cooperate()不同的是，它返回一个deferred对象。

def whenDone(self):
    d = defer.Deferred()
    if self._completionState
is None:
        self._deferreds.append(d)
    else:
       
d.callback(self._completionResult)
    return d

整个过程实质就是在CooperativeTask的_deferreds列表中保存deferred对象，并为这个对象添加一个回调指向doneDeferred的回调，然后返回doneDeferred

结果就是当iterator结束或异常时都会调用CooperativeTask的_completeWith

def _completeWith(self, completionState,
deferredResult):
    self._completionState
= completionState
    self._completionResult =
deferredResult
    if not self._pauseCount:
       
self._cooperator._removeTask(self)

    for d in self._deferreds:
        d.callback(deferredResult)

调用_completeWith就标志着迭代结束或异常，做了两个操作，清除task，调用self._deferreds中每个对象的回调。

这一操作会触发doneDeferred的回调；至此所有处理流程结束，cooperator执行操作，返回延迟调用对象，触发回调。