Python的并发并行[2] -> 队列[1] -> 使用队列进行任务控制

使用队列进行任务控制

---

1 FIFO与LIFO队列

FIFO(First In First Out)与LIFO(Last In First Out)分别是两种队列形式，在FIFO中，满足先入先出的队列方式，而LIFO则是后入先出的队列形式，利用这两种方式可以实现不同的队列功能。

 from random import randint
 from time import sleep, ctime
 from queue import Queue, LifoQueue
 from threading import Thread
 
 COUNT = 0
 
 class MyThread(Thread):
     """
     Bulid up a Module to make this subclass more general
     And get return value by add a function named 'getResult()'
     """
     def __init__(self, func, args, name=''):
         Thread.__init__(self)
         self.name = name
         self.func = func
         self.args = args
 
     def getResult(self):
         return self.res
 
     def run(self):
         print('Starting', self.name, 'at:', ctime())
         # Call function here and calculate the running time
         self.res = self.func(*self.args)
         print(self.name, 'finished at:', ctime())
 
 class MyQueue():
     def __init__(self):
         self.funcs = [self.writer, self.reader]
         self.nfuncs = range(len(self.funcs))
 
     def writeQ(self, queue):
         global COUNT
         print('Producing object OBJ_%d for Q...' % COUNT, end=' ')
         queue.put('OBJ_%d' % COUNT, True)
         print('size now:', queue.qsize())
         COUNT += 1
 
     def readQ(self, queue):
         # If queue is empty, block here until queue available
         val = queue.get(True)
         print('Consumed object %s from Q... size now:' % val, queue.qsize())
 
     def writer(self, queue, loops):
         for i in range(loops):
             self.writeQ(queue)
             sleep(randint(1, 3))
 
     def reader(self, queue, loops):
         for i in range(loops):
             self.readQ(queue)
             sleep(randint(2, 5))
 
     def main(self):
         nloops = randint(2, 5)
         fifoQ = Queue(32)
         lifoQ = LifoQueue(32)
 
         # First In First Out mode for Queue
         print('-----Start FIFO Queue-----')
         threads = []
         for i in self.nfuncs:
             threads.append(MyThread(self.funcs[i], (fifoQ, nloops), self.funcs[i].__name__))
         for t in threads:
             t.start()
         for t in threads:
             t.join()
         # Last In First Out mode for LifoQueue
         print('-----Start LIFO Queue-----')
         threads = []
         for i in self.nfuncs:
             threads.append(MyThread(self.funcs[i], (lifoQ, nloops), self.funcs[i].__name__))
         for t in threads:
             t.start()
         for t in threads:
             t.join()
 
         print('All DONE')
 
 if __name__ == '__main__':
      MyQueue().main()

第 1-27 行，首先对需要的模块进行导入，并定义一个全局变量的计数器，派生一个MyThread线程类，用于调用函数及其返回值（本例中MyThread可用于接受writer和reader函数，同时将Queue的实例作为参数传给这两个函数）。

第 30-79 行，定义一个队列类，用于进行队列一系列处理，其中writeQ与readQ会分别对队列执行put和get函数，在writeQ中利用全局变量设置每个加入队列的对象的名字。而writer和reader则会利用循环多次执行writeQ和readQ函数。最后定义一个main函数，用于生成队列，同时调用FIFO以及LIFO两种队列方式。

运行得到结果

-----Start FIFO Queue-----
Starting writer at: Tue Aug  1 21:43:22 2017
Producing object OBJ_0 for Q... size now: 1
Starting reader at: Tue Aug  1 21:43:22 2017
Consumed object OBJ_0 from Q... size now: 0
Producing object OBJ_1 for Q... size now: 1
Producing object OBJ_2 for Q... size now: 2
Producing object OBJ_3 for Q... size now: 3
Consumed object OBJ_1 from Q... size now: 2
writer finished at: Tue Aug  1 21:43:26 2017
Consumed object OBJ_2 from Q... size now: 1
Consumed object OBJ_3 from Q... size now: 0
reader finished at: Tue Aug  1 21:43:34 2017
-----Start LIFO Queue-----
Starting writer at: Tue Aug  1 21:43:34 2017
Producing object OBJ_4 for Q... size now: 1
Starting reader at: Tue Aug  1 21:43:34 2017
Consumed object OBJ_4 from Q... size now: 0
Producing object OBJ_5 for Q... size now: 1
Producing object OBJ_6 for Q... size now: 2
Producing object OBJ_7 for Q... size now: 3
writer finished at: Tue Aug  1 21:43:38 2017
Consumed object OBJ_7 from Q... size now: 2
Consumed object OBJ_6 from Q... size now: 1
Consumed object OBJ_5 from Q... size now: 0
reader finished at: Tue Aug  1 21:43:53 2017
All DONE  

从输出可以看出，FIFO满足先入先出，LIFO满足后入先出的队列形式。

2 join挂起与task_done信号

在queue模块中，Queue类提供了两个用于跟踪监测任务完成的函数，join和task_done，对于join函数来说，当Queue的类实例调用了join函数挂起时，join函数会阻塞等待，一直到join之前进入队列的所有任务全部标记为task_done后才会解除阻塞。

Note: 通过查看Queue的源码可以看出，在调用put函数时，会对类变量unfinished_tasks进行数值加1，而调用get函数时并不会将unfinished_tasks进行减1，只有调用task_done函数才会导致变量减1。而调用join函数时，join函数会对这个unfinished_tasks变量进行获取，也就是说，join函数会获取到在调用之前所有被put进队列里的任务中，还没有调用过task_done函数的任务数量，无论这个任务是否已经被get出列。

下面的例子中，以Queue_FIFO_LIFO.py中的MyQueue为基类，派生出一个新类，用于测试join函数与task_done函数。

 from Queue_FIFO_LIFO import *
 
 class NewQueue(MyQueue):
     def __init__(self):
         MyQueue.__init__(self)
 
     def writer(self, queue, loops):
         for i in range(loops):
             self.writeQ(queue)
             sleep(randint(1, 3))
         print('Producing join here, waiting consumer')
         queue.join()
 
     def reader(self, queue, loops):
         for i in range(loops):
             self.readQ(queue)
             sleep(randint(2, 5))
             print('OBJ_%d task done' % i)
             queue.task_done()
 
     def main(self):
         nloops = randint(2, 5)
         fifoQ = Queue(32)
 
         print('-----Start FIFO Queue-----')
         threads = []
         for i in self.nfuncs:
             threads.append(MyThread(self.funcs[i], (fifoQ, nloops), self.funcs[i].__name__))
         for t in threads:
             t.start()
         for t in threads:
             t.join()
 
         print('All DONE')
 
 if __name__ == '__main__':
     NewQueue().main()

上面的代码，在导入模块后，调用MyQueue的初始化函数进行初始化设置。在新类NewQueue中，对原基类的writer和reader以及main方法进行了重载，加入了join函数和task_done函数，并在main函数中只采用FIFO队列进行试验。

运行得到结果

-----Start FIFO Queue-----
Starting writer at: Wed Aug  2 09:06:40 2017
Producing object OBJ_0 for Q... size now: 1
Starting reader at: Wed Aug  2 09:06:40 2017
Consumed object OBJ_0 from Q... size now: 0
Producing object OBJ_1 for Q... size now: 1
Producing object OBJ_2 for Q... size now: 2
OBJ_0 task done
Consumed object OBJ_1 from Q... size now: 1
Producing object OBJ_3 for Q... size now: 2
Producing object OBJ_4 for Q... size now: 3
Producing join here, waiting consumer
OBJ_1 task done
Consumed object OBJ_2 from Q... size now: 2
OBJ_2 task done
Consumed object OBJ_3 from Q... size now: 1
OBJ_3 task done
Consumed object OBJ_4 from Q... size now: 0
OBJ_4 task done
reader finished at: Wed Aug  2 09:07:02 2017
writer finished at: Wed Aug  2 09:07:02 2017
All DONE  

通过得到的结果可以看出，当新类里的writer完成了自己的Producing任务后，会由join挂起，一直等待直到reader的Consuming全部完成且标记task_done之后，才会解除挂起，此时writer和reader将会一起结束退出。

相关阅读

---

1. 多线程的建立

2. queue 模块

参考链接

---

《Python 核心编程 第3版》