import threading
import random,time
import queue
q_init = queue.Queue(maxsize=5)
import logging
logging.basicConfig(level = logging.INFO,format = '%(asctime)s - %(levelname)s -->%(funcName)s at line %(lineno)d: \n %(message)s')
log= logging.getLogger() def jobs(item):
t=random.randint(1,15)
time.sleep(t)
status = random.randint(0, 1)
if status == 0:
return ("success",item)
else:
return ("failed",item) def do_work(item):
logging.info("do something %s,time start %s" % (item, time.asctime()))
a = jobs(item)
return a def worker(q):
while True:
# if queue get None by thread and stop that thread called by break
item = q.get()
if item is None:
break
st = do_work(item)
if st[0] in ["success", "failed"]:
logging.info("%s task finished status is %s" % (st[1],st[0]))
q_init.task_done() if __name__ == '__main__':
threads = []
thread_num=5
for i in range(thread_num):
t = threading.Thread(target=worker,args=(q_init,))
t.start()
threads.append(t)
for item in range(1,50):
q_init.put(item)
# block until all tasks are done
q_init.join()
# stop workers
for i in range(thread_num):
q_init.put(None)
for t in threads:
t.join()

  

结果:

2019-12-17 12:49:45,417  - INFO -->do_work  at line 21:
do something 1,time start Tue Dec 17 12:49:45 2019
2019-12-17 12:49:45,417 - INFO -->do_work at line 21:
do something 2,time start Tue Dec 17 12:49:45 2019
2019-12-17 12:49:45,417 - INFO -->do_work at line 21:
do something 3,time start Tue Dec 17 12:49:45 2019
2019-12-17 12:49:45,417 - INFO -->do_work at line 21:
do something 4,time start Tue Dec 17 12:49:45 2019
2019-12-17 12:49:45,417 - INFO -->do_work at line 21:
do something 5,time start Tue Dec 17 12:49:45 2019
2019-12-17 12:49:48,417 - INFO -->worker at line 33:
1 task finished status is success
2019-12-17 12:49:48,417 - INFO -->do_work at line 21:
do something 6,time start Tue Dec 17 12:49:48 2019
2019-12-17 12:49:49,420 - INFO -->worker at line 33:
4 task finished status is success
2019-12-17 12:49:49,420 - INFO -->do_work at line 21:
do something 7,time start Tue Dec 17 12:49:49 2019
2019-12-17 12:49:53,424 - INFO -->worker at line 33:
5 task finished status is failed
2019-12-17 12:49:53,424 - INFO -->do_work at line 21:
do something 8,time start Tue Dec 17 12:49:53 2019
2019-12-17 12:49:55,436 - INFO -->worker at line 33:
6 task finished status is failed
2019-12-17 12:49:55,436 - INFO -->do_work at line 21:
do something 9,time start Tue Dec 17 12:49:55 2019
2019-12-17 12:49:55,436 - INFO -->worker at line 33:
7 task finished status is success
2019-12-17 12:49:55,437 - INFO -->do_work at line 21:
do something 10,time start Tue Dec 17 12:49:55 2019
2019-12-17 12:49:55,437 - INFO -->worker at line 33:
2 task finished status is success
2019-12-17 12:49:55,437 - INFO -->do_work at line 21:
do something 11,time start Tue Dec 17 12:49:55 2019
2019-12-17 12:49:56,430 - INFO -->worker at line 33:
8 task finished status is failed
2019-12-17 12:49:56,430 - INFO -->do_work at line 21:
do something 12,time start Tue Dec 17 12:49:56 2019
2019-12-17 12:49:57,443 - INFO -->worker at line 33:
10 task finished status is failed
2019-12-17 12:49:57,443 - INFO -->do_work at line 21:
do something 13,time start Tue Dec 17 12:49:57 2019
2019-12-17 12:49:59,419 - INFO -->worker at line 33:
3 task finished status is success
2019-12-17 12:49:59,419 - INFO -->do_work at line 21:
do something 14,time start Tue Dec 17 12:49:59 2019
2019-12-17 12:50:02,444 - INFO -->worker at line 33:
13 task finished status is success
2019-12-17 12:50:02,444 - INFO -->do_work at line 21:
do something 15,time start Tue Dec 17 12:50:02 2019
2019-12-17 12:50:06,444 - INFO -->worker at line 33:
9 task finished status is success
2019-12-17 12:50:06,444 - INFO -->do_work at line 21:
do something 16,time start Tue Dec 17 12:50:06 2019
2019-12-17 12:50:07,446 - INFO -->worker at line 33:
11 task finished status is success
2019-12-17 12:50:07,446 - INFO -->do_work at line 21:
do something 17,time start Tue Dec 17 12:50:07 2019
2019-12-17 12:50:07,447 - INFO -->worker at line 33:
15 task finished status is success
2019-12-17 12:50:07,447 - INFO -->do_work at line 21:
do something 18,time start Tue Dec 17 12:50:07 2019
2019-12-17 12:50:11,434 - INFO -->worker at line 33:
12 task finished status is failed
2019-12-17 12:50:11,434 - INFO -->do_work at line 21:
do something 19,time start Tue Dec 17 12:50:11 2019
2019-12-17 12:50:11,451 - INFO -->worker at line 33:
17 task finished status is failed
2019-12-17 12:50:11,451 - INFO -->do_work at line 21:
do something 20,time start Tue Dec 17 12:50:11 2019
2019-12-17 12:50:13,445 - INFO -->worker at line 33:
16 task finished status is success
2019-12-17 12:50:13,445 - INFO -->do_work at line 21:
do something 21,time start Tue Dec 17 12:50:13 2019
2019-12-17 12:50:14,423 - INFO -->worker at line 33:
14 task finished status is success
2019-12-17 12:50:14,423 - INFO -->do_work at line 21:
do something 22,time start Tue Dec 17 12:50:14 2019
2019-12-17 12:50:14,438 - INFO -->worker at line 33:
19 task finished status is failed
2019-12-17 12:50:14,438 - INFO -->do_work at line 21:
do something 23,time start Tue Dec 17 12:50:14 2019
2019-12-17 12:50:15,453 - INFO -->worker at line 33:
20 task finished status is failed
2019-12-17 12:50:15,453 - INFO -->do_work at line 21:
do something 24,time start Tue Dec 17 12:50:15 2019
2019-12-17 12:50:18,450 - INFO -->worker at line 33:
21 task finished status is failed
2019-12-17 12:50:18,450 - INFO -->do_work at line 21:
do something 25,time start Tue Dec 17 12:50:18 2019
2019-12-17 12:50:20,450 - INFO -->worker at line 33:
18 task finished status is failed
2019-12-17 12:50:20,450 - INFO -->do_work at line 21:
do something 26,time start Tue Dec 17 12:50:20 2019
2019-12-17 12:50:24,456 - INFO -->worker at line 33:
26 task finished status is failed
2019-12-17 12:50:24,456 - INFO -->do_work at line 21:
do something 27,time start Tue Dec 17 12:50:24 2019
2019-12-17 12:50:25,433 - INFO -->worker at line 33:
22 task finished status is success
2019-12-17 12:50:25,433 - INFO -->do_work at line 21:
do something 28,time start Tue Dec 17 12:50:25 2019
2019-12-17 12:50:27,443 - INFO -->worker at line 33:
23 task finished status is failed
2019-12-17 12:50:27,443 - INFO -->do_work at line 21:
do something 29,time start Tue Dec 17 12:50:27 2019
2019-12-17 12:50:27,459 - INFO -->worker at line 33:
27 task finished status is success
2019-12-17 12:50:27,459 - INFO -->do_work at line 21:
do something 30,time start Tue Dec 17 12:50:27 2019
2019-12-17 12:50:28,468 - INFO -->worker at line 33:
24 task finished status is failed
2019-12-17 12:50:28,468 - INFO -->do_work at line 21:
do something 31,time start Tue Dec 17 12:50:28 2019
2019-12-17 12:50:30,445 - INFO -->worker at line 33:
29 task finished status is failed
2019-12-17 12:50:30,445 - INFO -->do_work at line 21:
do something 32,time start Tue Dec 17 12:50:30 2019
2019-12-17 12:50:32,470 - INFO -->worker at line 33:
31 task finished status is failed
2019-12-17 12:50:32,470 - INFO -->do_work at line 21:
do something 33,time start Tue Dec 17 12:50:32 2019
2019-12-17 12:50:33,448 - INFO -->worker at line 33:
32 task finished status is success
2019-12-17 12:50:33,448 - INFO -->do_work at line 21:
do something 34,time start Tue Dec 17 12:50:33 2019
2019-12-17 12:50:33,455 - INFO -->worker at line 33:
25 task finished status is failed
2019-12-17 12:50:33,455 - INFO -->do_work at line 21:
do something 35,time start Tue Dec 17 12:50:33 2019
2019-12-17 12:50:38,440 - INFO -->worker at line 33:
28 task finished status is failed
2019-12-17 12:50:38,440 - INFO -->do_work at line 21:
do something 36,time start Tue Dec 17 12:50:38 2019
2019-12-17 12:50:39,463 - INFO -->worker at line 33:
30 task finished status is failed
2019-12-17 12:50:39,463 - INFO -->do_work at line 21:
do something 37,time start Tue Dec 17 12:50:39 2019
2019-12-17 12:50:44,450 - INFO -->worker at line 33:
34 task finished status is failed
2019-12-17 12:50:44,450 - INFO -->do_work at line 21:
do something 38,time start Tue Dec 17 12:50:44 2019
2019-12-17 12:50:44,468 - INFO -->worker at line 33:
37 task finished status is success
2019-12-17 12:50:44,468 - INFO -->do_work at line 21:
do something 39,time start Tue Dec 17 12:50:44 2019
2019-12-17 12:50:46,459 - INFO -->worker at line 33:
35 task finished status is failed
2019-12-17 12:50:46,459 - INFO -->do_work at line 21:
do something 40,time start Tue Dec 17 12:50:46 2019
2019-12-17 12:50:46,477 - INFO -->worker at line 33:
33 task finished status is success
2019-12-17 12:50:46,478 - INFO -->do_work at line 21:
do something 41,time start Tue Dec 17 12:50:46 2019
2019-12-17 12:50:50,443 - INFO -->worker at line 33:
36 task finished status is failed
2019-12-17 12:50:50,443 - INFO -->do_work at line 21:
do something 42,time start Tue Dec 17 12:50:50 2019
2019-12-17 12:50:55,474 - INFO -->worker at line 33:
39 task finished status is success
2019-12-17 12:50:55,474 - INFO -->do_work at line 21:
do something 43,time start Tue Dec 17 12:50:55 2019
2019-12-17 12:50:58,457 - INFO -->worker at line 33:
38 task finished status is success
2019-12-17 12:50:58,457 - INFO -->do_work at line 21:
do something 44,time start Tue Dec 17 12:50:58 2019
2019-12-17 12:50:58,477 - INFO -->worker at line 33:
43 task finished status is success
2019-12-17 12:50:58,478 - INFO -->do_work at line 21:
do something 45,time start Tue Dec 17 12:50:58 2019
2019-12-17 12:51:00,460 - INFO -->worker at line 33:
40 task finished status is failed
2019-12-17 12:51:00,460 - INFO -->do_work at line 21:
do something 46,time start Tue Dec 17 12:51:00 2019
2019-12-17 12:51:00,480 - INFO -->worker at line 33:
41 task finished status is failed
2019-12-17 12:51:00,480 - INFO -->do_work at line 21:
do something 47,time start Tue Dec 17 12:51:00 2019
2019-12-17 12:51:02,460 - INFO -->worker at line 33:
44 task finished status is failed
2019-12-17 12:51:02,460 - INFO -->do_work at line 21:
do something 48,time start Tue Dec 17 12:51:02 2019
2019-12-17 12:51:03,444 - INFO -->worker at line 33:
42 task finished status is success
2019-12-17 12:51:03,444 - INFO -->do_work at line 21:
do something 49,time start Tue Dec 17 12:51:03 2019
2019-12-17 12:51:04,453 - INFO -->worker at line 33:
49 task finished status is failed
2019-12-17 12:51:06,482 - INFO -->worker at line 33:
45 task finished status is failed
2019-12-17 12:51:07,485 - INFO -->worker at line 33:
47 task finished status is failed
2019-12-17 12:51:12,463 - INFO -->worker at line 33:
48 task finished status is success
2019-12-17 12:51:13,477 - INFO -->worker at line 33:
46 task finished status is success Process finished with exit code 0

  

help of org:

17.7. queue — 一个同步的队列类

源代码: Lib/queue.py


queue 模块实现多生产者,多消费者队列。当信息必须安全的在多线程之间交换时,它在线程编程中是特别有用的。此模块中的 Queue 类实现了所有锁定需求的语义。它依赖于Python支持的线程可用性;请参阅 threading 模块。

模块实现了三种类型的队列,它们的区别仅仅是条目取回的顺序。在 FIFO 队列中,先添加的任务先取回。在 LIFO 队列中,最近被添加的条目先取回(操作类似一个堆栈)。优先级队列中,条目将保持排序( 使用 heapq 模块 ) 并且最小值的条目第一个返回。

Internally, the module uses locks to temporarily block competing threads; however, it is not designed to handle reentrancy within a thread.

queue 模块定义了下列类和异常:

class queue.Queue(maxsize=0)

Constructor for a FIFO queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

class queue.LifoQueue(maxsize=0)

LIFO 队列构造函数。 maxsize 是个整数,用于设置可以放入队列中的项目数的上限。当达到这个大小的时候,插入操作将阻塞至队列中的项目被消费掉。如果 maxsize 小于等于零,队列尺寸为无限大。

class queue.PriorityQueue(maxsize=0)

优先级队列构造函数。 maxsize 是个整数,用于设置可以放入队列中的项目数的上限。当达到这个大小的时候,插入操作将阻塞至队列中的项目被消费掉。如果 maxsize 小于等于零,队列尺寸为无限大。

最小值先被取出( 最小值条目是由 sorted(list(entries))[0] 返回的条目)。条目的典型模式是一个以下形式的元组: (priority_number, data) 。

exception queue.Empty

对空的 Queue 对象,调用非阻塞的 get() (or get_nowait()) 时,引发的异常。

exception queue.Full

对满的 Queue 对象,调用非阻塞的 put() (or put_nowait()) 时,引发的异常。

17.7.1. Queue对象

队列对象 (QueueLifoQueue, 或者 PriorityQueue) 提供下列描述的公共方法。

Queue.qsize()

返回队列的大致大小。注意,qsize() > 0 不保证后续的 get() 不被阻塞,qsize() < maxsize 也不保证 put() 不被阻塞。

Queue.empty()

如果队列为空,返回 True ,否则返回 False 。如果 empty() 返回 True ,不保证后续调用的 put() 不被阻塞。类似的,如果 empty() 返回 False ,也不保证后续调用的 get() 不被阻塞。

Queue.full()

如果队列是满的返回 True ,否则返回 False 。如果 full() 返回 True 不保证后续调用的 get() 不被阻塞。类似的,如果 full() 返回 False 也不保证后续调用的 put() 不被阻塞。

Queue.put(itemblock=Truetimeout=None)

将 item 放入队列。如果可选参数 block 是 true 并且 timeout 是 None (默认),则在必要时阻塞至有空闲插槽可用。如果 timeout 是个正数,将最多阻塞 timeout 秒,如果在这段时间没有可用的空闲插槽,将引发 Full 异常。反之 (block 是 false),如果空闲插槽立即可用,则把 item 放入队列,否则引发 Full 异常 ( 在这种情况下,timeout 将被忽略)。

Queue.put_nowait(item)

相当于 put(item, False) 。

Queue.get(block=Truetimeout=None)

从队列中移除并返回一个项目。如果可选参数 block 是 true 并且 timeout 是 None (默认值),则在必要时阻塞至项目可得到。如果 timeout 是个正数,将最多阻塞 timeout 秒,如果在这段时间内项目不能得到,将引发 Empty 异常。反之 (block 是 false) , 如果一个项目立即可得到,则返回一个项目,否则引发 Empty 异常 (这种情况下,timeout 将被忽略)。

Queue.get_nowait()

相当于 get(False) 。

提供了两个方法,用于支持跟踪 排队的任务 是否 被守护的消费者线程 完整的处理。

Queue.task_done()

表示前面排队的任务已经被完成。被队列的消费者线程使用。每个 get() 被用于获取一个任务, 后续调用 task_done() 告诉队列,该任务的处理已经完成。

如果 join() 当前正在阻塞,在所有条目都被处理后,将解除阻塞(意味着每个 put() 进队列的条目的 task_done() 都被收到)。

如果被调用的次数多于放入队列中的项目数量,将引发 ValueError 异常 。

Queue.join()

阻塞至队列中所有的元素都被接收和处理完毕。

当条目添加到队列的时候,未完成任务的计数就会增加。每当消费者线程调用 task_done() 表示这个条目已经被回收,该条目所有工作已经完成,未完成计数就会减少。当未完成计数降到零的时候, join() 阻塞被解除。

如何等待排队的任务被完成的示例:

def worker():
while True:
item = q.get()
if item is None:
break
do_work(item)
q.task_done() q = queue.Queue()
threads = []
for i in range(num_worker_threads):
t = threading.Thread(target=worker)
t.start()
threads.append(t) for item in source():
q.put(item) # block until all tasks are done
q.join() # stop workers
for i in range(num_worker_threads):
q.put(None)
for t in threads:
t.join()

一个用于多进程上下文的队列类(而不是多线程)。

collections.deque 是无界队列的替代实现,具有快速原子的 append() 和 popleft() 操作,不需要锁定。

queue 官方运用的更多相关文章

  1. 搭建高可用的rabbitmq集群 + Mirror Queue + 使用C#驱动连接

    我们知道rabbitmq是一个专业的MQ产品,而且它也是一个严格遵守AMQP协议的玩意,但是要想骚,一定需要拿出高可用的东西出来,这不本篇就跟大家说 一下cluster的概念,rabbitmq是erl ...

  2. Unity性能优化(4)-官方教程Optimizing graphics rendering in Unity games翻译

    本文是Unity官方教程,性能优化系列的第四篇<Optimizing graphics rendering in Unity games>的翻译. 相关文章: Unity性能优化(1)-官 ...

  3. Berkeley DB的数据存储结构——哈希表(Hash Table)、B树(BTree)、队列(Queue)、记录号(Recno)

    Berkeley DB的数据存储结构 BDB支持四种数据存储结构及相应算法,官方称为访问方法(Access Method),分别是哈希表(Hash Table).B树(BTree).队列(Queue) ...

  4. SWFUpload 2.5.0版 官方说明文档 中文翻译版

    原文地址:http://www.cnblogs.com/youring2/archive/2012/07/13/2590010.html#setFileUploadLimit SWFUpload v2 ...

  5. python中的进程、线程(threading、multiprocessing、Queue、subprocess)

    Python中的进程与线程 学习知识,我们不但要知其然,还是知其所以然.你做到了你就比别人NB. 我们先了解一下什么是进程和线程. 进程与线程的历史 我们都知道计算机是由硬件和软件组成的.硬件中的CP ...

  6. RabbitMQ学习总结 第三篇:工作队列Work Queue

    目录 RabbitMQ学习总结 第一篇:理论篇 RabbitMQ学习总结 第二篇:快速入门HelloWorld RabbitMQ学习总结 第三篇:工作队列Work Queue RabbitMQ学习总结 ...

  7. 关于Plupload结合上传插件jquery.plupload.queue的使用

    之前使用过很多的上传组件,但对各种浏览器的兼容性太差,不得不放弃!! plupload 是款很强大的上传组件,不得不推荐.plupload 前端根据浏览器不同选择使用Html5. Gears, Sil ...

  8. Android官方多媒体API Mediacodec翻译(一)

    因近期工作调整,关于Mediacodec部分的翻译会暂停,后续有时间一定补上,非常抱歉. 本文章为根据Android Mediacodec官方英文版的原创翻译,转载请注明出处:http://www.c ...

  9. 【iOS开发必备指南合集】申请企业级IDP、真机调试、游戏接入GameCenter 指南(实现仿官方的成就提示)、游戏接入OpenFeint指南;

    本站文章均为李华明Himi原创,转载务必在明显处注明:(作者新浪微博:@李华明Himi) 转载自[黑米GameDev街区] 原文链接: http://www.himigame.com/iphone-c ...

随机推荐

  1. [SUCTF 2019]EasySQL(堆叠注入配合sql_mode)

    考点:1.堆叠注入 2.set sql_mode=PIPES_AS_CONCAT;将||视为字符串的连接操作符而非或运算符 意外:注入* 复现: 1;set sql_mode=PIPES_AS_CON ...

  2. AcWing 891. Nim游戏

    //a1 ^ a2 ^ ··· ^ an = 0 –>先手必败: //a1 ^ a2 ^ ··· ^ an != 0 –>先手必胜: #include<iostream> us ...

  3. 【转】Error,java对常量池来说字符串xxx的UTF8表示过长的解决及其理解

    做项目,客户端这里自己模拟json数据时,一时没忍住搞得json字符串太长了(idea上大概有600多行吧),这个问题就蹦出来了.老方法直接google.百度然后看到一堆有关String 字符串常量池 ...

  4. NG-ALAIN 边学边记2

    1. 下载Git上的源码  : https://github.com/ng-alain/ng-alain.git 2.解压文件: 3.进入到目录下: 4.打开CMD 切换到 E:\NgAlain\ng ...

  5. 题解【洛谷P3958】[NOIP2017]奶酪

    题面 题解 我们考虑使用一个并查集维护空洞之间的关系. 如果两个空洞能相互到达,那么它们的祖先也是相同的. 枚举从哪一个空洞开始,能否到达奶酪的上表面. 如果能到达就输出Yes,否则输出No. 注意开 ...

  6. Python之路Day02

    一.While while -- 循环 while 条件: 循环体 break -- 终止当前循环 while True:    print("爱情买卖")    break   ...

  7. centos7 tar.gz zip 解压命令

    tar负责打包,gzip负责压缩 tar -c: 建立压缩档案 -x:解压 -t:查看内容 -r:向压缩归档文件末尾追加文件 -u:更新原压缩包中的文件 这五个是独立的命令,压缩解压都要用到其中一个, ...

  8. C++-main函数与命令行参数

    1.main函数的概念 C语言中main函数称之为主函数 —个C程序是从main函数开始执行的 下面的main函数定义正确吗? //1 main(){ } //2 void main(){ } //3 ...

  9. 基于Java在线学习系统设计与实现

                 Spring+SpringMVC+MyBatis+Bootstrap+Vue开发在线学习系统 本课题的主要内容是开发基于Java EE的在线学习平台,使用MVC经典开发模式. ...

  10. python3练习100题——006

    继续做题-经过py3测试 原题链接:http://www.runoob.com/python/python-exercise-example6.html 题目:斐波那契数列. 我的代码: def fi ...