Python的并发并行[1] -> 线程[2] -> 锁与信号量

锁与信号量

---

目录

1. 添加线程锁
2. 锁的本质
3. 互斥锁与可重入锁
4. 死锁的产生
5. 锁的上下文管理
6. 信号量与有界信号量

1 添加线程锁

由于多线程对资源的抢占顺序不同，可能会产生冲突，通过添加线程锁来对共有资源进行控制。

 import atexit
 from random import randrange
 from threading import Thread, Lock, current_thread # or currentThread
 from time import ctime, sleep

 # Use set to record the running thread
 # If we print set directly, it will shows set([a, b, c])
 # So we reload the __str__ function to show it more clean(shows a, b, c as we want)
 class CleanOutputSet(set):
     def __str__(self):
         return ', '.join(x for x in self)

 # randrange(2, 5) will generate a num in range(2, 5)
 # for x in range(randrange(3, 7)) will do "for" loop 3-7 times
 # Below code generate a list which contains 3-6 numbers in range 2-4
 # If not list(), it will return a generator, and it will be null after one time iteration
 loops = list((randrange(2, 5) for x in range(randrange(3, 7))))

 remaining = CleanOutputSet()
 lock = Lock()

 def loop_without_lock(nsec):
     myname = current_thread().name
     remaining.add(myname)
     print('[%s] Start %s' % (ctime(), myname))
     sleep(nsec)
     remaining.remove(myname)
     print('[%s] Completed %s (%d secs)' % (ctime(), myname, nsec))
     # Note: remaining or 'NONE', will return 'NONE' if remaining is Null
     # Null including: None, False, (), [], {}, 0
     print('    (remaining: %s)' % (remaining or 'NONE'))

 def loop_with_lock(nsec):
     myname = current_thread().name
     # When we need to modifiy public resource, acquire lock to block other threads
     # Lock acquire and release can use 'with lock' to simplify
     lock.acquire()
     remaining.add(myname)
     print('[%s] Start %s' % (ctime(), myname))
     # After using resource, release lock for other threads to use
     lock.release()
     sleep(nsec)

     lock.acquire()
     remaining.remove(myname)
     print('[%s] Completed %s (%d secs)' % (ctime(), myname, nsec))
     print('    (remaining: %s)' % (remaining or 'NONE'))
     lock.release()

 def _main():
     print('-----Below threads without lock-----')
     threads = []
     for pause in loops:
         threads.append(Thread(target=loop_without_lock, args=(pause, )))
     for t in threads:
         t.start()
     for t in threads:
         t.join()
     print('-----Below threads with lock-----')
     threads = []
     for pause in loops:
         threads.append(Thread(target=loop_with_lock, args=(pause, )))
     for t in threads:
         t.start()
     for t in threads:
         t.join()

 # This is an exit function, when script exit, this function will be called
 # You can use atexit.register(_atexit) to replace @atexit.register
 # The function name '_atexit' can be change to others
 @atexit.register
 def _atexit():
     print('All DONE at:', ctime())

 if __name__ == '__main__':
     _main()

第 1-4 行，首先导入需要的模块，atexit用于设置退出脚本时的处理函数，random用于产生随机数来增加线程的不确定性。

第 7- 12 行，定义一个新的集合类，用于输出当前运行线程的集合，新的集合类CleanOutputSet重载了__str__方法，使集合的显示由set([a, b, c])变为我们想要的a, b, c形式。

第 14-21行，利用随机数模块，产生一个生成器，包含3-6个大小在2-4之间的随机数，为了后续重复使用，此处对生成器进行list操作，否则生成器在迭代一次之后将变为空，无法复用。同时对全局锁和集合输出类进行实例化。

第 23-32 行，定义一个不加锁的线程函数，该函数会在进入时向集合添加线程名，sleep相应时间后，移除线程名，同时显示集合(共有资源)内剩余的线程名。

第 34-49 行，定义一个加锁的线程函数，该函数会在进入时获取线程锁，之后再向集合添加线程名，添加完成后释放线程锁，sleep相应时间后，获取线程锁，移除线程名，同时显示集合(共有资源)内剩余的线程名，最后释放线程锁。

第 51-67 行，主函数中分别对加锁和不加锁的两种线程方式进行调用，并利用join()方法挂起线程以区分开两种方式的运行。

第 69-77 行，利用atexit.register函数/@register装饰器定义脚本退出函数。

最后输出结果

-----Below threads without lock-----
[Tue Aug  1 11:01:57 2017] Start Thread-1
[Tue Aug  1 11:01:57 2017] Start Thread-2[Tue Aug  1 11:01:57 2017] Start Thread-3
[Tue Aug  1 11:01:57 2017] Start Thread-4  

[Tue Aug  1 11:01:59 2017] Completed Thread-3 (2 secs)
    (remaining: Thread-1, Thread-4, Thread-2)
[Tue Aug  1 11:02:00 2017] Completed Thread-1 (3 secs)
    (remaining: Thread-4, Thread-2)
[Tue Aug  1 11:02:01 2017] Completed Thread-2 (4 secs)[Tue Aug  1 11:02:01 2017] Completed Thread-4 (4 secs)  

    (remaining: NONE)    (remaining: NONE)  

-----Below threads with lock-----
[Tue Aug  1 11:02:01 2017] Start Thread-5
[Tue Aug  1 11:02:01 2017] Start Thread-6
[Tue Aug  1 11:02:01 2017] Start Thread-7
[Tue Aug  1 11:02:01 2017] Start Thread-8
[Tue Aug  1 11:02:03 2017] Completed Thread-7 (2 secs)
    (remaining: Thread-8, Thread-6, Thread-5)
[Tue Aug  1 11:02:04 2017] Completed Thread-5 (3 secs)
    (remaining: Thread-8, Thread-6)
[Tue Aug  1 11:02:05 2017] Completed Thread-6 (4 secs)
    (remaining: Thread-8)
[Tue Aug  1 11:02:05 2017] Completed Thread-8 (4 secs)
    (remaining: NONE) 

从输出的结果中可以看到，不加线程锁时，可能出现print的错乱以及同时输出两个剩余线程为NONE的结果，而加了线程锁之后，线程对资源的使用则按照顺序进行。

2 锁的本质

为验证锁的本质作用并非通过锁定资源实现，利用代码创建两个线程，其中一个正常线程会遵循约定，在获取锁之后再对共有资源进行修改，而另一个异常线程则不会尝试获取锁，且在没有锁的权限基础上，直接对共有资源进行修改。

 from threading import Thread, Lock
 import time
 lock = Lock()
 COUNT = 0

 def gentleCounter():
     name = 'gentleCounter'
     lock.acquire()
     print('%s acquired the lock' % name)
     global COUNT
     COUNT += 1
     print('%s made a count plus, now the COUNT is %d' % (name, COUNT))
     print('%s is taking a rest...' % name)
     time.sleep(3)
     COUNT += 1
     print('%s made a count plus again, now the COUNT is %d' % (name, COUNT))
     lock.release()
     print('%s released the lock' % name)

 def wildCounter():
     time.sleep(1)
     name = 'wildCounter'
     print('%s didn\'t acquire the lock' % name)
     global COUNT
     COUNT += 1
     print('%s made a count plus, now the COUNT is %d' % (name, COUNT))

 Thread(target=gentleCounter).start()
 Thread(target=wildCounter).start()

第 1-4 行，导入线程和锁两个类，并定义一个锁对应的全局变量COUNT，赋予初值。

第 6-18 行，定义一个正常计数方法，该方法遵循规则，会在获取锁后对COUNT进行+1操作，随后休眠3秒再次进行COUNT+1操作，在这期间不会对锁释放，两次加值之后，正常计数方法会释放锁。

第 20-26 行，定义一个异常计数方法，该方法不会对锁进行获取，它会在进入时等待1秒，当正常计数方法的线程进入休眠状态时，不获取锁权限直接对COUNT进行+1操作。

第 28-29 行，将两种方法以两个线程分别启动。

最后得到结果

gentleCounter acquired the lock
gentleCounter made a count plus, now the COUNT is 1
gentleCounter is taking a rest...
wildCounter didn't acquire the lock
wildCounter made a count plus, now the COUNT is 2
gentleCounter made a count plus again, now the COUNT is 3
gentleCounter released the lock  

从输出的结果中可以看到，wildCounter在gentleCounter休眠且锁未释放的期间，成功修改了共有资源，影响了gentleCounter的下一次计数。因此锁在物理上并未真正的对资源进行锁死，而是线程之间相互约定的一种修改权限。

3 互斥锁与可重入锁

Python threading模块有两类锁，互斥锁（threading.Lock ）和可重用锁（threading.RLock）。两者的用法基本相同。但互斥锁只能被获取一次，若多次获取则会产生阻塞，需等待原锁释放后才能再次入锁。而可重入锁则可被本线程多次acquire入锁，但是要求入锁次数与释放次数相同，才能完全解锁，且锁的释放需要在同一个线程中进行。

 from threading import Lock, RLock

 def call():
     print('This is call() function')
     with lock:
         g()
         h()

 def g():
     if not lock.acquire(True, 1):
         print('g() acquires lock failed')
     else:
         print('This is g() function')
         lock.release()
     h()

 def h():
     if not lock.acquire(True, 1):
         print('h() acquires lock failed')
     else:
         print('This is h() function')
         lock.release()

 print('\n-------Using Lock-------')
 lock = Lock()
 call()
 print('\n-------Using RLock-------')
 lock = RLock()
 call()

第 3-22 行，定义call函数，在call函数中会利用with lock进行一次入锁，入锁后调用g()和h()，其中g和h函数的功能都是尝试对锁进行获取，获取失败或超时则输出失败语句，成功则显示函数调用并释放锁，而在g中会对h函数再进行一次调用。

第 24-29 行，运行程序，分别使用互斥锁和可重入锁进行试验。

输出结果

-------Using Lock-------
This is call() function
g() acquires lock failed
h() acquires lock failed
h() acquires lock failed  

-------Using RLock-------
This is call() function
This is g() function
This is h() function
This is h() function  

最终程序输出中可以看出，对于互斥锁，在call函数获取锁之后，所有的函数都无法再次入锁，而对于可重入锁，其余函数可以再次入锁并且入锁与释放成对即可。

Note: 值得注意的是，对于互斥锁，当一个线程获取锁之后，可以在另一个线程中释放锁，而对于可重入锁，则必须在同一个线程中对锁的获取进行释放。

4 死锁的产生

死锁出现在一个资源被多次调用，而调用方均未能释放资源，便会造成死锁现象。死锁大致可分为两种形式出现，迭代死锁和相互调用死锁。

4.1 迭代死锁

迭代死锁一般出现在同一个线程中对锁的多次获取，例如函数迭代嵌套，最终导致无法释放的现象。其本质依旧是acquire的阻塞，本线程进行了acquire时，其他线程或本线程再次acquire则会产生线程的阻塞等待，若是其他线程的acquire则可以等待至release，而若是本线程的二次acquire，则阻塞后无法调用release进行释放，因此产生了死锁现象。

 from threading import Thread, Lock, RLock, current_thread

 mutex = Lock()
 reentrant = RLock()
 class MyThread(Thread):
     def __init__(self, lock):
         Thread.__init__(self)
         self.lock = lock

     def run(self):
         self.name = current_thread().name
         print('-------This is %s-------' % self.name)
         if self.lock.acquire():
             print('%s get lock one' % self.name, '\nTrying to get second lock')
             self.lock.acquire()
             print('Got second lock')
             print('Trying to release lock...')
             self.lock.release()
             print('First lock released')
             self.lock.release()
             print('Second lock released')
             print('Lock all released')
         print('--------Exit %s---------' %
               self.name)

 t = MyThread(reentrant)
 t.start()
 t.join()
 t = MyThread(mutex)
 t.start()
 t.join()

在上面代码的run函数中，首先对传入的lock进行获取，在未释放的前提下，再次获取锁，若此时的锁为互斥锁，则会造成死锁现象。

运行的到结果

-------This is Thread-1-------
Thread-1 get lock one
Trying to get second lock
Got second lock
Trying to release lock...
First lock released
Second lock released
Lock all released
--------Exit Thread-1---------
-------This is Thread-2-------
Thread-2 get lock one
Trying to get second lock  

从结果中可以看到，可重入锁完成了两次入锁与释放，而互斥锁则造成了死锁阻塞

4.2 互调死锁

互调死锁是在不同的线程中互相对对方已占有的资源进行了acquire，即两个线程均占有部分资源后，需要互相获取对方已占有资源锁时，都陷入阻塞等待中，从而造成了互调死锁的现象。

可以假设，有两把资源锁，一把A锁，一把B锁，当线程1执行顺序为获取A锁，获取B锁，释放B锁，释放A锁，线程2为获取A，获取B，释放B，释放A。当启动两个线程时，线程1获取到A锁，线程2获取到B锁，此时，线程1需要获取B锁，线程2需要获取A锁，而两个锁均被对方锁定，没有被释放，最终造成了互调死锁的现象。

 from threading import Thread, Lock, RLock, current_thread
 import time

 mutex_1 = Lock()
 mutex_2 = Lock()
 reentrant_1 = RLock()
 reentrant_2 = RLock()
 COUNT = 0
 class MyThread(Thread):
     def __init__(self, lock_1, lock_2):
         Thread.__init__(self)
         self.lock_1 = lock_1
         self.lock_2 = lock_2

     def run(self):
         self.name = current_thread().name
         if self.lock_1.acquire():
             print('%s got its first lock' % self.name)
             global COUNT
             COUNT += 1
             print('%s make COUNT plus one, now COUNT is %d' % (self.name, COUNT))
             time.sleep(2)
             print('%s trying to get another one...' % self.name)
             if self.lock_2.acquire():
                 print('%s got its second lock' % self.name)
                 self.lock_2.release()
                 print('%s release its second lock' % self.name)
             self.lock_1.release()
             print('%s release its first lock' % self.name)

 threads = [MyThread(mutex_1, mutex_2), MyThread(mutex_2, mutex_1)]
 #threads = [MyThread(reentrant_1, reentrant_2), MyThread(reentrant_2, reentrant_1)]
 for t in threads:
     t.start()
 for t in threads:
     t.join()

第 8-29 行，在导入必须模块后，派生了一个线程子类，这个线程子类要求先后传入两把锁的实例，在run函数中，会对先传入的锁进行获取，获取之后将COUNT+1并休眠等待（以保证另一个线程有足够的时间获取它的第一把锁），随后尝试获取传入的第二把锁，获取成功后依次释放前面的两把锁。

第 31-36 行，此处对两个线程进行实例化，两个线程的区别在于，传入锁的顺序不同，因此线程1会先获取锁1，再获取锁2，而线程2则相反。传入的锁类型可选择互斥锁或可重入锁。

运行程序得到结果

Thread-1 got its first lock
Thread-1 make COUNT plus one, now COUNT is 1
Thread-1 trying to get another one...
Thread-2 got its first lock
Thread-2 make COUNT plus one, now COUNT is 2
Thread-2 trying to get another one...  

查看程序运行结果可以发现，两个线程都在尝试获取对方手里的锁时，产生了阻塞，造成了互调死锁的现象。

Note: 值得注意的是，此处即便换成可重入锁，也不能解决互调死锁的问题，因为可重入锁仅对本线程支持可重入，对于其他线程依旧互斥。

5 锁的上下文管理

对于锁来说，同样支持上下文管理器with，下面使用with进行锁的管理

 import atexit
 from random import randrange
 import threading
 from threading import Thread, Lock
 from time import ctime, sleep

 class CleanOutputSet(set):
     def __str__(self):
         return ', '.join(x for x in self)

 # Use generator is much better than list
 # loops = (randrange(1, 7) for x in range(2, 8))
 loops = list(randrange(1, 7) for x in range(2, 8))
 remaining = CleanOutputSet()
 lock = Lock()

 def loop(nsec):
     myname = threading.current_thread().name
     with lock:
         remaining.add(myname)
         print('[{0}] Start {1}'.format(ctime(), myname))
     # loops is generator, after 'for' iteration, remains []
     if len(remaining) == len(loops):
         func_for_trial()
     sleep(nsec)
     with lock:
         remaining.remove(myname)
         print('[{0}] Completed {1} ({2} secs)'.format(ctime(), myname, nsec))
         print('    (remaining: {0})'.format(remaining or 'NONE'))

 def func_for_trial():
     count = threading.active_count()
     active_thread = threading.enumerate()
     print('There are %d active threads, \n%s' % (count, str(active_thread).replace(', ', '\n')))

 def _main():
     threads = []
     for pause in loops:
         threads.append(Thread(target=loop, args=(pause, )))
     for t in threads:
         t.start()
     for t in threads:
         t.join()

 @atexit.register
 def _atexit():
     print('All DONE at:', ctime())

 if __name__ == '__main__':
     _main()

6 信号量与有界信号量

对于信号量，是最古老的同步原语之一，是一个计数器，在资源消耗时递减，在资源释放时递增，可以认为信号量代表着资源的可用与否。信号量比锁更加灵活，可以处理当有多个线程，每个线程拥有有限资源的实例。

下面的例子模拟了一个有限资源的糖果机，使用信号量对资源进行跟踪。

 from atexit import register
 from random import randrange
 from threading import Semaphore, BoundedSemaphore, Lock, Thread
 from time import sleep, ctime
 import threading

 """
 # This Obj reload the __len__ method to return current number of semaphore
 class MySemaphore(BoundedSemaphore):
     def __len__(self):
         return self._value
 candy = MySemaphore(5)
 print(len(candy))
 """
 lock = Lock()
 MAX = 5

 def refill():
     lock.acquire()
     print('Refilling candy...')
     try:
         candyTray.release()
     except ValueError:
         print('Full, skipping')
     else:
         print('OK, current candy num is %d' % candyTray._value)
     lock.release()

 def buy():
     lock.acquire()
     print('Buying candy...')
     if candyTray.acquire(False):
         print('OK, current candy num is %d' % candyTray._value)
     else:
         print('Empty, skipping')
     lock.release()

 def producer(loops):
     for i in range(loops):
         refill()
         sleep(randrange(3))

 def consumer(loops):
     for i in range(loops):
         buy()
         sleep(randrange(3))

 def _main():
     print('Starting at', ctime())
     nloops = randrange(2, 6)
     print('THE CANDY MACHINE (full with %d bars)!' % MAX)
     # Buyer Thread
     buyer = Thread(target=consumer, args=(randrange(nloops, nloops+MAX+2), ))
     buyer.start()
     # Vendor Thread
     vendor = Thread(target=producer, args=(nloops, ))
     vendor.start()
     for t in [buyer, vendor]:
         t.join()

 @register
 def _atexit():
     print('All DONE at:', ctime())

 if __name__ == '__main__':
     print('-------BoundedSemaphore-------')
     candyTray = BoundedSemaphore(MAX)
     _main()
     print('-------Semaphore------')
     candyTray = Semaphore(MAX)
     _main()

第 7-14 行，在导入必须模块之后，派生的类重定义了一个__len__方法，调用len()方法后，会返回当前信号量的可用资源数。这样的方式使得查看资源的方式变得更加清晰。

第 15-16 行，初始化锁，设置初始信号量参数，用于模拟糖果机初始糖果数量，当使用有界信号量时，糖果数量不允许超过初始值，即糖果最多为5个。

第 18-27 行，定义一个refill函数，这个函数会在调用时进行加锁，然后释放一个信号量+1(模拟糖果机补充糖果)，并在信号量将要超过初始值时捕获异常。

第 29-36 行，定义一个buy函数，这个函数会在调用时进行加锁，然后获取一个信号量-1(模拟用户购买一个糖果)，并在信号量为0时处理阻塞。

第 38-46 行，定义生产者与购买者函数，分别以随机间隔时间执行糖果购买与糖果补充。

第 48-63 行，定义主函数，在主函数中启动购买者与糖果机线程，通过join挂起线程等待其余线程结束再退出主函数。

运行结果

-------BoundedSemaphore-------
Starting at Tue Aug  1 16:53:32 2017
THE CANDY MACHINE (full with 5 bars)!
Buying candy...
OK, current candy num is 4
Refilling candy...
OK, current candy num is 5
Refilling candy...
Full, skipping
Refilling candy...
Full, skipping
Buying candy...
OK, current candy num is 4
Buying candy...
OK, current candy num is 3
Refilling candy...
OK, current candy num is 4
Buying candy...
OK, current candy num is 3
Refilling candy...
OK, current candy num is 4
Buying candy...
OK, current candy num is 3
-------Semaphore------
Starting at Tue Aug  1 16:53:38 2017
THE CANDY MACHINE (full with 5 bars)!
Buying candy...
OK, current candy num is 4
Refilling candy...
OK, current candy num is 5
Buying candy...
OK, current candy num is 4
Refilling candy...
OK, current candy num is 5
Buying candy...
OK, current candy num is 4
Refilling candy...
OK, current candy num is 5
Refilling candy...
OK, current candy num is 6
Buying candy...
OK, current candy num is 5
Buying candy...
OK, current candy num is 4
Buying candy...
OK, current candy num is 3
Buying candy...
OK, current candy num is 2
Buying candy...
OK, current candy num is 1  

查看最终输出结果可以看到，对于有界信号量模拟的糖果机来说，当信号量(糖果)大于初始值的时候，会抛出一个异常，此处改为输出糖果机已满。而对于非有界信号量，信号量(糖果)可以超过初始值，且不设上限。

相关阅读

---

1. 基本概念

2. threading 模块

3. 多线程的建立

参考链接

---

《Python 核心编程 第3版》

https://stackoverflow.com/questions/22885775/what-is-the-difference-between-lock-and-rlock

http://www.jb51.net/article/74426.htm