(由于这是发在memcached邮件列表的，所以只能用一下蹩脚的英文了）

(you should read the discuss about issue #260 first:  https://github.com/memcached/memcached/pull/67)

Thanks for dormando to pull me back to the do_item_alloc function, and i also read again carefully about how to reproduce and fix the issue #260. It is awful  to find the extremely path to reproduce the but. But i m afraid there MAY have other problems, even at vervion .20, more higher problibility.　　

reproduce step by step in v 1.4.20:

1) hash expanding, switch the item_lock to global_lock;
2) In thread A, do_item_alloc try to get a new item, and try to get a ITEM from the LRU tailer. but it does not lock the ITEM, because it try to hold the item_locks[];
3) after do_item_alloc check the refcount(after line items.c:139), other thread B try to hold the ITEM, and increase the refcount, and believe itself hold a refcount;
4) In thread A, do_item_alloc evicted the ITEM, and initilize the ITEM(reset the refcount to 1),  as a new one returning to the caller;
5) In thread B, it call item_remove to dereference the refcount, and make the refcount to 0, so the ITEM is free! (the thread A is holding the ITEM, and the ITME is in the hash table yet);
6) So, any terrible thing may happen, including crash and dead loop.

In version lower than 1.4.19, it easier to occurr even if there is no hash expanding.  Because the code in do do_item_alloc as following:

1) if the cur_hv == cur_hv, (it is possible at replace operation);
or
2) if the first loop hv!=cur_hv, and hold a lock; then the second loop hv==cur_hv(without a lock, but the lock's pointer is not reset in the first loop!), then release the lock wrong.

  1.     for (; tries >  && search != NULL; tries--, search=search->prev) {
  2.         uint32_t hv = hash(ITEM_key(search), search->nkey, );
  3.         /* Attempt to hash item lock the "search" item. If locked, no
  4.          * other callers can incr the refcount
  5.          */
  6.         /* FIXME: I think we need to mask the hv here for comparison? */
  7.         if (hv != cur_hv && (hold_lock = item_trylock(hv)) == NULL)   ------>
  8.             continue;
  9.         /* Now see if the item is refcount locked */
  10.         if (refcount_incr(&search->refcount) != ) {
  11.             refcount_decr(&search->refcount);
  12.             /* Old rare bug could cause a refcount leak. We haven't seen
  13.              * it in years, but we leave this code in to prevent failures
  14.              * just in case */
  15.             if (search->time + TAIL_REPAIR_TIME < current_time) {
  16.                 itemstats[id].tailrepairs++;
  17.                 search->refcount = ;
  18.                 do_item_unlink_nolock(search, hv);
  19.             }
  20.             if (hold_lock)
  21.                 item_trylock_unlock(hold_lock);
  22.             continue;
  23.         }

I think it's clear enough, but i also try to show more detail as following, and try to reproduce the phenomenon in gdb:

I don't know why the function "assoc_maintenance_thread" writed as following, but other threads will get the global_lock before "assoc_maintenance_thread" get it again.

  1. static void *assoc_maintenance_thread(void *arg) {
  2.  
  3.     while (do_run_maintenance_thread) {
  4.         int ii = ;
  5.  
  6.         /* Lock the cache, and bulk move multiple buckets to the new
  7.          * hash table. */
  8.         item_lock_global();                   ----------------> step 5) try to get the global_lock, it have to wait for other threads to                                                   -------------release the lock.
  9.         mutex_lock(&cache_lock);
  10.  
  11.         for (ii = ; ii < hash_bulk_move && expanding; ++ii) {
  12.            .....
  13.         }
  14.  
  15.         mutex_unlock(&cache_lock);
  16.         item_unlock_global();
  17.  
  18.         if (!expanding) {
  19.             /* finished expanding. tell all threads to use fine-grained locks */
  20.             switch_item_lock_type(ITEM_LOCK_GRANULAR);
  21.             slabs_rebalancer_resume();
  22.             /* We are done expanding.. just wait for next invocation */
  23.             mutex_lock(&cache_lock);
  24.             started_expanding = false;
  25.             pthread_cond_wait(&maintenance_cond, &cache_lock);    ------------>step 1) wait here for expanding notify.
  26.             /* Before doing anything, tell threads to use a global lock */
  27.             mutex_unlock(&cache_lock);
  28.             slabs_rebalancer_pause();
  29.             switch_item_lock_type(ITEM_LOCK_GLOBAL);   ----------->step 2) switch to the global_lock, without holding the global_lock. 
                                                           ---- other thread will get the global_lock first. it is not thread-safe.
  30.             mutex_lock(&cache_lock);
  31.             assoc_expand();                            ---->step 3) expand the hash size, but the items are not moved to the new buckets.
  32.             mutex_unlock(&cache_lock);                 --->step 4) release the lock, MAY be not thread-safe.
  33.         }
  34.     }
  35.     return NULL;
  36. }

and in the function "do_item_alloc", using the function "item_trylock" to get the item's lock. please look carefully at "item_trylock", it directly access "item_locks", hoping a "no-op" as it said in the comment.  the item(search) is not lock at all.

So, after the checker "if (refcount_incr(&search->refcount) != 2) ", other threads may hold the item and increase the refcounte, that will make

  1. item *do_item_alloc(char *key, const size_t nkey, const int flags,
  2.                     const rel_time_t exptime, const int nbytes,
  3.                     const uint32_t cur_hv) {
  4.     uint8_t nsuffix;
  5.     item *it = NULL;
  6.     char suffix[];
  7.     size_t ntotal = item_make_header(nkey + , flags, nbytes, suffix, &nsuffix);
  8.     if (settings.use_cas) {
  9.         ntotal += sizeof(uint64_t);
  10.     }
  11.  
  12.     unsigned int id = slabs_clsid(ntotal);
  13.     if (id == )
  14.         return ;
  15.  
  16.     mutex_lock(&cache_lock);
  17.     /* do a quick check if we have any expired items in the tail.. */
  18.     int tries = ;
  19.     int tried_alloc = ;
  20.     item *search;
  21.     void *hold_lock = NULL;
  22.     rel_time_t oldest_live = settings.oldest_live;
  23.  
  24.     search = tails[id];
  25.     /* We walk up *only* for locked items. Never searching for expired.
  26.      * Waste of CPU for almost all deployments */
  27.     for (; tries >  && search != NULL; tries--, search=search->prev) {
  28.         if (search->nbytes ==  && search->nkey ==  && search->it_flags == ) {
  29.             /* We are a crawler, ignore it. */
  30.             tries++;
  31.             continue;
  32.         }
  33.         uint32_t hv = hash(ITEM_key(search), search->nkey);
  34.         /* Attempt to hash item lock the "search" item. If locked, no
  35.          * other callers can incr the refcount
  36.          */
  37.         /* Don't accidentally grab ourselves, or bail if we can't quicklock */
  38.         if (hv == cur_hv || (hold_lock = item_trylock(hv)) == NULL)      ---------------> 1) item_trylock always get lock from item_locks[], not global_lock
  39.             continue;
  40.         /* Now see if the item is refcount locked */
  41.         if (refcount_incr(&search->refcount) != ) {                     ---------------> 2) now the serch->refcount==2, means only the lru-link reference the item.
  42.             refcount_decr(&search->refcount);
  43.             /* Old rare bug could cause a refcount leak. We haven't seen
  44.              * it in years, but we leave this code in to prevent failures
  45.              * just in case */
  46.             if (settings.tail_repair_time &&
  47.                     search->time + settings.tail_repair_time < current_time) {
  48.                 itemstats[id].tailrepairs++;
  49.                 search->refcount = ;
  50.                 do_item_unlink_nolock(search, hv);
  51.             }
  52.             if (hold_lock)
  53.                 item_trylock_unlock(hold_lock);
  54.             continue;
  55.         }
  56.  
  57.         /* Expired or flushed */
  58.         if ((search->exptime !=  && search->exptime < current_time)   ------------------> 3) after this line, other thread may got hold the item and increase the refcount.
                                                                          ----------- ------- so if the item  is alloc as a new item (refcount reset to 1), then the other thread(
                                                                           -------------------hold the item) call do_item_remove, it will free the "new" item(because the refcount                                                                        -------------------is 0). this is very highly possible at the item evicted case.
  59.             || (search->time <= oldest_live && oldest_live <= current_time)) {
  60.             itemstats[id].reclaimed++;
  61.             if ((search->it_flags & ITEM_FETCHED) == ) {
  62.                 itemstats[id].expired_unfetched++;
  63.             }
  64.             it = search;
  65.             slabs_adjust_mem_requested(it->slabs_clsid, ITEM_ntotal(it), ntotal);
  66.             do_item_unlink_nolock(it, hv);
  67.             /* Initialize the item block: */
  68.             it->slabs_clsid = ;
  69.         } else if ((it = slabs_alloc(ntotal, id)) == NULL) {
  70.             tried_alloc = ;
  71.             if (settings.evict_to_free == ) {
  72.                 itemstats[id].outofmemory++;
  73.             } else {
  74.                 itemstats[id].evicted++;
  75.                 itemstats[id].evicted_time = current_time - search->time;
  76.                 if (search->exptime != )
  77.                     itemstats[id].evicted_nonzero++;
  78.                 if ((search->it_flags & ITEM_FETCHED) == ) {
  79.                     itemstats[id].evicted_unfetched++;
  80.                 }
  81.   
  1. /* Special case. When ITEM_LOCK_GLOBAL mode is enabled, this should become a
  2.  * no-op, as it's only called from within the item lock if necessary.
  3.  * However, we can't mix a no-op and threads which are still synchronizing to
  4.  * GLOBAL. So instead we just always try to lock. When in GLOBAL mode this
  5.  * turns into an effective no-op. Threads re-synchronize after the power level
  6.  * switch so it should stay safe.
  7.  */
  8. void *item_trylock(uint32_t hv) {
  9.     pthread_mutex_t *lock = &item_locks[hv & hashmask(item_lock_hashpower)];
  10.     if (pthread_mutex_trylock(lock) == ) {
  11.         return lock;
  12.     }
  13.     return NULL;
  14. } 

gdb reproduce(partial):

  1. b assoc.c:211
    b item_get
    b items.c:158
  1. (gdb) info thread
  2.   Id   Target Id         Frame
  3.   6    Thread 0x7ffff4d71700 (LWP 12647) "memcached" (running)
  4.   5    Thread 0x7ffff5572700 (LWP 12646) "memcached" (running)
  5.   4    Thread 0x7ffff5d73700 (LWP 12645) "memcached" (running)
  6. * 3    Thread 0x7ffff6574700 (LWP 12644) "memcached" (running)
  7.   2    Thread 0x7ffff6d75700 (LWP 12643) "memcached" (running)
  8.   1    Thread 0x7ffff7fe0740 (LWP 12642) "memcached" 0x00007ffff76ad9a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  1. ////step 1////add a new item, modify the value of hash_items to 100000, to making a hash expanding.
  2. (gdb) thread
  3. [Switching to thread  (Thread 0x7ffff6574700 (LWP ))]
  4. #  do_item_alloc (key=0x7ffff0026414 "e", nkey=, flags=, exptime=, nbytes=, cur_hv=) at items.c:
  5.                 tried_alloc = ;
  6. (gdb) c
  7. Continuing.
  8.  
  9. Breakpoint , assoc_insert (it=0x7ffff7f35f70, hv=) at assoc.c:
  10.         if (! expanding && hash_items > (hashsize(hashpower) * ) / ) {
  11. (gdb)         it = do_item_get(key, nkey, hv);
  12. p hash_items
  13. $ =
  14. Breakpoint , assoc_maintenance_thread (arg=0x0) at assoc.c:
  15.             item_lock_global();
  16. (gdb) info thread
  17.   Id   Target Id         Frame
  18.       Thread 0x7ffff4d71700 (LWP ) "memcached" assoc_maintenance_thread (arg=0x0) at assoc.c:
  19.       Thread 0x7ffff5572700 (LWP ) "memcached" (running)
  20.       Thread 0x7ffff5d73700 (LWP ) "memcached" (running)
  21. *     Thread 0x7ffff6574700 (LWP ) "memcached" (running)
  22.       Thread 0x7ffff6d75700 (LWP ) "memcached" (running)
  23.       Thread 0x7ffff7fe0740 (LWP ) "memcached" (running)

  24. /////delete all key in the cache.
    /////step 2/// add a item, key="71912_yhd.serial.product.get_1.0_0", to the slot 2, as the first item now.
  1. ////step 3///get the key="71912_yhd.serial.product.get_1.0_0",
  2.  
  3. ///step 4 //and try to add a new key="Y_ORDER_225426358_02" to the slot 2.
  1. Breakpoint , do_item_alloc (key=0x7ffff0026414 "71912_yhd.serial.product.get_1.0_0", nkey=, flags=, exptime=, nbytes=, cur_hv=) at items.c:
  2.                         const uint32_t cur_hv) {
  3. (gdb) n

  4. ....
  5.  
  6. Breakpoint , item_get (key=0x7fffe0026414 "71912_yhd.serial.product.get_1.0_0", nkey=) at thread.c:
  7.         hv = hash(key, nkey);
  8. (gdb) info thread
  9.   Id   Target Id         Frame
  10.       Thread 0x7ffff4d71700 (LWP ) "memcached" assoc_maintenance_thread (arg=0x0) at assoc.c:
  11.       Thread 0x7ffff5572700 (LWP ) "memcached" (running)
  12.       Thread 0x7ffff5d73700 (LWP ) "memcached" (running)
  13.       Thread 0x7ffff6574700 (LWP ) "memcached" do_item_alloc (key=0x7ffff0026414 "Y_ORDER_225426358_02", nkey=, flags=, exptime=, nbytes=, cur_hv=) at items.c:
  14. *     Thread 0x7ffff6d75700 (LWP ) "memcached" item_get (key=0x7fffe0026414 "71912_yhd.serial.product.get_1.0_0", nkey=) at thread.c:
  15.       Thread 0x7ffff7fe0740 (LWP ) "memcached" (running)
  16. (gdb) n
  17.         item_lock(hv);
  18. (gdb)
  19.         it = do_item_get(key, nkey, hv);
  20. (gdb)
  21.         item_unlock(hv);
  22.  
  23.  
  24. (gdb) thread
  25. [Switching to thread  (Thread 0x7ffff6574700 (LWP ))]
  26. #  do_item_alloc (key=0x7ffff0026414 "Y_ORDER_225426358_02", nkey=, flags=, exptime=, nbytes=, cur_hv=) at items.c:
  27.             if ((search->exptime !=  && search->exptime < current_time)
  28. (gdb) p search->refcount
  29. $ = 3                                      -------> now the refcount is wrong. (may be in the evicted process is better).
  30. (gdb) bt
  31. #  do_item_alloc (key=0x7ffff0026414 "Y_ORDER_225426358_02", nkey=, flags=, exptime=, nbytes=, cur_hv=) at items.c:
  32. #  0x0000000000417c26 in item_alloc (key=0x7ffff0026414 "Y_ORDER_225426358_02", nkey=, flags=, exptime=, nbytes=) at thread.c:
  33. #  0x000000000040ace9 in process_update_command (c=0x7ffff0026200, tokens=0x7ffff6573be0, ntokens=, comm=, handle_cas=false) at memcached.c:
  34. #  0x000000000040bde2 in process_command (c=0x7ffff0026200, command=0x7ffff0026410 "set") at memcached.c:
  35. #  0x000000000040cc47 in try_read_command (c=0x7ffff0026200) at memcached.c:
  36. #  0x000000000040d958 in drive_machine (c=0x7ffff0026200) at memcached.c:
  37. #  0x000000000040e4f7 in event_handler (fd=, which=, arg=0x7ffff0026200) at memcached.c:
  38. #  0x00007ffff7ba3f24 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.
  39. #  0x0000000000417926 in worker_libevent (arg=0x635ea0) at thread.c:
  40. #  0x00007ffff7980182 in start_thread (arg=0x7ffff6574700) at pthread_create.c:
  41. # 0x00007ffff76ad30d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
  42. (gdb) 
  1. Connected to 127.0.0.1.
  2. Escape character is '^]'.
  3. set a   
  4.  
  5. STORED
  6. ^[[A^[[B
  7. ERROR
  8. set b    
  9.  
  10. STORED
  11. get 71912_yhd.serial.product.get_1.0_0
  12.  
  13. jason@gy:~$ telnet 127.0.0.1
  14. Trying 127.0.0.1...
  15. Connected to 127.0.0.1.
  16. Escape character is '^]'.
  17. set c     
  18.  
  19. STORED
  20. set 71912_yhd.serial.product.get_1.0_0   
  21.  
  22. STORED
  23. delete 71912_yhd.serial.product.get_1.0_0
  24. DELETED
  25. set 71912_yhd.serial.product.get_1.0_0   
  26.  
  27. STORED
  28. delete Y_ORDER_225426358_02
  29. delete a
  30. delete b
  31. delete c
  32. NOT_FOUND
  33. DELETED
  34. NOT_FOUND
  35. NOT_FOUND
  36. set Y_ORDER_225426358_02
  37. aaaaaaaaaaaaaaaaaaaa

I will try to write a test script to reproduce it, and try to commit a patch to fix it.

------------------------------------------------

I have got a deaploop on memcached(v1.4.15, on centos 6.3 x86_64), it can't be reproduced. In our product environment, there are hundreds of memcached instances running, and this bug happend 3 times this years.  when the bug occurred, thouands of tcp connections keep in CLOSE_WAIT status and reached the maximum connection number, then clients cann't connect to the cache servers.  The SA have to restart the memcached instance to recover our business,  but the recent time i got the chance to create a core file.

FYI:     before start, you can get the the memcached package(with a debug-info package)at: http://mirrors.htbindustries.org/CentOS/6/x86_64/,and you can get the core file at: http://pan.baidu.com/s/1kTFTRQf

backtrack in gdb.

  1. Thread  (Thread 0x7fa8a1ee1700 (LWP )):
  2. #  0x0000003c9e2e7c73 in epoll_wait () from /lib64/libc.so.
  3. #  0x000000323cc12e4b in ?? () from /usr/lib64/libevent-1.4.so.
  4. #  0x000000323cc068c3 in event_base_loop () from /usr/lib64/libevent-1.4.so.
  5. #  0x0000000000406447 in main (argc=<value optimized out>,
  6.     argv=<value optimized out>) at memcached.c:
  7.  
  8. Thread  (Thread 0x7fa89e021700 (LWP )):
  9. #  0x0000003c9e60b43c in pthread_cond_wait@@GLIBC_2.3.2 ()
  10.    from /lib64/libpthread.so.
  11. #  0x000000000040cb33 in slab_rebalance_thread (arg=<value optimized out>)
  12.     at slabs.c:
  13. #  0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  14. #  0x0000003c9e2e767d in clone () from /lib64/libc.so.
  15.  
  16. Thread  (Thread 0x7fa89ea22700 (LWP )):
  17. #  0x0000003c9e2ab91d in nanosleep () from /lib64/libc.so.
  18. #  0x0000003c9e2ab790 in sleep () from /lib64/libc.so.
  19. #  0x000000000040d0ad in slab_maintenance_thread (arg=<value optimized out>)
  20.     at slabs.c:
  21. #  0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  22. #  0x0000003c9e2e767d in clone () from /lib64/libc.so.
  23. ---Type <return> to continue, or q <return> to quit---
  24.  
  25. Thread  (Thread 0x7fa89f423700 (LWP )):
  26. #  0x0000003c9e60b43c in pthread_cond_wait@@GLIBC_2.3.2 ()
  27.    from /lib64/libpthread.so.
  28. #  0x000000000040fa8d in assoc_maintenance_thread (arg=<value optimized out>)
  29.     at assoc.c:
  30. #  0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  31. #  0x0000003c9e2e767d in clone () from /lib64/libc.so.
  32.  
  33. Thread  (Thread 0x7fa89fe24700 (LWP )):
  34. #  0x000000000040fd34 in assoc_find (key=<value optimized out>,
  35.     nkey=<value optimized out>, hv=<value optimized out>) at assoc.c:
  36. #  0x000000000040ef2e in do_item_get (key=0x7fa88008bd34 "B920818_0",
  37.     nkey=<value optimized out>, hv=) at items.c:
  38. #  0x0000000000411076 in item_get (key=0x7fa88008bd34 "B920818_0", nkey=)
  39.     at thread.c:
  40. #  0x000000000040731e in process_get_command (c=0x7fa88008bb30,
  41.     tokens=0x7fa89fe23bf0, ntokens=<value optimized out>, return_cas=false)
  42.     at memcached.c:
  43. #  0x0000000000409b63 in process_command (c=0x7fa88008bb30,
  44.     command=<value optimized out>) at memcached.c:
  45. #  0x000000000040a7e2 in try_read_command (c=0x7fa88008bb30)
  46.     at memcached.c:
  47. ---Type <return> to continue, or q <return> to quit---
  48. #  0x000000000040b478 in drive_machine (fd=<value optimized out>,
  49.     which=<value optimized out>, arg=0x7fa88008bb30) at memcached.c:
  50. #  event_handler (fd=<value optimized out>, which=<value optimized out>,
  51.     arg=0x7fa88008bb30) at memcached.c:
  52. #  0x000000323cc06b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.
  53. #  0x000000000041070d in worker_libevent (arg=0x1ad5d88) at thread.c:
  54. # 0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  55. # 0x0000003c9e2e767d in clone () from /lib64/libc.so.
  56.  
  57. Thread  (Thread 0x7fa8a0825700 (LWP )):
  58. #  0x0000003c9e6094b3 in pthread_mutex_trylock () from /lib64/libpthread.so.
  59. #  0x0000000000410d10 in mutex_lock (hv=<value optimized out>)
  60.     at memcached.h:
  61. #  item_lock (hv=<value optimized out>) at thread.c:
  62. #  0x0000000000411069 in item_get (
  63.     key=0x7fa8985d8944 "1368_yhd.orders.get_1.0_visitDateList_0", nkey=)
  64.     at thread.c:
  65. #  0x000000000040731e in process_get_command (c=0x7fa89829b7e0,
  66.     tokens=0x7fa8a0824bf0, ntokens=<value optimized out>, return_cas=false)
  67.     at memcached.c:
  68. #  0x0000000000409b63 in process_command (c=0x7fa89829b7e0,
  69.     command=<value optimized out>) at memcached.c:
  70. #  0x000000000040a7e2 in try_read_command (c=0x7fa89829b7e0)
  71. ---Type <return> to continue, or q <return> to quit---
  72.     at memcached.c:
  73. #  0x000000000040b478 in drive_machine (fd=<value optimized out>,
  74.     which=<value optimized out>, arg=0x7fa89829b7e0) at memcached.c:
  75. #  event_handler (fd=<value optimized out>, which=<value optimized out>,
  76.     arg=0x7fa89829b7e0) at memcached.c:
  77. #  0x000000323cc06b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.
  78. # 0x000000000041070d in worker_libevent (arg=0x1ad2a00) at thread.c:
  79. # 0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  80. # 0x0000003c9e2e767d in clone () from /lib64/libc.so.
  81.  
  82. Thread  (Thread 0x7fa8a1226700 (LWP )):
  83. #  0x0000003c9e6094b3 in pthread_mutex_trylock () from /lib64/libpthread.so.
  84. #  0x0000000000410d10 in mutex_lock (hv=<value optimized out>)
  85.     at memcached.h:
  86. #  item_lock (hv=<value optimized out>) at thread.c:
  87. #  0x0000000000410d7d in store_item (item=0x7fa89d0dddb8, comm=,
  88.     c=0x7fa880118a00) at thread.c:
  89. #  0x000000000040bed4 in complete_nread_ascii (fd=<value optimized out>,
  90.     which=<value optimized out>, arg=0x7fa880118a00) at memcached.c:
  91. #  complete_nread (fd=<value optimized out>, which=<value optimized out>,
  92.     arg=0x7fa880118a00) at memcached.c:
  93. #  drive_machine (fd=<value optimized out>, which=<value optimized out>,
  94.     arg=0x7fa880118a00) at memcached.c:
  95. ---Type <return> to continue, or q <return> to quit---
  96. #  event_handler (fd=<value optimized out>, which=<value optimized out>,
  97.     arg=0x7fa880118a00) at memcached.c:
  98. #  0x000000323cc06b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.
  99. #  0x000000000041070d in worker_libevent (arg=0x1acf678) at thread.c:
  100. # 0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  101. # 0x0000003c9e2e767d in clone () from /lib64/libc.so.
  102.  
  103. Thread  (Thread 0x7fa8a1c27700 (LWP )):
  104. #  0x0000003c9e6094b3 in pthread_mutex_trylock () from /lib64/libpthread.so.
  105. #  0x0000000000410d10 in mutex_lock (hv=<value optimized out>)
  106.     at memcached.h:
  107. #  item_lock (hv=<value optimized out>) at thread.c:
  108. #  0x0000000000411069 in item_get (
  109.     key=0x7fa881755f64 "4872_yhd.orders.get_1.0_0", nkey=) at thread.c:
  110. #  0x000000000040731e in process_get_command (c=0x7fa88049e180,
  111.     tokens=0x7fa8a1c26bf0, ntokens=<value optimized out>, return_cas=false)
  112.     at memcached.c:
  113. #  0x0000000000409b63 in process_command (c=0x7fa88049e180,
  114.     command=<value optimized out>) at memcached.c:
  115. #  0x000000000040a7e2 in try_read_command (c=0x7fa88049e180)
  116.     at memcached.c:
  117. #  0x000000000040b478 in drive_machine (fd=<value optimized out>,
  118.     which=<value optimized out>, arg=0x7fa88049e180) at memcached.c:
  119. ---Type <return> to continue, or q <return> to quit---
  120. #  event_handler (fd=<value optimized out>, which=<value optimized out>,
  121.     arg=0x7fa88049e180) at memcached.c:
  122. #  0x000000323cc06b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.
  123. # 0x000000000041070d in worker_libevent (arg=0x1acc2f0) at thread.c:
  124. # 0x0000003c9e607851 in start_thread () from /lib64/libpthread.so.
  125. # 0x0000003c9e2e767d in clone () from /lib64/libc.so.
  126. (gdb)
  127. (gdb)

It's easy to know that there is something wrong in thread 4, also know that the thread 4 got a lock of an item then other thread is block by the lock.

  1. (gdb) thread
  2. [Switching to thread  (Thread 0x7fa89fe24700 (LWP ))]#  0x000000000040fd34 in assoc_find (key=<value optimized out>, nkey=<value optimized out>,
  3.     hv=<value optimized out>) at assoc.c:
  4.               if ((nkey == it->nkey) && (memcmp(key, ITEM_key(it), nkey) == )) {
  5. (gdb) bt
  6. #  0x000000000040fd34 in assoc_find (key=<value optimized out>,
  7.     nkey=<value optimized out>, hv=<value optimized out>) at assoc.c:
  8. #  0x000000000040ef2e in do_item_get (key=0x7fa88008bd34 "B920818_0",
  9.     nkey=<value optimized out>, hv=) at items.c:
  10. #  0x0000000000411076 in item_get (key=0x7fa88008bd34 "B920818_0", nkey=)
  11.     at thread.c:
  12. #  0x000000000040731e in process_get_command (c=0x7fa88008bb30,
  13.     tokens=0x7fa89fe23bf0, ntokens=<value optimized out>, return_cas=false)
  14.     at memcached.c:
  15. #  0x0000000000409b63 in process_command (c=0x7fa88008bb30,
  16.     command=<value optimized out>) at memcached.c:
  17. #  0x000000000040a7e2 in try_read_command (c=0x7fa88008bb30)
  18.     at memcached.c:
  19. #  0x000000000040b478 in drive_machine (fd=<value optimized out>,
  20.     which=<value optimized out>, arg=0x7fa88008bb30) at memcached.c:
  21. #  event_handler (fd=<value optimized out>, which=<value optimized out>,
  22.     arg=0x7fa88008bb30) at memcached.c:
  23. #  0x000000323cc06b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.2
  1. (gdb) p it
  2. $ = (item *) 0x7fa89799e5b0
  3. (gdb) p *it
  4. $ = {next = 0x7fa89d0646c0, prev = 0x7fa8979b0760, h_next = 0x7fa89799e5b0, time = , exptime = , nbytes = , refcount = , nsuffix =  '\n',
  5.   it_flags =  '\003', slabs_clsid =  '\002', nkey =  '"', data = 0x7fa89799e5b0}
  6. (gdb)

so the it->h_next is point to itself, then the deadloop happend.

i have check all the code that modify the value it->h_next, (the assoc_insert/assoc_deleteassoc_maintenance_thread function), whenever it is modified, it  have to get the lock "cache_lock". so i can't find the reason why an item pointed to itself.

 more infomation:  

In thread 4:

si r8 --> nkey=9

di r9 --> key ="B920818_0"
hv = 0xe1db11c7,  so offset in primary_hashtable is=0x111c7

(gdb) p primary_hashtable
$22 = (item **) 0x7fa8840008c

so i get the first item in the hash bucket.

  1. (gdb) x /10g 0x00007fa8840008c0+0x8*0x111c7
  2. 0x7fa8840896f8: 0x00007fa890b46ef0

and then the deadloop item's address is the second item in the hash bucket list. yes, there is hash conflict.

  1. (gdb) x /10xg 0x00007fa890b46ef0
  2. 0x7fa890b46ef0: 0x00007fa89d1884f0 0x00007fa89d1ee2b0
  3. 0x7fa890b46f00: 0x00007fa89799e5b0 0x01f3710601f362f6
  4. 0x7fa890b46f10: 0x0307000100000003 0x0000000000001501
  5. 0x7fa890b46f20: 0x000000030ed02eb6 0x4544524f5f594247
  6. 0x7fa890b46f30: 0x3632343532325f52 0x332032305f383533
  7. (gdb)

the item in the hash bucket list is:

  1. (gdb) p it
  2. $ = (item *) 0x7fa89799e5b0
  3. (gdb) p *it
  4. $ = {next = 0x7fa89d0646c0, prev = 0x7fa8979b0760, h_next = 0x7fa89799e5b0, time = , exptime = , nbytes = , refcount = , nsuffix =  '\n',
  5.   it_flags =  '\003', slabs_clsid =  '\002', nkey =  '"', data = 0x7fa89799e5b0}
  6. (gdb)
  7. (gdb) p *(item *)0x00007fa89d1884f0
  8. $ = {next = 0x7fa890b4f110, prev = 0x7fa890b46ef0, h_next = 0x0, time = , exptime = , nbytes = , refcount = , nsuffix =  '\a',
  9.   it_flags =  '\003', slabs_clsid =  '\001', nkey =  '\025', data = 0x7fa89d1884f0}
  10. (gdb)

item[1]:"71912_yhd.serial.product.get_1.0_0 16384 8\r\n" nsuffix=10; slab 2; nkey=34; nbytes=10

(little endian)
(gdb) x /100xb 0x00007fa89799e5b0
0x7fa89799e5b0: 0xc0 0x46 0x06 0x9d 0xa8 0x7f 0x00 0x00
0x7fa89799e5b8: 0x60 0x07 0x9b 0x97 0xa8 0x7f 0x00 0x00
0x7fa89799e5c0: 0xb0 0xe5 0x99 0x97 0xa8 0x7f 0x00 0x00
0x7fa89799e5c8: 0x79 0x48 0xe3 0x01 0xf9 0x99 0xe4 0x01
0x7fa89799e5d0: 0x0a 0x00 0x00 0x00 0x01 0x00 0x0a 0x03
0x7fa89799e5d8: 0x02 0x22 0x00 0x00 0x00 0x00 0x00 0x00
0x7fa89799e5e0: 0x6d 0x33 0x4f 0xd6 0x02 0x00 0x00 0x00
0x7fa89799e5e8: 0x37 0x31 0x39 0x31 0x32 0x5f 0x79 0x68
0x7fa89799e5f0: 0x64 0x2e 0x73 0x65 0x72 0x69 0x61 0x6c
0x7fa89799e5f8: 0x2e 0x70 0x72 0x6f 0x64 0x75 0x63 0x74
0x7fa89799e600: 0x2e 0x67 0x65 0x74 0x5f 0x31 0x2e 0x30
0x7fa89799e608: 0x5f 0x30 0x20 0x20 0x31 0x36 0x33 0x38
0x7fa89799e610: 0x34 0x20 0x38 0x0d 0x0a 0x00 [0x00 0x00   8\r\n"
0x7fa89799e618: 0x00 0x00 0x00 0x00 0x2d 0x0d] 0x0a 0x0a
0x7fa89799e620: 0x08 0xe1 0x0d 0x0a 0x0d 0x70 0x0d 0x0a
(key

  1.  

item[0]:"Y_ORDER_225426358_02 32 1\r\n1\r\n513\r\n" nssuffix=7,slab=1,nkey=21,nbytes=3
item[1]:"71912_yhd.serial.product.get_1.0_0  16384 8\r\n"   nsuffix=10;  slab 2; nkey=34; nbytes=10
when the deadloop happened, it is looking for key "B920818_0". and the key is not in the list(may be it is).

it is odd that the item[1] have no data, although the nbyte is 10!

----------------

i guess that the bug occurred when hash expanding.  the hashpower=17(the default is 16), and   hash_items=101025, the hash expand when  hash_items>98304. It is very possible that the dealoop hapened after expanding, and then all the thread is hanged.

  1. (gdb) p expand_bucket
  2. $ =
  3. (gdb) p stats.hash_is_expanding
  4. $ = false
  5. (gdb) p hash_items
  6. $ =
  7. (gdb) p **/
  8. $ = 98304
  9.  
  10. (gdb) p expanding

$20 = false
(gdb)

(gdb) p hashpower
$21 = 17
(gdb)

  1.  

i can't not find the bug in the code, so if any guys have any suggestion, please tell me.

thanks a lot.

memcached空指针内存错误与死循环问题分析（memcached dead loop and crash bug! issue #260 and issue #370）的更多相关文章

  1. AddressSanitizer —— ASAN分析内存错误

    简介 AddressSanitizer 是一个性能非常好的C/C++ 内存错误探测工具. 它由编译器的插桩模块和替换了malloc函数的运行时库组成. 这个工具可以探测如下这些类型的错误: 对堆.栈和 ...

  2. [转]C++常见内存错误汇总

    在系统开发过程中出现的bug相对而言是比较好解决的,花费在这个上面的调试代价不是很大,但是在系统集成后的bug往往是难以定位的bug(最好方式是打桩,通过打桩可以初步锁定出错的位置,如:进入函数前打印 ...

  3. memcached学习笔记——存储命令源码分析上篇

    原创文章,转载请标明,谢谢. 上一篇分析过memcached的连接模型,了解memcached是如何高效处理客户端连接,这一篇分析memcached源码中的process_update_command ...

  4. memcached的内存管理与删除机制

    memcached的内存管理与删除机制 简介 注意:Memcache最大的value也只能是1M的空间,超过1M的数据无法保存(修改memcache源代码).   注意:内存碎片化永远都存在,只是哪一 ...

  5. setter方法的内存错误

    - (void)setList:(ClassicList *)list { self.list = list; _titleLabel.text = list.activityName; _addre ...

  6. memcached学习笔记——存储命令源码分析下篇

    上一篇回顾:<memcached学习笔记——存储命令源码分析上篇>通过分析memcached的存储命令源码的过程,了解了memcached如何解析文本命令和mencached的内存管理机制 ...

  7. memcached学习——memcached的内存分配机制Slab Allocation、内存使用机制LRU、常用监控记录（四）

    内存分配机制Slab Allocation 本文参考博客:https://my.oschina.net/bieber/blog/505458 Memcached的内存分配是以slabs为单位的,会根据 ...

  8. Memcached源代码分析 - Memcached源代码分析之消息回应（3）

    文章列表: <Memcached源代码分析 - Memcached源代码分析之基于Libevent的网络模型(1)> <Memcached源代码分析 - Memcached源代码分析 ...

  9. 小心DLL链接静态库时的内存错误

    本文转自http://www.bennychen.cn/2010/09/%E5%B0%8F%E5%BF%83dll%E9%93%BE%E6%8E%A5%E9%9D%99%E6%80%81%E5%BA% ...

随机推荐

  1. python内置下载服务器

    python内置了一个下载服务器.例如你的同事要让你传的文件位于某一个目录下面,那么你可以进入这个目录,然后执行下面的命令启动一个下载服务器 python2 python -m SimpleHTTPS ...

  2. redis 模拟redis server接收信息

    一.实现说明 客户端使用jedis正常set值到redis服务器   2. 模拟服务器接收jedis发送的信息 二.jedis客户端代码 package com.ahd.redis; import r ...

  3. 关于IDEA中@Autowired 注解报错~图文

    例如鼠标放上去会报错如下: Could not autowire. No beans of 'StudentMapper' type found. less... (Ctrl+F1) Inspecti ...

  4. lilo - 安装引导装入程序

    总述 主要功能: ” /sbin/lilo” - 安装引导装入程序 辅助用途: ”/sbin/lilo –q” - 查询影射表 ”/sbin/lilo –R” - 设置下次启动的默认命令行 ”/sbi ...

  5. python-文件的修改

    python-文件的修改 修改文件的方法 第一种方法: 第二种方法: f=open("my-heart","r") f_new=open("my-he ...

  6. Nginx Windows下安装使用及权重分配

    内容目录 Nginx 下载启动Nginx关闭NginxNginx使用注意事项使用Nginx代理服务器做负载均衡Nginx配置静态资源Nginx权重分配方式Nginx负载均衡参数描述写在最后 Nginx ...

  7. thinkphp 多条件联合查询 where例句

    $where['username'] = array("eq",$username); $where['phone'] = array("eq",$userna ...

  8. [工具] BurpSuite--XssValidator插件

    0x00 安装 所需软件: 1.burpsuite 2.xssvalidator 源码:https://github.com/nVisium/xssValidator(按照编译指导编译) burpsu ...

  9. 安装BCG界面库 会导致vs2013qt库配置消失

    安装BCG界面库 会导致vs2013qt库配置消失 安装BCG界面库 会导致vs2013qt库配置消失 安装BCG界面库 会导致vs2013qt库配置消失

  10. layui中从子窗口传递数据到父窗口,第三个子弹层的值传给第二个弹层

    最近做一个项目的需要多个弹层,每个弹层中还需要数据传递, 经过测试,以下方法三个弹层才有效,如果只是有两个弹层,请用其它方法 大概如图,看图自己应该明白 如何在在b页面选择好的值传给a页面的问题,这个 ...