用STRACE解决公司真实故障一例
这是相关分析文档。为了职业操守，已修改相关公司敏感信息~~~
~~~~~~~~~~~~~~~~~~~~~~~~
关于论坛每五分钟左右，会有warning.html跳转出现的原因调查
（warning.html）页面显示的即服务器正繁忙，请稍后重试
最近两三周，收到客服及公司同事反映，打开BBS时，会时不时出现warning.html页面的提示跳转。
本周开始正规此问题，但调查一直没有明显头绪。数据库，连接数，触发器，SQL语句抓取，PHP-FPM配置，等等都彻查了一次。
最后，用STRACE进程调用跟踪，发现主要原因，现在反馈给大家。当然，这是不是终极原因，有待进一步验证。
STRACE命令对比产生WARNING.HTML和没有产生的情况，会发现，WARING.HTML页出跳转前，出现了服务器500的未知错误，原因是调用相关的/SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/服务出错。
线索如下：
一，以下为正常调用（关键顺序蓝色标出）：
0.001946 getsockopt(6, SOL_SOCKET, SO_ERROR, [-8916466799302672384], [4]) = 0 <0.000010>
0.000044 fcntl(6, F_SETFL, O_RDWR) = 0 <0.000016>
0.000077 sendto(6, "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1\r\nHost: 1.2.3.4:8080\r\nConnection: Keep-Alive\r\nUser-Agent: PHP-SOAP/5.3.10\r\nContent-Type: text/xml; charset=utf-8\r\nSOAPAction: \"urn:search\"\r\nContent-Length: 525\r\n\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns1=\"http://service.webservice.abc.com\"><SOAP-ENV:Body><ns1:search><ns1:args0>eyJrZXkiOiJcdTMwMTA5OVx1OGQyMlx1NWJjY1x1MzAxMVAyUFx1ODg0Y1x1NGUxYVx1NWMwNlx1NGYxYVx1NjYyZlx1MjAxY1x1NGUwMFx1NWUyNlx1NGUwMFx1OGRlZlx1MjAxZFx1NjIxOFx1NzU2NVx1NGUyZFx1NzY4NFx1OTFjZFx1ODk4MVx1NGUwMFx1OTBlOFx1NTIwNiIsInN0YXR1cyI6IndkemoiLCJzdHlsZSI6IjIifQ==</ns1:args0></ns1:search></SOAP-ENV:Body></SOAP-ENV:Envelope>\n", 786, MSG_DONTWAIT, NULL, 0) = 786 <0.000034>
0.000185 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.076195>
0.076290 recvfrom(6, "HTTP/1.1 200 OK\r\nServer: nginx\r\nDate: Sat, 13 Jun 2015 06:18:59 GMT\r\nContent-Type: text/xml;charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nVary: Accept-Encoding\r\n\r\n26a7\r\n<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"><soapenv:Body><ns:searchResponse xmlns:ns=\"http://service.webservice.abc.com\"><ns:return>{\"key\":\"\343\200\22099\350\264\242\345\257\214\343\200\221P2P\350\241\214\344\270\232\345\260\206\344\274\232\346\230\257\342\200\234\344\270\200\345\270\246\344\270\200\350\267\257\342\200\235\346\210\230\347\225\245\344\270\255\347\232\204\351\207\215\350\246\201\344\270\200\351\203\250\345\210\206\",\"list\":[{\"id\":8918,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1392090431000,\"shortLink\":\"guonei\",\"title\":\"\344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\347\220\206\350\264\242\344\272\247\345\223\201\346\224\266\347\233\212\347\216\207\344\270\200\350\267\257\350\265\260\344\275\216\",\"type\":1},{\"id\":4158,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1366334071000,\"shortLink\":\"guonei\",\"title\":\"\347\275\221\347\273\234\350\236\215\350\265\204 \342\200\234\345\260\217\345\276\256\342\200\235\347\246\217\351\237\263\",\"type\":1},{\"id\":16403,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1420901203000,\"shortLink\":\"hangye\",\"title\":\"\347\275\221\350\264\267\342\200\234\346\207\222\344\272\272\342\200\235\347\246\217\351\237\263\357\274\232\346\217\255\347\247\230P2P\346\234\272\346\236\204\346\212\225\350\265\204\350\200\205"..., 8192, MSG_DONTWAIT, NULL, NULL) = 2920 <0.000018>
0.000461 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.000020>
0.000068 recvfrom(6, "0,\"shortLink\":\"guowai\",\"title\":\"\346\257\224\347\211\271\345\270\201\346\234\200\345\244\247\344\272\244\346\230\223\346\211\200Mt. Gox \347\224\263\350\257\267\347\240\264\344\272\247\346\270\205\347\256\227\",\"type\":1},{\"id\":2875,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1352164359000,\"shortLink\":\"guonei\",\"title\":\"\345\273\272\350\241\214\347\224\263\350\257\267\350\257\201\345\210\270\347\211\214\347\205\247\345\217\257\350\203\275\346\200\247\346\236\201\345\244\247\",\"type\":1},{\"id\":9545,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1394250574000,\"shortLink\":\"guonei\",\"title\":\"\347\231\276\345\272\246\345\267\262\347\224\263\350\257\267\344\270\255\345\233\275\346\260\221\350\220\245\351\223\266\350\241\214\347\211\214\347\205\247\",\"type\":1},{\"id\":19378,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1433040598000,\"shortLink\":\"hydongtai\",\"title\":\"\344\270\212\346\265\267\345\270\202\351\225\277\357\274\232\344\272\222\351\207\221\346\230\257\344\270\212\346\265\267\351\207\221\350\236\215\344\270\255\345\277\203\347\232\204\346\234\200\345\244\247\346\234\272\351\201\207\",\"type\":1},{\"id\":12248,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1403327611000,\"shortLink\":\"guonei\",\"title\":\"\344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\345\217\221\345\261\225\346\230\257\344\270\200\346\254\241\345\205\250\346\226\260\347\232\204\346\234\272\351\201\207\",\"type\":1},{\"id\":11467,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1400424772000,\"shortLink\":\"guonei\",\"title\":\"\344\270\201\344\277\212\345\263\260\357\274\232\351\207\221\350\236\215\346\212\221\345\210\266\346\230\257\346\234\272\351\201\207 \344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\346\214\221\346\210\230\345\244\232\",\"type\":1},{\"id\""..., 8192, MSG_DONTWAIT, NULL, NULL) = 5840 <0.000037>
0.000276 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.001508>
0.001584 recvfrom(6, "0,\"publishDate\":1406076160000,\"shortLink\":\"hangye\",\"title\":\"\344\270\212\345\270\202\345\205\254\345\217\270\346\266\214\345\205\245P2P\350\241\214\344\270\232 \346\230\257\347\202\222\344\275\234\350\277\230\346\230\257\344\275\234\347\247\200\357\274\237\",\"type\":1},{\"id\":19619,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1433465298000,\"shortLink\":\"hangye\",\"title\":\"P2P\346\210\220\350\202\241\344\273\267\347\202\222\344\275\234\347\245\236\345\231\250 \344\270\212\345\270\202\345\205\254\345\217\270\347\253\236\347\233\270\346\224\266\350\264\255\",\"type\":1},{\"id\":7938,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1387868704000,\"shortLink\":\"licai\",\"title\":\"\344\270\232\345\206\205\344\272\272\345\243\253\347\233\264\350\250\200\345\237\272\351\207\221\345\205\254\345\217\270\344\270\212\345\270\202\346\227\266\346\234\272\345\260\232\346\227\251\",\"type\":1},{\"id\":17738,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1426483246000,\"shortLink\":\"pingtai\",\"title\":\"\344\270\255\345\233\275\345\271\263\345\256\211\357\274\232\351\231\206\351\207\221\346\211\200\344\270\232\345\212\241\346\250\241\345\274\217\344\270\216P2P\345\205\254\345\217\270\344\270\215\345\220\214\",\"type\":1},{\"id\":17816,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1426810621000,\"shortLink\":\"pingtai\",\"title\":\"\345\271\263\345\256\211\345\271\264\346\212\245\357\274\232\351\231\206\351\207\221\346\211\200\347\275\221\350\264\267\344\272\244\346\230\223\350\247\204\346\250\241\344\270\255\345\233\275\347\254\254\344\270\200\",\"type\":1},{\"id\":18246,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1428823514000,\"shortLink\":\"zhongchou\",\"title\":\"\345\271\263\345\256\211\350\220\275\345\255\220\342\200\234\345\211\215\346\265\267\344\274\227\347\255\271\342\200\235"..., 8192, MSG_DONTWAIT, NULL, NULL) = 1332 <0.000030>
0.001170 close(6)
二，以下为出现warning.html的调用（关键顺序蓝色标出）：
0.002344 getsockopt(6, SOL_SOCKET, SO_ERROR, [-8916466799302672384], [4]) = 0 <0.000012>
0.000044 fcntl(6, F_SETFL, O_RDWR) = 0 <0.000011>
0.000071 sendto(6, "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1\r\nHost: 1.2.3.4:8080\r\nConnection: Keep-Alive\r\nUser-Agent: PHP-SOAP/5.3.10\r\nContent-Type: text/xml; charset=utf-8\r\nSOAPAction: \"urn:search\"\r\nContent-Length: 525\r\n\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns1=\"http://service.webservice.abc.com\"><SOAP-ENV:Body><ns1:search><ns1:args0>eyJrZXkiOiJcdTk4ODRcdTZkNGJcdTc5ZjBcdTUxNjhcdTc0MDNcdTY3MDBcdTViY2MxJVx1NGViYVx1NTNlM1x1OGQyMlx1NWJjY1x1NjIxNlx1NWMwNlx1OGQ4NVx1OGZjN1x1NTE3Nlx1NGY1OTk5JVx1NGU0Ylx1NTQ4Y1x1MzAxMFx1OTQ2Ylx1NTM0ZVx1NThlYlx1MzAxMSIsInN0YXR1cyI6IndkemoiLCJzdHlsZSI6IjIifQ==</ns1:args0></ns1:search></SOAP-ENV:Body></SOAP-ENV:Envelope>\n", 786, MSG_DONTWAIT, NULL, 0) = 786 <0.000031>
0.000131 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.010941>
0.011081 recvfrom(6, "HTTP/1.1 500 Internal Server Error\r\nServer: nginx\r\nDate: Sat, 13 Jun 2015 03:15:06 GMT\r\nContent-Type: text/xml;charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\n\r\n14a\r\n<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"><soapenv:Body><soapenv:Fault><faultcode>soapenv:Server</faultcode><faultstring>Exception occurred while trying to invoke service method search</faultstring><detail /></soapenv:Fault></soapenv:Body></soapenv:Envelope>\r\n0\r\n\r\n", 8192, MSG_DONTWAIT, NULL, NULL) = 522 <0.000020>
0.000346 close(6) = 0 <0.000036>
0.000184 stat("/home/abc/abc_bbs//source//class/discuz/discuz_error.php", {st_mode=S_IFREG|0644, st_size=12895, ...}) = 0 <0.000027>
0.000341 lstat("/var/log/discuz/exception_error.log", {st_mode=S_IFREG|0644, st_size=2050807, ...}) = 0 <0.000023>
0.000143 lstat("/var/log/discuz", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000018>
0.000076 lstat("/var/log", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000015>
0.000076 open("/var/log/discuz/exception_error.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 6 <0.000023>
0.000068 fstat(6, {st_mode=S_IFREG|0644, st_size=2050807, ...}) = 0 <0.000014>
0.000068 lseek(6, 0, SEEK_CUR) = 0 <0.000014>
0.000054 lseek(6, 0, SEEK_CUR) = 0 <0.000014>
0.000049 write(6, "2015-06-13 11:14:32 http://bbs.abc.com/forum.php?mod=viewthread&tid=379354&extra=page%3D120&page=1 \nException occurred while trying to invoke service method search\nArray\n:\350\241\214,\350\260\203\347\224\250\346\226\271\346\263\225:handleException\n\narray (\n 'mod' => 'viewthread',\n 'tid' => '379354',\n 'extra' => 'page=120',\n 'page' => '1',\n)\narray (\n)\n------------------\n", 344) = 344 <0.000022>
0.000107 close(6) = 0 <0.000015>
0.000074 chdir("/home/abc/abc_bbs") = 0 <0.000020>
0.000066 times({tms_utime=47093, tms_stime=6066, tms_cutime=0, tms_cstime=0}) = 3696665972 <0.000015>
0.000134 fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 <0.000018>
0.002718 rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, 8) = 0 <0.000018>
0.000114 write(5, "\5\0\0\0\0\0\0\1\0\0\0\1", 12) = 12 <0.000046>
0.000095 shutdown(5, 2 /* send and receive */) = 0 <0.000028>
0.000071 close(5) = 0 <0.000022>
0.000084 rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, 8) = 0 <0.000018>
0.000246 write(4, "\1\6\0\1\3\331\7\0Status: 302 Moved Temporarily\r\nX-Powered-By: PHP/5.3.10\r\nSet-Cookie: Z6wq_f5c1_saltkey=SOo43Zag; expires=Mon, 13-Jul-2015 03:14:32 GMT; path=/; domain=.abc.com; httponly\r\nContent-Type: text/html; charset=gbk\r\nSet-Cookie: Z6wq_f5c1_lastvisit=1434161672; expires=Mon, 13-Jul-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_sid=fkJJhh; expires=Sun, 14-Jun-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nExpires: -1\r\nCache-Control: no-store, private, post-check=0, pre-check=0, max-age=0\r\nPragma: no-cache\r\nSet-Cookie: Z6wq_f5c1_lastact=1434165272%09forum.php%09viewthread; expires=Sun, 14-Jun-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_stats_qc_reg=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_cloudstatpost=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.abc.com\r\nlocation:http://bbs.abc.com/abc/warning.html\r\n\r\n\0\0\0\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0ZaZ", 1016) = 1016 <0.000083>
0.000234 shutdown(4, 1 /* send */) = 0 <0.000023>
三，对比可以看出，最重要的出错信息如下：
POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1
Host: 1.2.3.4:8080
500 Internal Server Error
<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">
<soapenv:Body>
<soapenv:Fault>
<faultcode>soapenv:Server
</faultcode>
<faultstring>Exception occurred while trying to invoke service method search
</faultstring>
<detail />
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>
四，寻着此线索继续追踪
Host: 1.2.3.4:8080在NGINX里指向2.3.4.5:8080。
其中的TOMCAT日志和NGINX对比日志如下（已重启），从中可以发现很强的相关性。
NGINX：
1.190.229.13 - - [13/Jun/2015:14:54:30 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://www.abc.com/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 1.0" "1.190.229.13" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:30 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:30 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
221.204.196.144 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://bbs.abc.com/thread-81726-1-210.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "221.204.196.144" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.001" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.001" "-" "-"
116.21.109.46 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://www.abc.com/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" "116.21.109.46" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.003" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.001" "-" "-"
124.130.149.182 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://bbs.abc.com/thread-489495-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36 2345chrome v3.0.0.6869" "124.130.149.182" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.003" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
79.171.81.130 - - [13/Jun/2015:14:54:34 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" "79.171.81.130" "0.002" "-" "-"
101.23.157.6 - - [13/Jun/2015:14:54:50 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"
TOMCAT：
115.29.174.159 - - [13/Jun/2015:14:55:00 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:00 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9752
115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9590
115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 10446
115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9754
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342
10.132.44.218 - - [13/Jun/2015:14:55:03 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330
五，解决思路：
希望我们能配合小峤，一起解决2.3.4.5服务器上TOMCAT每隔五分钟左右报的HTTP 500内部错误。
用STRACE解决公司真实故障一例的更多相关文章
- 如何解决dns解析故障
在实际应用过程中可能会遇到DNS解析错误的问题,就是说当我们访问一个域名时无法完成将其解析到IP地址的工作,而直接输入网站IP却可以正常访问,这就是因为DNS解析出现故障造成的.这个现象发生的机率比较 ...
- ORA-01092 ORA-12432: LBAC error: zllegnp:OCIStmtExecute 故障一例
最近由于数据库hang住,无奈之下直接干掉了pmon进程,再次启动的时候收到了ORA-01092: ORACLE instance terminated. Disconnection forced以及 ...
- Android 打开方式选定后默认了改不回来?解决方法(三星s7为例)
Android 打开方式选定后默认了改不回来?解决方法(三星s7为例) 刚刚在测试东西,打开一个gif图,然后我故意选择用支付宝打开,然后...支付宝当然不支持,我觉得第二次打开它应该还会问我,没想到 ...
- Openvpn完美解决公司网络没有固定公网IP的问题
方案背景: 公司办公网络使用长城宽带上网有一段时间了,有4个固定IP(2个电信,2个网通),链路不太稳定,经常有问题,因此考虑取消长城宽带,采用原来的adsl上网.但是有个问题,因为公司内网有几台服务 ...
- 解决Kubernetes Pod故障的5个简单技巧
在很多情况下,你可能会发现Kubernetes中的应用程序没有正确地部署,或者没有正常地工作.今天这篇文章就提供了如何去快速解决这类故障以及一些技巧. 在阅读了这篇文章之后,你还将深入了解Kubern ...
- TestNg失败重跑—解决使用 dataProvider 参数化用例次数冲突问题
问题背景 在使用 testng 执行 UI 自动化用例时,由于 UI自动化的不稳定性,我们在测试的时候,往往会加上失败重跑机制.在不使用 @DataProvider 提供用例参数化时,是不会有什么问题 ...
- Pandas之:Pandas高级教程以铁达尼号真实数据为例
Pandas之:Pandas高级教程以铁达尼号真实数据为例 目录 简介 读写文件 DF的选择 选择列数据 选择行数据 同时选择行和列 使用plots作图 使用现有的列创建新的列 进行统计 DF重组 简 ...
- MYSQL主从同步故障一例及解决过程
公司里有两个mysql服务器做主从同步,某天Nagios发来报警短信,mysqla is down...赶紧联系机房,机房的人反馈来的信息是 HARDWARE ERROR 后面信息省略,让机房记下错误 ...
- strace排除Linux服务器故障
strace是一个有用的小工具 – 大多数Linux系统默认已经安装 – 可以通过跟踪系统调用来让你知道一个程序在后台所做的事情.Strace是一个基础的调试工具;但是即便你不是在跟踪一个问题的时候它 ...
随机推荐
- 小猪的Android入门之路 Day 3 - part 3
小猪的Android入门之路 Day 3 - part 3 各种UI组件的学习 Part 3 本节引言: 在前面两个部分中我们对Android中一些比較经常使用的基本组件进行了一个了解, part 1 ...
- android 38 Abdroid客户端和服务端交互
服务端: package com.sxt.day05; import java.io.IOException; import java.util.ArrayList; import javax.ser ...
- CFS： 虚拟运行时间
http://edsionte.com/techblog/archives/4331 nice和prio的关系如下: #define NICE_TO_PRIO(nice) (MAX_RT_PRIO+n ...
- svn is alread locked
右键文件夹team →Cleanup就可以了 is not under version control team→ clean 等等更新
- 汉诺塔-Hanoi
1. 问题来源: 汉诺塔(河内塔)问题是印度的一个古老的传说. 法国数学家爱德华·卢卡斯曾编写过一个印度的古老传说:在世界中心贝拿勒斯(在印度北部)的圣庙里,一块黄铜板上插着三根宝石针.印度教的主神梵 ...
- PHP编译错误Don't know how to define struct flock on this system, set --enable-opcache=no
编辑 /etc/ld.so.conf 加入 /usr/local/lib 再执行 ldconfig
- JavaSE、JavaEE、JavaME三者的区别
1. Java SE(Java Platform,Standard Edition). Java SE 以前称为 J2SE. 它允许开发和部署在桌面.服务器.嵌入式环境和实时环境中使用的 Java 应 ...
- SQL server 如何附加、还原、分离、备份数据库文件
No1 : 附加 No2 : 还原 一.(需要 .bak文件)首先建立一个数据库,数据库名称与你的.bak文件名要相同. 当然.这时候的这个数据库还是空的,需要还原回去数据.右键 ...
- 进程识别号（PID）的理解
PID(Process Identification)操作系统里指进程识别号,也就是进程标识符.操作系统里每打开一个程序都会创建一个进程ID,即PID. PID(进程控制符)英文全称为Process ...
- PHPCMS二层栏目调用
{pc:content action="category" catid="0" num="25" siteid="$siteid& ...