on namespace ceilometer.$cmd failed: Authentication failed. 问题处理方案
on namespace ceilometer.$cmd failed: Authentication failed.
UserNotFound: Could not find user ceilometer@ceilometer
背景介绍
1、Ceilometer 项目是 OpenStack 中用来做计量计费功能的一个组件，后来又逐步发展增加了部分监控采集、告警的功能。
2、MongoDB 是一个基于分布式文件存储的数据库。由 C++ 语言编写。旨在为 WEB 应用提供可扩展的高性能数据存储解决方案。
3、前几年的一个项目就使用到了 Ceilometer 和 MongoDB（3.2.9版本） 结合，用于存储性能和告警数据。
问题说明
最近，在某个现场环境上，MongoDB 挂载的存储设备出现了故障。但是，存储设备故障恢复后，MongoDB服务无法正常启动。
启动日志报错如下：
--31T16::27.651+ I CONTROL [main] ***** SERVER RESTARTED *****
--31T16::27.658+ I CONTROL [initandlisten] MongoDB starting : pid= port= dbpath=/var/lib/mongodb -bit host=ubuntu
--31T16::27.658+ I CONTROL [initandlisten] db version v3.2.9
--31T16::27.658+ I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
--31T16::27.658+ I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f Jan
--31T16::27.658+ I CONTROL [initandlisten] allocator: tcmalloc
--31T16::27.658+ I CONTROL [initandlisten] modules: none
--31T16::27.658+ I CONTROL [initandlisten] build environment:
--31T16::27.658+ I CONTROL [initandlisten] distmod: ubuntu1404
--31T16::27.658+ I CONTROL [initandlisten] distarch: x86_64
--31T16::27.658+ I CONTROL [initandlisten] target_arch: x86_64
--31T16::27.658+ I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { directoryForIndexes: true, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
--31T16::27.676+ I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=13G,session_max=,eviction=(threads_max=),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=),checkpoint=(wait=,log_size=2GB),statistics_log=(wait=),
--31T16::29.185+ E STORAGE [initandlisten] WiredTiger (-) [:][:0x7f1a16a0bcc0], txn-recover: Recovery failed: WT_NOTFOUND: item not found
--31T16::29.195+ I - [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
--31T16::29.195+ I STORAGE [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
--31T16::29.195+ I CONTROL [initandlisten] dbexit: rc: 100
截图如下：
报错的大致意思就是 MongoDB 挂载的存储设备出现了异常，item not found。MongoDB服务终止，并退出。
查询不到mongo的进程，截图如下：
由于 Ceilometer 依赖MongoDB服务，导致 Ceilometer的服务启动也是异常的。
最终结果：整个性能采集功能基本瘫痪了。。。
问题处理
由于是现场环境，并且 MongoDB 中已经存了近 300 GB 的性能数据，现场同事还是希望能尽量恢复下 MongoDB的服务。
我们的环境是使用Vmware 开的虚拟机，并且现场同事会定期对现场机器做“快照”（Snapshot）备份。
Plan A：让MongoDB所在的机器恢复到 存储设备出现故障前几天。
Plan B：在 Plan A 失败的情况下，重装 MongoDB服务。
【Plan A】：抢救 MongoDB
1、当MongoDB所在机器，恢复到故障前几天的虚拟机快照。
MongoDB 服务启动依然异常。错误日志如下：
--01T10::39.171+ I CONTROL [main] ***** SERVER RESTARTED *****
--01T10::39.178+ I CONTROL [initandlisten] MongoDB starting : pid= port= dbpath=/var/lib/mongodb -bit host=ubuntu
--01T10::39.178+ I CONTROL [initandlisten] db version v3.2.9
--01T10::39.178+ I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
--01T10::39.178+ I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f Jan
--01T10::39.178+ I CONTROL [initandlisten] allocator: tcmalloc
--01T10::39.178+ I CONTROL [initandlisten] modules: none
--01T10::39.178+ I CONTROL [initandlisten] build environment:
--01T10::39.178+ I CONTROL [initandlisten] distmod: ubuntu1404
--01T10::39.178+ I CONTROL [initandlisten] distarch: x86_64
--01T10::39.178+ I CONTROL [initandlisten] target_arch: x86_64
--01T10::39.178+ I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { directoryForIndexes: true, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
--01T10::39.196+ W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
--01T10::39.196+ W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
--01T10::39.196+ I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=13G,session_max=,eviction=(threads_max=),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=),checkpoint=(wait=,log_size=2GB),statistics_log=(wait=),
--01T10::39.206+ E STORAGE [initandlisten] WiredTiger () [:][:0x7fbc63db5cc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
--01T10::39.206+ E STORAGE [initandlisten] WiredTiger (-31804) [1572573939:206917][1992:0x7fbc63db5cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
--01T10::39.206+ I - [initandlisten] Fatal Assertion 28558
--01T10::39.206+ I - [initandlisten] ***aborting after fassert() failure --01T10::39.224+ F - [initandlisten] Got signal: (Aborted). 0x13225f2 0x1321749 0x1321f52 0x7fbc62a32340 0x7fbc62692f79 0x7fbc62696388 0x12a8662 0x10a28b3 0x1a88fcc 0x1a8944d 0x1a89834 0x1a509bf 0x1a4f42e 0x1a0b5e1 0x1a87f17 0x1a88459 0x1a8857b 0x1a1a208 0x1a850b5 0x1a4ed4f 0x1a4ee4e 0x1a07e63 0x108a81f 0x1086ad3 0xfafb38 0x9b61ed 0x9b87b0 0x96f4ad 0x7fbc6267dec5 0x9b2af7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"","o":"F225F2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"","o":"F21749"},{"b":"","o":"F21F52"},{"b":"7FBC62A22000","o":""},{"b":"7FBC6265C000","o":"36F79","s":"gsignal"},{"b":"7FBC6265C000","o":"3A388","s":"abort"},{"b":"","o":"EA8662","s":"_ZN5mongo13fassertFailedEi"},{"b":"","o":"CA28B3"},{"b":"","o":"1688FCC","s":"__wt_eventv"},{"b":"","o":"168944D","s":"__wt_err"},{"b":"","o":"","s":"__wt_panic"},{"b":"","o":"16509BF","s":"__wt_turtle_read"},{"b":"","o":"164F42E","s":"__wt_metadata_search"},{"b":"","o":"160B5E1","s":"__wt_conn_btree_open"},{"b":"","o":"1687F17","s":"__wt_session_get_btree"},{"b":"","o":"","s":"__wt_session_get_btree"},{"b":"","o":"168857B","s":"__wt_session_get_btree_ckpt"},{"b":"","o":"161A208","s":"__wt_curfile_open"},{"b":"","o":"16850B5"},{"b":"","o":"164ED4F","s":"__wt_metadata_cursor_open"},{"b":"","o":"164EE4E","s":"__wt_metadata_cursor"},{"b":"","o":"1607E63","s":"wiredtiger_open"},{"b":"","o":"C8A81F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"","o":"C86AD3"},{"b":"","o":"BAFB38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"","o":"5B61ED"},{"b":"","o":"5B87B0","s":"_ZN5mongo13initAndListenEi"},{"b":"","o":"56F4AD","s":"main"},{"b":"7FBC6265C000","o":"21EC5","s":"__libc_start_main"},{"b":"","o":"5B2AF7"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-24-generic", "version" : "#46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : , "b" : "", "buildId" : "78E57AF736DDF3E8C558F60DB63F68BCF686D70A" }, { "b" : "7FFFC96FE000", "elfType" : , "buildId" : "6755FAD2CADACDF1667E5B57FF1EDFC28DD1C976" }, { "b" : "7FBC63942000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : , "buildId" : "70F9A7F3734C01FF8D442C21A03B631588C2FECD" }, { "b" : "7FBC63568000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : , "buildId" : "8D6FFA819931E68666BCEF4424BA6289838309D7" }, { "b" : "7FBC63360000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : , "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FBC6315C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : , "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FBC62E56000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : , "buildId" : "574C6350381DA194C00FF555E0C1784618C05569" }, { "b" : "7FBC62C40000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : , "buildId" : "CC0D578C2E0D86237CA7B0CE8913261C506A629A" }, { "b" : "7FBC62A22000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : , "buildId" : "FE662C4D7B14EE804E0C1902FB55218A106BC5CB" }, { "b" : "7FBC6265C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : , "buildId" : "B571F83A8A6F5BB22D3558CDDDA9F943A2A67FD1" }, { "b" : "7FBC63BA0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : , "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13225f2]
mongod(+0xF21749) [0x1321749]
mongod(+0xF21F52) [0x1321f52]
libpthread.so.(+0x10340) [0x7fbc62a32340]
libc.so.(gsignal+0x39) [0x7fbc62692f79]
libc.so.(abort+0x148) [0x7fbc62696388]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a8662]
mongod(+0xCA28B3) [0x10a28b3]
mongod(__wt_eventv+0x42C) [0x1a88fcc]
mongod(__wt_err+0x8D) [0x1a8944d]
mongod(__wt_panic+0x24) [0x1a89834]
mongod(__wt_turtle_read+0x2AF) [0x1a509bf]
mongod(__wt_metadata_search+0xBE) [0x1a4f42e]
mongod(__wt_conn_btree_open+0x61) [0x1a0b5e1]
mongod(__wt_session_get_btree+0xE7) [0x1a87f17]
mongod(__wt_session_get_btree+0x629) [0x1a88459]
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a8857b]
mongod(__wt_curfile_open+0x218) [0x1a1a208]
mongod(+0x16850B5) [0x1a850b5]
mongod(__wt_metadata_cursor_open+0x5F) [0x1a4ed4f]
mongod(__wt_metadata_cursor+0x7E) [0x1a4ee4e]
mongod(wiredtiger_open+0x1433) [0x1a07e63]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x108a81f]
mongod(+0xC86AD3) [0x1086ad3]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfafb38]
mongod(+0x5B61ED) [0x9b61ed]
mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b87b0]
mongod(main+0x15D) [0x96f4ad]
libc.so.(__libc_start_main+0xF5) [0x7fbc6267dec5]
mongod(+0x5B2AF7) [0x9b2af7]
----- END BACKTRACE -----
2、步骤1中出现的问题，网上说是需要删除 mongod.lock文件，重启MongoDB。
但是，删除了 mongod.lock文件后，MongoDB服务启动依然报错。报错日志如下：
--01T10::28.872+ I CONTROL [main] ***** SERVER RESTARTED *****
--01T10::28.879+ I CONTROL [initandlisten] MongoDB starting : pid= port= dbpath=/var/lib/mongodb -bit host=ubuntu
--01T10::28.879+ I CONTROL [initandlisten] db version v3.2.9
--01T10::28.879+ I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
--01T10::28.879+ I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f Jan
--01T10::28.879+ I CONTROL [initandlisten] allocator: tcmalloc
--01T10::28.879+ I CONTROL [initandlisten] modules: none
--01T10::28.879+ I CONTROL [initandlisten] build environment:
--01T10::28.879+ I CONTROL [initandlisten] distmod: ubuntu1404
--01T10::28.879+ I CONTROL [initandlisten] distarch: x86_64
--01T10::28.879+ I CONTROL [initandlisten] target_arch: x86_64
--01T10::28.879+ I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { directoryForIndexes: true, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
--01T10::28.897+ I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=13G,session_max=,eviction=(threads_max=),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=),checkpoint=(wait=,log_size=2GB),statistics_log=(wait=),
--01T10::28.908+ E STORAGE [initandlisten] WiredTiger () [:][:0x7ffdfc075cc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
--01T10::28.908+ E STORAGE [initandlisten] WiredTiger (-31804) [1572574168:908264][2032:0x7ffdfc075cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-11-01T10:09:28.908+0800 I - [initandlisten] Fatal Assertion 28558
--01T10::28.908+ I - [initandlisten] ***aborting after fassert() failure --01T10::28.925+ F - [initandlisten] Got signal: (Aborted). 0x13225f2 0x1321749 0x1321f52 0x7ffdfacf2340 0x7ffdfa952f79 0x7ffdfa956388 0x12a8662 0x10a28b3 0x1a88fcc 0x1a8944d 0x1a89834 0x1a509bf 0x1a4f42e 0x1a0b5e1 0x1a87f17 0x1a88459 0x1a8857b 0x1a1a208 0x1a850b5 0x1a4ed4f 0x1a4ee4e 0x1a07e63 0x108a81f 0x1086ad3 0xfafb38 0x9b61ed 0x9b87b0 0x96f4ad 0x7ffdfa93dec5 0x9b2af7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"","o":"F225F2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"","o":"F21749"},{"b":"","o":"F21F52"},{"b":"7FFDFACE2000","o":""},{"b":"7FFDFA91C000","o":"36F79","s":"gsignal"},{"b":"7FFDFA91C000","o":"3A388","s":"abort"},{"b":"","o":"EA8662","s":"_ZN5mongo13fassertFailedEi"},{"b":"","o":"CA28B3"},{"b":"","o":"1688FCC","s":"__wt_eventv"},{"b":"","o":"168944D","s":"__wt_err"},{"b":"","o":"","s":"__wt_panic"},{"b":"","o":"16509BF","s":"__wt_turtle_read"},{"b":"","o":"164F42E","s":"__wt_metadata_search"},{"b":"","o":"160B5E1","s":"__wt_conn_btree_open"},{"b":"","o":"1687F17","s":"__wt_session_get_btree"},{"b":"","o":"","s":"__wt_session_get_btree"},{"b":"","o":"168857B","s":"__wt_session_get_btree_ckpt"},{"b":"","o":"161A208","s":"__wt_curfile_open"},{"b":"","o":"16850B5"},{"b":"","o":"164ED4F","s":"__wt_metadata_cursor_open"},{"b":"","o":"164EE4E","s":"__wt_metadata_cursor"},{"b":"","o":"1607E63","s":"wiredtiger_open"},{"b":"","o":"C8A81F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"","o":"C86AD3"},{"b":"","o":"BAFB38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"","o":"5B61ED"},{"b":"","o":"5B87B0","s":"_ZN5mongo13initAndListenEi"},{"b":"","o":"56F4AD","s":"main"},{"b":"7FFDFA91C000","o":"21EC5","s":"__libc_start_main"},{"b":"","o":"5B2AF7"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-24-generic", "version" : "#46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : , "b" : "", "buildId" : "78E57AF736DDF3E8C558F60DB63F68BCF686D70A" }, { "b" : "7FFFBA0FE000", "elfType" : , "buildId" : "6755FAD2CADACDF1667E5B57FF1EDFC28DD1C976" }, { "b" : "7FFDFBC02000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : , "buildId" : "70F9A7F3734C01FF8D442C21A03B631588C2FECD" }, { "b" : "7FFDFB828000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : , "buildId" : "8D6FFA819931E68666BCEF4424BA6289838309D7" }, { "b" : "7FFDFB620000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : , "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FFDFB41C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : , "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FFDFB116000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : , "buildId" : "574C6350381DA194C00FF555E0C1784618C05569" }, { "b" : "7FFDFAF00000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : , "buildId" : "CC0D578C2E0D86237CA7B0CE8913261C506A629A" }, { "b" : "7FFDFACE2000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : , "buildId" : "FE662C4D7B14EE804E0C1902FB55218A106BC5CB" }, { "b" : "7FFDFA91C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : , "buildId" : "B571F83A8A6F5BB22D3558CDDDA9F943A2A67FD1" }, { "b" : "7FFDFBE60000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : , "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13225f2]
mongod(+0xF21749) [0x1321749]
mongod(+0xF21F52) [0x1321f52]
libpthread.so.(+0x10340) [0x7ffdfacf2340]
libc.so.(gsignal+0x39) [0x7ffdfa952f79]
libc.so.(abort+0x148) [0x7ffdfa956388]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a8662]
mongod(+0xCA28B3) [0x10a28b3]
mongod(__wt_eventv+0x42C) [0x1a88fcc]
mongod(__wt_err+0x8D) [0x1a8944d]
mongod(__wt_panic+0x24) [0x1a89834]
mongod(__wt_turtle_read+0x2AF) [0x1a509bf]
mongod(__wt_metadata_search+0xBE) [0x1a4f42e]
mongod(__wt_conn_btree_open+0x61) [0x1a0b5e1]
mongod(__wt_session_get_btree+0xE7) [0x1a87f17]
mongod(__wt_session_get_btree+0x629) [0x1a88459]
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a8857b]
mongod(__wt_curfile_open+0x218) [0x1a1a208]
mongod(+0x16850B5) [0x1a850b5]
mongod(__wt_metadata_cursor_open+0x5F) [0x1a4ed4f]
mongod(__wt_metadata_cursor+0x7E) [0x1a4ee4e]
mongod(wiredtiger_open+0x1433) [0x1a07e63]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x108a81f]
mongod(+0xC86AD3) [0x1086ad3]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfafb38]
mongod(+0x5B61ED) [0x9b61ed]
mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b87b0]
mongod(main+0x15D) [0x96f4ad]
libc.so.(__libc_start_main+0xF5) [0x7ffdfa93dec5]
mongod(+0x5B2AF7) [0x9b2af7]
----- END BACKTRACE -----
3、经过步骤1，2，依旧不可以，网上说是执行 mongod --repair 命令 可以解决。
但是，执行后，MongoDB服务启动依然报错。报错日志如下：
--01T10::56.685+ I CONTROL [main] ***** SERVER RESTARTED *****
--01T10::56.691+ I CONTROL [initandlisten] MongoDB starting : pid= port= dbpath=/var/lib/mongodb -bit host=ubuntu
--01T10::56.691+ I CONTROL [initandlisten] db version v3.2.9
--01T10::56.691+ I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
--01T10::56.691+ I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f Jan
--01T10::56.691+ I CONTROL [initandlisten] allocator: tcmalloc
--01T10::56.691+ I CONTROL [initandlisten] modules: none
--01T10::56.691+ I CONTROL [initandlisten] build environment:
--01T10::56.691+ I CONTROL [initandlisten] distmod: ubuntu1404
--01T10::56.692+ I CONTROL [initandlisten] distarch: x86_64
--01T10::56.692+ I CONTROL [initandlisten] target_arch: x86_64
--01T10::56.692+ I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { directoryForIndexes: true, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
--01T10::56.711+ E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted
--01T10::56.711+ I - [initandlisten] Fatal Assertion 28578
--01T10::56.711+ I - [initandlisten] ***aborting after fassert() failure
4、经过前三步的测试验证，证明抢救无效。MongoDB官网上说是，针对此类挂载存储设备故障引发的服务不能正常重启的问题，在做改善。
但是，目前个人还没有找到完美的抢救方案。断断续续地折腾了两三天，最后无奈宣布 “MongoDB 抢救无效！！！”。
为了尽快回复现场大环境的正常功能使用，再者虽然是300GB的性能数据，但也是历史性能数据，不具有太多的使用价值，不再做无用功。
决定启动 Plan B。（此情此景，有感而发：往者不可谏，来者犹可追！不念过去，珍惜当下，展望未来。。。）
【Plan B】： 重装MongoDB
1、准备一台虚拟机，和现有虚拟机操作系统和 IP 一致，保证 MongoDB 服务的兼容性和其他服务的依赖性，免得修改其他服务的MongoDB的连接信息。
2、往往事情不会一蹴而就的解决，总会磕磕绊绊。这不，按照步骤1重装后，MongoDB 和 Ceilometer-api 服务启动均出现异常。报错日志分别如下：
1）MongoDB服务启动日志如下（UserNotFound: Could not find user ceilometer@ceilometer）：
--04T14::46.130+ I CONTROL [signalProcessingThread] dbexit: rc:
--04T14::53.758+ I CONTROL [main] ***** SERVER RESTARTED *****
--04T14::53.765+ I CONTROL [initandlisten] MongoDB starting : pid= port= dbpath=/var/lib/mongodb -bit host=ubuntu
--04T14::53.765+ I CONTROL [initandlisten] db version v3.2.9
--04T14::53.765+ I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
--04T14::53.765+ I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f Jan
--04T14::53.765+ I CONTROL [initandlisten] allocator: tcmalloc
--04T14::53.765+ I CONTROL [initandlisten] modules: none
--04T14::53.765+ I CONTROL [initandlisten] build environment:
--04T14::53.765+ I CONTROL [initandlisten] distmod: ubuntu1404
--04T14::53.765+ I CONTROL [initandlisten] distarch: x86_64
--04T14::53.765+ I CONTROL [initandlisten] target_arch: x86_64
--04T14::53.765+ I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { directoryForIndexes: true, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
--04T14::53.783+ I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=13G,session_max=,eviction=(threads_max=),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=),checkpoint=(wait=,log_size=2GB),statistics_log=(wait=),
--04T14::58.241+ I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
--04T14::58.241+ I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
--04T14::58.241+ I NETWORK [initandlisten] waiting for connections on port
--04T14::59.019+ I NETWORK [initandlisten] connection accepted from 10.117.26.104: # ( connection now open)
--04T14::59.272+ I NETWORK [initandlisten] connection accepted from 127.0.0.1: # ( connections now open)
--04T14::59.417+ I NETWORK [initandlisten] connection accepted from 127.0.0.1: # ( connections now open)
--04T14::59.418+ I ACCESS [conn3] SCRAM-SHA- authentication failed for ceilometer on ceilometer from client 127.0.0.1 ; UserNotFound: Could not find user ceilometer@ceilometer
--04T14::59.802+ I NETWORK [initandlisten] connection accepted from 127.0.0.1: # ( connections now open)
--04T14::09.420+ I ACCESS [conn3] SCRAM-SHA- authentication failed for ceilometer on ceilometer from client 127.0.0.1 ; UserNotFound: Could not find user ceilometer@ceilometer
--04T14::19.421+ I ACCESS [conn3] SCRAM-SHA- authentication failed for ceilometer on ceilometer from client 127.0.0.1 ; UserNotFound: Could not find user ceilometer@ceilometer
--04T14::29.423+ I ACCESS [conn3] SCRAM-SHA- authentication failed for ceilometer on ceilometer from client 127.0.0.1 ; UserNotFound: Could not find user ceilometer@ceilometer
2）Ceilometer 服务启动日志如下（on namespace ceilometer.$cmd failed: Authentication failed）：
-- ::59.875 TRACE ceilometer Traceback (most recent call last):
-- ::59.875 TRACE ceilometer File "/usr/bin/ceilometer-api", line , in <module>
-- ::59.875 TRACE ceilometer sys.exit(main())
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in build_server
-- ::59.875 TRACE ceilometer app = load_app()
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in load_app
-- ::59.875 TRACE ceilometer return deploy.loadapp("config:" + cfg_file)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in loadapp
-- ::59.875 TRACE ceilometer return loadobj(APP, uri, name=name, **kw)
-- ::59.875 TRACE ceilometer app = load_app()
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in load_app
-- ::59.875 TRACE ceilometer return deploy.loadapp("config:" + cfg_file)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in loadapp
-- ::59.875 TRACE ceilometer return loadobj(APP, uri, name=name, **kw)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in loadobj
-- ::59.875 TRACE ceilometer return context.create()
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in create
-- ::59.875 TRACE ceilometer return self.object_type.invoke(self)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in invoke
-- ::59.875 TRACE ceilometer app = context.app_context.create()
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in create
-- ::59.875 TRACE ceilometer return self.object_type.invoke(self)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line , in invoke
-- ::59.875 TRACE ceilometer return fix_call(context.object, context.global_conf, **context.local_conf)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line , in fix_call
-- ::59.875 TRACE ceilometer val = callable(*args, **kw)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in app_factory
-- ::59.875 TRACE ceilometer return VersionSelectorApplication()
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in __init__
-- ::59.875 TRACE ceilometer self.v2 = setup_app(pecan_config=pc)
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line , in setup_app
-- ::59.875 TRACE ceilometer hooks.DBHook(),
-- ::59.875 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/hooks.py", line , in __init__
-- ::59.875 TRACE ceilometer ', '.join(['metering', 'event', 'alarm']))
-- ::59.875 TRACE ceilometer Exception: Api failed to start. Failed to connect to databases, purpose: metering, event, alarm
-- ::59.875 TRACE ceilometer
-- ::00.427 INFO ceilometer.api.app [-] Full WSGI config used: /etc/ceilometer/api_paste.ini
-- ::00.467 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', )]
-- ::30.497 ERROR ceilometer.api.hooks [-] Failed to connect to db, purpose metering retry later: command SON([('saslStart', ), ('mechanism', 'SCRAM-SHA-1'), ('payload', Binary('n,,n=ceilometer,r=ODUxNzE5MDAwMjM3', )), ('autoAuthorize', )]) on namespace ceilometer.$cmd failed: Authentication failed.
3、mongodb服务重装后，ceilometer-api服务连接mongodb的时候，日志报认证失败，导致8777端口一直用不了。
1）【问题原因】是因为mongodb 缺少ceilometer账号。
2）【解决方案】
在ceilometer库中新建用户ceilometer，如下三条命令:
mongo
use ceilometer;
db.createUser( { user: "ceilometer", pwd: "password",roles: [ "readWrite", "dbAdmin" ] } );
具体命令操作，截图如下：
【注意】
如上图所示，3.2.9版本的MongoDB，不再支持 db.addUser() 方法创建用户，需要使用 db.createUser() 方法进行创建用户。
总结
以上，是个人根据自己最近几天现场问题的处理情况。
最终使 MongoDB 和 Ceilometer 服务可以正常使用。
处理过程中，有所感悟，做下心得记录，希望能帮到有需要的同学。
on namespace ceilometer.$cmd failed: Authentication failed. 问题处理方案的更多相关文章
- 解决Idea GitLab Clone failed: Authentication failed for的问题
刚使用GitLab做项目管理,在idea-check versionControl中使用git clone工程,一直报Clone failed: Authentication failed for ‘ ...
- IDEA报错： Clone failed: Authentication failed for 'http://10.70.XXXXXXXXXXXXXXXXX'
今天从git上导入公司的项目,总是报错Clone failed: Authentication failed for 'http://10.70.XXXXXXXXXXXXXX' 在网上百度了一下,大致 ...
- AS 新电脑clone项目报错：Clone failed: Authentication failed for 'https://gitee.com/XXX/Demo.git/'
在新的电脑上安装Android Studio,并且使用git clone 项目,报以下错误: Clone failed: Authentication failed for 'https://gite ...
- MongoDB3.4安装配置以及与Robomongo1.1的连接——解决Authentication Failed导致的不能连接问题
本文环境:win10(64)+MongoDB(3.4.5)+Robomongo(1.1) 目录: MongoDB的安装 MongoDB的配置 Robomongo的安装以及与MongoDB的连接 一些新 ...
- git [command line] fatal: Authentication failed for
fatal: Authentication failed for https://www.jianshu.com/p/8a7f257e07b8 git.exe fetch -v --progress ...
- MongoDB authentication failed
0.随笔摘要: MongoDB 安装配置 MongoDB 权限控制 MongoDB 注意事项 authentication failed 1.MongoDB 下载安装配置 MongoDB官网 ...
- mongoexport遭遇Authentication failed
今天使用mongoexport工具导出数据,遇到权限不足报错:[host]$mongoexport -h 10.31.11.190:23820 -udbmgr -pMgr2mgdb -d rcmp ...
- 码云 fatal: Authentication failed for
最近push代码到码云时,push失败,提示fatal: Authentication failed for,解决方法就是: 在git命令行中输入 git config --system --unse ...
- fatal: Authentication failed for 'http://git
git pull 出现 fatal: Authentication failed for 'http://git... git config --system --unset credential.h ...
随机推荐
- python+Appium自动化：Capability配置简介
Capability配置简介 desired capability的功能是配置Appium会话. Desired Capabilities是一组设置的键值对的集合,其中键对应设置的名称,而值对应设置的 ...
- 第二章 Vue快速入门--14 使用v-model实现计算器的案例
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8&quo ...
- Java发送POST请求，参数为JSON格式，并接收返回JSON数据
原文地址:https://blog.csdn.net/qq_26975307/article/details/82713725 /** * 发送post请求 * @param url 路径 * @pa ...
- Java 转发和重定向的区别
转发是服务器行为,重定向是客户端行为 1.转发在服务器端完成的;重定向是在客户端完成的 2.转发的速度快;重定向速度慢 3.转发的是同一次请求;重定向是两次不同请求 4.转发不会执行转发后的代码;重定 ...
- hive的事物性 transaction manager
create table lk3 (id string,nname string,grade int,goldUser int); insert into lk3 values (,, ), (,, ...
- Java-收邮件
import java.util.Properties; import javax.mail.Folder; import javax.mail.Message; import javax.mail. ...
- 04 vue-cli 脚手架、webpack-simple模板项目生成、组件使用
alice https://www.cnblogs.com/alice-bj/p/9317504.html https://www.cnblogs.com/alice-bj/p/9318069.htm ...
- HDU 6035 - Colorful Tree | 2017 Multi-University Training Contest 1
/* HDU 6035 - Colorful Tree [ DFS,分块 ] 题意: n个节点的树,每个节点有一种颜色(1~n),一条路径的权值是这条路上不同的颜色的数量,问所有路径(n*(n-1)/ ...
- Servlet中的请求转发RequestDispatcher接口的forword与Include的区别
RequestDispatcher接口中具有两个方法: forward() 与 include() 均 可完成请求 的转发.区别如下: forword(): 使用该方法,则当前 的 Servlet 中 ...
- 实用——pojo （实体类） 常用函数
1,转义方法 @JsonProperty("n")