环境准备

基础环境

node00 192.168.247.144 node00
node01 192.168.247.135 node01
node02 192.168.247.143 node02

vmare在分配IP没有连续，没有关系继续吧

配置免密登录

  1. 修改主机名称
hostnamectl set-hostname node00

hostnamectl set-hostname node01

hostnamectl set-hostname node02
  1. 编辑hosts文件
[root@linux30 ~]# vi /etc/hosts

192.168.247.144	node00

192.168.247.135	node01

192.168.247.143	node02
  1. 官方建议不用系统内置用户， 创建名为ceph_user用户， 密码也设为123456：
useradd -d /home/ceph_user -m ceph_user

passwd ceph_user

# 设置root 权限

echo "ceph_user ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph_user

sudo chmod 0440 /etc/sudoers.d/ceph_user
  1. 生成密钥：切换用户： su ceph_user 执行ssh-keygen，一直按默认提示点击生成RSA密钥信息。
  2. 分发密钥至各机器节点
ssh-copy-id ceph_user@node00

ssh-copy-id ceph_user@node01

ssh-copy-id ceph_user@node02
  1. 修改管理节点上的 ~/.ssh/config 文件， 简化SSH远程连接时的输入信息：管理节点是会有root和ceph_user多个用户， ssh远程连接默认会以当前用户身份进行登陆， 如果我们是root身份进行远程连接， 还是需要输入密码， 我们想简化， 该怎么处理？
su root

vim /root/.ssh/config

复制一下内容即可

Host node00

   Hostname node00

   User ceph_user

Host node01

   Hostname node01

   User ceph_user

Host node02

   Hostname node02

   User ceph_user

注意修改文件权限， 不能采用777最大权限：

chmod 600 ~/.ssh/config

NTP时间工具同步

# 下载

yum install ntp ntpdate ntp-doc -y

# 确保时区是正确， 设置开机启动：

systemctl enable ntpd

# 将时间每隔1小时自动校准同步。编辑 vi /etc/rc.d/rc.local 追加：

echo "/usr/sbin/ntpdate ntp1.aliyun.com > /dev/null 2>&1; /sbin/hwclock -w" >> /etc/rc.d/rc.local

# 配置定时任务,  执行crontab -e 加入

crontab -e 0 */1 * * * ntpdate ntp1.aliyun.com > /dev/null 2>&1; /sbin/hwclock -w

ceph的镜像加速源头设置

rpm -ivh https://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ceph-release-1-1.el7.noarch.rpm

yum install epel-release

yum install ceph-deploy python-setuptools python2-subprocess32

并配置镜像加速源

echo '

[Ceph]

name=Ceph packages for $basearch

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/x86_64/

enabled=1

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[Ceph-noarch]

name=Ceph noarch packages

# 官方源

#baseurl=http://download.ceph.com/rpm-mimic/el7/noarch

# 清华源

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch/

enabled=1

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]

name=Ceph source packages

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS/

enabled=1

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc' > /etc/yum.repos.d/ceph.repo
  1. 开放端口， 非生产环境， 可以直接禁用防火墙：
systemctl stop firewalld.service

systemctl disable firewalld.service
  1. SELinux设为禁用：
setenforce 0

永久生效：编辑 vi /etc/selinux/config修改：

SELINUX=disabled

正式安装

  1. ceph-deploy 工具安装；主节点安装：
yum update && yum -y install ceph ceph-deploy

也可通过如下方式安装：

  1. 创建目录
mkdir -p /opt/ceph/ceph-cluster && cd /opt/ceph/ceph-cluster
  1. 创建集群；
ceph-deploy new node00  node01 node02

会生成配置文件；

vi /opt/ceph/ceph-cluster/ceph.conf


[global]

fsid = 7d75db39-d457-4764-b7d2-1b48645b2781

mon_initial_members = linux30, linux31, linux32

mon_host = 192.168.10.30,192.168.10.31,192.168.10.32

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

# 公网网络

public network = 192.168.247.100/24

# 设置pool池默认分配数量 默认副本数为3

osd pool default size = 2

# 容忍更多的时钟误差

mon clock drift allowed = 2

mon clock drift warn backoff = 30

# 允许删除pool

mon_allow_pool_delete = true

[mgr]

# 开启WEB仪表盘

mgr modules = dashboard

public network是其公网IP（虚拟机是vmnet8的网卡IP）

  1. 节点部署
ceph-deploy install  node00  node01 node02 --no-adjust-repos

--no-adjust-repos使用该命令可以实现加速效果；并且不改变镜像源。

  1. 初始monitor信息：
ceph-deploy mon create-initial

## ceph-deploy --overwrite-conf mon create-initial

执行完之后，本地目录下会生成若干keyring结尾的密钥文件；如下图：

  1. 同步管理信息
ceph-deploy admin  node00  node01 node02


[root@node00 ceph-cluster]# ceph-deploy admin  node00  node01 node02

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin node00 node01 node02

[ceph_deploy.cli][INFO  ] ceph-deploy options:

[ceph_deploy.cli][INFO  ]  username                      : None

[ceph_deploy.cli][INFO  ]  verbose                       : False

[ceph_deploy.cli][INFO  ]  overwrite_conf                : False

[ceph_deploy.cli][INFO  ]  quiet                         : False

[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4b18579878>

[ceph_deploy.cli][INFO  ]  cluster                       : ceph

[ceph_deploy.cli][INFO  ]  client                        : ['node00', 'node01', 'node02']

[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f4b18e12230>

[ceph_deploy.cli][INFO  ]  ceph_conf                     : None

[ceph_deploy.cli][INFO  ]  default_release               : False

[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node00

[node00][DEBUG ] connected to host: node00

[node00][DEBUG ] detect platform information from remote host

[node00][DEBUG ] detect machine type

[node00][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01

[node01][DEBUG ] connection detected need for sudo

[node01][DEBUG ] connected to host: node01

[node01][DEBUG ] detect platform information from remote host

[node01][DEBUG ] detect machine type

[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02

[node02][DEBUG ] connection detected need for sudo

[node02][DEBUG ] connected to host: node02

[node02][DEBUG ] detect platform information from remote host

[node02][DEBUG ] detect machine type

[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[root@node00 ceph-cluster]# vim /etc/ceph/

ceph.client.admin.keyring  rbdmap                     tmphCOe3j

ceph.conf                  tmp4R_f2T

[root@node00 ceph-cluster]# vim /etc/ceph/

ceph.client.admin.keyring  rbdmap                     tmphCOe3j

ceph.conf                  tmp4R_f2T

[root@node00 ceph-cluster]# vim /etc/ceph/ceph.conf

[root@node00 ceph-cluster]# clear

[root@node00 ceph-cluster]# ceph-deploy mgr create node00  node01 node02

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mgr create node00 node01 node02

[ceph_deploy.cli][INFO  ] ceph-deploy options:

[ceph_deploy.cli][INFO  ]  username                      : None

[ceph_deploy.cli][INFO  ]  verbose                       : False

[ceph_deploy.cli][INFO  ]  mgr                           : [('node00', 'node00'), ('node01', 'node01'), ('node02', 'node02')]

[ceph_deploy.cli][INFO  ]  overwrite_conf                : False

[ceph_deploy.cli][INFO  ]  subcommand                    : create

[ceph_deploy.cli][INFO  ]  quiet                         : False

[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fcff90d9b00>

[ceph_deploy.cli][INFO  ]  cluster                       : ceph

[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7fcff99c9140>

[ceph_deploy.cli][INFO  ]  ceph_conf                     : None

[ceph_deploy.cli][INFO  ]  default_release               : False

[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts node00:node00 node01:node01 node02:node02

[node00][DEBUG ] connected to host: node00

[node00][DEBUG ] detect platform information from remote host

[node00][DEBUG ] detect machine type

[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core

[ceph_deploy.mgr][DEBUG ] remote host will use systemd

[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to node00

[node00][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node00][WARNIN] mgr keyring does not exist yet, creating one

[node00][DEBUG ] create a keyring file

[node00][DEBUG ] create path recursively if it doesn't exist

[node00][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.node00 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-node00/keyring

[node00][INFO  ] Running command: systemctl enable ceph-mgr@node00

[node00][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@node00.service to /usr/lib/systemd/system/ceph-mgr@.service.

[node00][INFO  ] Running command: systemctl start ceph-mgr@node00

[node00][INFO  ] Running command: systemctl enable ceph.target

[node01][DEBUG ] connection detected need for sudo

[node01][DEBUG ] connected to host: node01

[node01][DEBUG ] detect platform information from remote host

[node01][DEBUG ] detect machine type

[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core

[ceph_deploy.mgr][DEBUG ] remote host will use systemd

[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to node01

[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node01][WARNIN] mgr keyring does not exist yet, creating one

[node01][DEBUG ] create a keyring file

[node01][DEBUG ] create path recursively if it doesn't exist

[node01][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.node01 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-node01/keyring

[node01][INFO  ] Running command: sudo systemctl enable ceph-mgr@node01

[node01][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@node01.service to /usr/lib/systemd/system/ceph-mgr@.service.

[node01][INFO  ] Running command: sudo systemctl start ceph-mgr@node01

[node01][INFO  ] Running command: sudo systemctl enable ceph.target

[node02][DEBUG ] connection detected need for sudo

[node02][DEBUG ] connected to host: node02

[node02][DEBUG ] detect platform information from remote host

[node02][DEBUG ] detect machine type

[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core

[ceph_deploy.mgr][DEBUG ] remote host will use systemd

[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to node02

[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node02][WARNIN] mgr keyring does not exist yet, creating one

[node02][DEBUG ] create a keyring file

[node02][DEBUG ] create path recursively if it doesn't exist

[node02][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.node02 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-node02/keyring

[node02][INFO  ] Running command: sudo systemctl enable ceph-mgr@node02

[node02][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@node02.service to /usr/lib/systemd/system/ceph-mgr@.service.

[node02][INFO  ] Running command: sudo systemctl start ceph-mgr@node02

[node02][INFO  ] Running command: sudo systemctl enable ceph.target
  1. 安装mgr(管理守护进程)， 大于12.x版本需安装， 我们装的是最新版，需执行：
ceph-deploy mgr create node00  node01 node02
  1. 安装OSD(对象存储设备）并且完成挂载

fdisk -l查看加挂磁盘

加挂节点：

ceph-deploy osd create --data /dev/sdb node00

ceph-deploy osd create --data /dev/sdb node01

ceph-deploy osd create --data /dev/sdb node02
  1. 测试安装情况：
ceph-s



ceph config set mgr mgr/dashboard/server_addr 192.168.247.146

ceph config set mgr mgr/dashboard/server_port 18843

ceph config set mgr mgr/dashboard/server_addr node01

安装管理后台

  1. 开启dashboard模块
ceph mgr module enable dashboard
  1. 生成签名
ceph dashboard create-self-signed-cert
  1. 创建目录
mkdir mgr-dashboard&&cd mgr-dashboard

[root@node00 mgr-dashboard]# pwd

/opt/ceph/ceph-cluster/mgr-dashboard

  1. 生成密钥对
cd  /opt/ceph/ceph-cluster/mgr-dashboard

openssl req -new -nodes -x509   -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650   -keyout dashboard.key -out dashboard.crt -extensions v3_ca

[root@linux30 mgr-dashboard]# ll

total 8

-rw-rw-r-- 1 ceph_user ceph_user 1155 Jul 14 02:26 dashboard.crt

-rw-rw-r-- 1 ceph_user ceph_user 1704 Jul 14 02:26 dashboard.key

  1. 启动dashboard
ceph mgr module disable dashboard

ceph mgr module enable dashboard
  1. 设置IP与PORT
ceph config set mgr mgr/dashboard/server_addr 192.168.247.146

ceph config set mgr mgr/dashboard/server_port 18843
  1. 关闭HTTPS
ceph config set mgr mgr/dashboard/ssl false
  1. 查看服务信息
[root@node00 ceph-cluster]# ceph mgr services

{

    "dashboard": "http://192.168.247.146:18843/"

}


ceph config set mgr mgr/dashboard/server_addr node00
  1. 设置管理用户与密码
ceph dashboard set-login-credentials admin admin
  1. 访问 http://192.168247.146:18843/

安装问题记录

ceph -s出现问题：

可能是时间同步问题，

ystemctl start ntpd     #新增的节点没有启动ntpd

systemctl  restart  ceph-mon.target

systemctl  restart  ceph-mon.target

ceph-deploy mon相关问题汇总：

ceph-deploy mon出现mon.node40 monitor is not yet in quorum, tries left: 5错误：

[root@node40 ceph-cluster]# ceph-deploy --overwrite-conf mon create-initial

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf mon create-initial

[ceph_deploy.cli][INFO  ] ceph-deploy options:

[ceph_deploy.cli][INFO  ]  username                      : None

[ceph_deploy.cli][INFO  ]  verbose                       : False

[ceph_deploy.cli][INFO  ]  overwrite_conf                : True

[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial

[ceph_deploy.cli][INFO  ]  quiet                         : False

[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fbd853c1f80>

[ceph_deploy.cli][INFO  ]  cluster                       : ceph

[ceph_deploy.cli][INFO  ]  func                          : <function mon at 0x7fbd8562c410>

[ceph_deploy.cli][INFO  ]  ceph_conf                     : None

[ceph_deploy.cli][INFO  ]  default_release               : False

[ceph_deploy.cli][INFO  ]  keyrings                      : None

[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node40 node41 node42

[ceph_deploy.mon][DEBUG ] detecting platform for host node40 ...

[node40][DEBUG ] connection detected need for sudo

[node40][DEBUG ] connected to host: node40

[node40][DEBUG ] detect platform information from remote host

[node40][DEBUG ] detect machine type

[node40][DEBUG ] find the location of an executable

[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.9.2009 Core

[node40][DEBUG ] determining if provided host has same hostname in remote

[node40][DEBUG ] get remote short hostname

[node40][DEBUG ] deploying mon to node40

[node40][DEBUG ] get remote short hostname

[node40][DEBUG ] remote hostname: node40

[node40][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node40][DEBUG ] create the mon path if it does not exist

[node40][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node40/done

[node40][DEBUG ] create a done file to avoid re-doing the mon deployment

[node40][DEBUG ] create the init path if it does not exist

[node40][INFO  ] Running command: sudo systemctl enable ceph.target

[node40][INFO  ] Running command: sudo systemctl enable ceph-mon@node40

[node40][INFO  ] Running command: sudo systemctl start ceph-mon@node40

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[node40][DEBUG ] ********************************************************************************

[node40][DEBUG ] status for monitor: mon.node40

[node40][DEBUG ] {

[node40][DEBUG ]   "election_epoch": 1,

[node40][DEBUG ]   "extra_probe_peers": [

[node40][DEBUG ]     {

[node40][DEBUG ]       "addrvec": [

[node40][DEBUG ]         {

[node40][DEBUG ]           "addr": "192.168.247.142:6789",

[node40][DEBUG ]           "nonce": 0,

[node40][DEBUG ]           "type": "v1"

[node40][DEBUG ]         }

[node40][DEBUG ]       ]

[node40][DEBUG ]     },

[node40][DEBUG ]     {

[node40][DEBUG ]       "addrvec": [

[node40][DEBUG ]         {

[node40][DEBUG ]           "addr": "192.168.247.141:3300",

[node40][DEBUG ]           "nonce": 0,

[node40][DEBUG ]           "type": "v2"

[node40][DEBUG ]         },

[node40][DEBUG ]         {

[node40][DEBUG ]           "addr": "192.168.247.141:6789",

[node40][DEBUG ]           "nonce": 0,

[node40][DEBUG ]           "type": "v1"

[node40][DEBUG ]         }

[node40][DEBUG ]       ]

[node40][DEBUG ]     },

[node40][DEBUG ]     {

[node40][DEBUG ]       "addrvec": [

[node40][DEBUG ]         {

[node40][DEBUG ]           "addr": "192.168.247.142:3300",

[node40][DEBUG ]           "nonce": 0,

[node40][DEBUG ]           "type": "v2"

[node40][DEBUG ]         },

[node40][DEBUG ]         {

[node40][DEBUG ]           "addr": "192.168.247.142:6789",

[node40][DEBUG ]           "nonce": 0,

[node40][DEBUG ]           "type": "v1"

[node40][DEBUG ]         }

[node40][DEBUG ]       ]

[node40][DEBUG ]     }

[node40][DEBUG ]   ],

[node40][DEBUG ]   "feature_map": {

[node40][DEBUG ]     "mon": [

[node40][DEBUG ]       {

[node40][DEBUG ]         "features": "0x3ffddff8ffecffff",

[node40][DEBUG ]         "num": 1,

[node40][DEBUG ]         "release": "luminous"

[node40][DEBUG ]       }

[node40][DEBUG ]     ]

[node40][DEBUG ]   },

[node40][DEBUG ]   "features": {

[node40][DEBUG ]     "quorum_con": "0",

[node40][DEBUG ]     "quorum_mon": [],

[node40][DEBUG ]     "required_con": "0",

[node40][DEBUG ]     "required_mon": []

[node40][DEBUG ]   },

[node40][DEBUG ]   "monmap": {

[node40][DEBUG ]     "created": "2022-04-08 14:14:20.855876",

[node40][DEBUG ]     "epoch": 0,

[node40][DEBUG ]     "features": {

[node40][DEBUG ]       "optional": [],

[node40][DEBUG ]       "persistent": []

[node40][DEBUG ]     },

[node40][DEBUG ]     "fsid": "b3299c95-745f-467f-91e4-a3e30c490483",

[node40][DEBUG ]     "min_mon_release": 0,

[node40][DEBUG ]     "min_mon_release_name": "unknown",

[node40][DEBUG ]     "modified": "2022-04-08 14:14:20.855876",

[node40][DEBUG ]     "mons": [

[node40][DEBUG ]       {

[node40][DEBUG ]         "addr": "192.168.247.140:6789/0",

[node40][DEBUG ]         "name": "node40",

[node40][DEBUG ]         "public_addr": "192.168.247.140:6789/0",

[node40][DEBUG ]         "public_addrs": {

[node40][DEBUG ]           "addrvec": [

[node40][DEBUG ]             {

[node40][DEBUG ]               "addr": "192.168.247.140:3300",

[node40][DEBUG ]               "nonce": 0,

[node40][DEBUG ]               "type": "v2"

[node40][DEBUG ]             },

[node40][DEBUG ]             {

[node40][DEBUG ]               "addr": "192.168.247.140:6789",

[node40][DEBUG ]               "nonce": 0,

[node40][DEBUG ]               "type": "v1"

[node40][DEBUG ]             }

[node40][DEBUG ]           ]

[node40][DEBUG ]         },

[node40][DEBUG ]         "rank": 0

[node40][DEBUG ]       },

[node40][DEBUG ]       {

[node40][DEBUG ]         "addr": "192.168.247.142:6789/0",

[node40][DEBUG ]         "name": "node42",

[node40][DEBUG ]         "public_addr": "192.168.247.142:6789/0",

[node40][DEBUG ]         "public_addrs": {

[node40][DEBUG ]           "addrvec": [

[node40][DEBUG ]             {

[node40][DEBUG ]               "addr": "192.168.247.142:6789",

[node40][DEBUG ]               "nonce": 0,

[node40][DEBUG ]               "type": "v1"

[node40][DEBUG ]             }

[node40][DEBUG ]           ]

[node40][DEBUG ]         },

[node40][DEBUG ]         "rank": 1

[node40][DEBUG ]       },

[node40][DEBUG ]       {

[node40][DEBUG ]         "addr": "0.0.0.0:0/1",

[node40][DEBUG ]         "name": "node41",

[node40][DEBUG ]         "public_addr": "0.0.0.0:0/1",

[node40][DEBUG ]         "public_addrs": {

[node40][DEBUG ]           "addrvec": [

[node40][DEBUG ]             {

[node40][DEBUG ]               "addr": "0.0.0.0:0",

[node40][DEBUG ]               "nonce": 1,

[node40][DEBUG ]               "type": "v1"

[node40][DEBUG ]             }

[node40][DEBUG ]           ]

[node40][DEBUG ]         },

[node40][DEBUG ]         "rank": 2

[node40][DEBUG ]       }

[node40][DEBUG ]     ]

[node40][DEBUG ]   },

[node40][DEBUG ]   "name": "node40",

[node40][DEBUG ]   "outside_quorum": [

[node40][DEBUG ]     "node40"

[node40][DEBUG ]   ],

[node40][DEBUG ]   "quorum": [],

[node40][DEBUG ]   "rank": 0,

[node40][DEBUG ]   "state": "probing",

[node40][DEBUG ]   "sync_provider": []

[node40][DEBUG ] }

[node40][DEBUG ] ********************************************************************************

[node40][INFO  ] monitor: mon.node40 is running

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][DEBUG ] detecting platform for host node41 ...

[node41][DEBUG ] connection detected need for sudo

[node41][DEBUG ] connected to host: node41

[node41][DEBUG ] detect platform information from remote host

[node41][DEBUG ] detect machine type

[node41][DEBUG ] find the location of an executable

[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.9.2009 Core

[node41][DEBUG ] determining if provided host has same hostname in remote

[node41][DEBUG ] get remote short hostname

[node41][DEBUG ] deploying mon to node41

[node41][DEBUG ] get remote short hostname

[node41][DEBUG ] remote hostname: node41

[node41][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node41][DEBUG ] create the mon path if it does not exist

[node41][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node41/done

[node41][DEBUG ] create a done file to avoid re-doing the mon deployment

[node41][DEBUG ] create the init path if it does not exist

[node41][INFO  ] Running command: sudo systemctl enable ceph.target

[node41][INFO  ] Running command: sudo systemctl enable ceph-mon@node41

[node41][INFO  ] Running command: sudo systemctl start ceph-mon@node41

[node41][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node41.asok mon_status

[node41][DEBUG ] ********************************************************************************

[node41][DEBUG ] status for monitor: mon.node41

[node41][DEBUG ] {

[node41][DEBUG ]   "election_epoch": 117,

[node41][DEBUG ]   "extra_probe_peers": [],

[node41][DEBUG ]   "feature_map": {

[node41][DEBUG ]     "mon": [

[node41][DEBUG ]       {

[node41][DEBUG ]         "features": "0x3ffddff8ffecffff",

[node41][DEBUG ]         "num": 1,

[node41][DEBUG ]         "release": "luminous"

[node41][DEBUG ]       }

[node41][DEBUG ]     ]

[node41][DEBUG ]   },

[node41][DEBUG ]   "features": {

[node41][DEBUG ]     "quorum_con": "0",

[node41][DEBUG ]     "quorum_mon": [],

[node41][DEBUG ]     "required_con": "2449958747315912708",

[node41][DEBUG ]     "required_mon": [

[node41][DEBUG ]       "kraken",

[node41][DEBUG ]       "luminous",

[node41][DEBUG ]       "mimic",

[node41][DEBUG ]       "osdmap-prune",

[node41][DEBUG ]       "nautilus"

[node41][DEBUG ]     ]

[node41][DEBUG ]   },

[node41][DEBUG ]   "monmap": {

[node41][DEBUG ]     "created": "2022-04-08 14:02:08.362899",

[node41][DEBUG ]     "epoch": 1,

[node41][DEBUG ]     "features": {

[node41][DEBUG ]       "optional": [],

[node41][DEBUG ]       "persistent": [

[node41][DEBUG ]         "kraken",

[node41][DEBUG ]         "luminous",

[node41][DEBUG ]         "mimic",

[node41][DEBUG ]         "osdmap-prune",

[node41][DEBUG ]         "nautilus"

[node41][DEBUG ]       ]

[node41][DEBUG ]     },

[node41][DEBUG ]     "fsid": "b3299c95-745f-467f-91e4-a3e30c490483",

[node41][DEBUG ]     "min_mon_release": 14,

[node41][DEBUG ]     "min_mon_release_name": "nautilus",

[node41][DEBUG ]     "modified": "2022-04-08 14:02:08.362899",

[node41][DEBUG ]     "mons": [

[node41][DEBUG ]       {

[node41][DEBUG ]         "addr": "192.168.247.141:6789/0",

[node41][DEBUG ]         "name": "node41",

[node41][DEBUG ]         "public_addr": "192.168.247.141:6789/0",

[node41][DEBUG ]         "public_addrs": {

[node41][DEBUG ]           "addrvec": [

[node41][DEBUG ]             {

[node41][DEBUG ]               "addr": "192.168.247.141:6789",

[node41][DEBUG ]               "nonce": 0,

[node41][DEBUG ]               "type": "v1"

[node41][DEBUG ]             }

[node41][DEBUG ]           ]

[node41][DEBUG ]         },

[node41][DEBUG ]         "rank": 0

[node41][DEBUG ]       },

[node41][DEBUG ]       {

[node41][DEBUG ]         "addr": "192.168.247.142:6789/0",

[node41][DEBUG ]         "name": "node42",

[node41][DEBUG ]         "public_addr": "192.168.247.142:6789/0",

[node41][DEBUG ]         "public_addrs": {

[node41][DEBUG ]           "addrvec": [

[node41][DEBUG ]             {

[node41][DEBUG ]               "addr": "192.168.247.142:6789",

[node41][DEBUG ]               "nonce": 0,

[node41][DEBUG ]               "type": "v1"

[node41][DEBUG ]             }

[node41][DEBUG ]           ]

[node41][DEBUG ]         },

[node41][DEBUG ]         "rank": 1

[node41][DEBUG ]       },

[node41][DEBUG ]       {

[node41][DEBUG ]         "addr": "0.0.0.0:0/1",

[node41][DEBUG ]         "name": "node40",

[node41][DEBUG ]         "public_addr": "0.0.0.0:0/1",

[node41][DEBUG ]         "public_addrs": {

[node41][DEBUG ]           "addrvec": [

[node41][DEBUG ]             {

[node41][DEBUG ]               "addr": "0.0.0.0:0",

[node41][DEBUG ]               "nonce": 1,

[node41][DEBUG ]               "type": "v1"

[node41][DEBUG ]             }

[node41][DEBUG ]           ]

[node41][DEBUG ]         },

[node41][DEBUG ]         "rank": 2

[node41][DEBUG ]       }

[node41][DEBUG ]     ]

[node41][DEBUG ]   },

[node41][DEBUG ]   "name": "node41",

[node41][DEBUG ]   "outside_quorum": [

[node41][DEBUG ]     "node41"

[node41][DEBUG ]   ],

[node41][DEBUG ]   "quorum": [],

[node41][DEBUG ]   "rank": 0,

[node41][DEBUG ]   "state": "probing",

[node41][DEBUG ]   "sync_provider": []

[node41][DEBUG ] }

[node41][DEBUG ] ********************************************************************************

[node41][INFO  ] monitor: mon.node41 is running

[node41][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node41.asok mon_status

[ceph_deploy.mon][DEBUG ] detecting platform for host node42 ...

[node42][DEBUG ] connection detected need for sudo

[node42][DEBUG ] connected to host: node42

[node42][DEBUG ] detect platform information from remote host

[node42][DEBUG ] detect machine type

[node42][DEBUG ] find the location of an executable

[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.9.2009 Core

[node42][DEBUG ] determining if provided host has same hostname in remote

[node42][DEBUG ] get remote short hostname

[node42][DEBUG ] deploying mon to node42

[node42][DEBUG ] get remote short hostname

[node42][DEBUG ] remote hostname: node42

[node42][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node42][DEBUG ] create the mon path if it does not exist

[node42][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node42/done

[node42][DEBUG ] create a done file to avoid re-doing the mon deployment

[node42][DEBUG ] create the init path if it does not exist

[node42][INFO  ] Running command: sudo systemctl enable ceph.target

[node42][INFO  ] Running command: sudo systemctl enable ceph-mon@node42

[node42][INFO  ] Running command: sudo systemctl start ceph-mon@node42

[node42][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node42.asok mon_status

[node42][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

[node42][WARNIN] monitor: mon.node42, might not be running yet

[node42][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node42.asok mon_status

[node42][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

[node42][WARNIN] monitor node42 does not exist in monmap

[ceph_deploy.mon][INFO  ] processing monitor mon.node40

[node40][DEBUG ] connection detected need for sudo

[node40][DEBUG ] connected to host: node40

[node40][DEBUG ] detect platform information from remote host

[node40][DEBUG ] detect machine type

[node40][DEBUG ] find the location of an executable

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node40 monitor is not yet in quorum, tries left: 5

[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node40 monitor is not yet in quorum, tries left: 4

[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node40 monitor is not yet in quorum, tries left: 3

[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node40 monitor is not yet in quorum, tries left: 2

[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying

[node40][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node40.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node40 monitor is not yet in quorum, tries left: 1

[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying

[ceph_deploy.mon][INFO  ] processing monitor mon.node41

[node41][DEBUG ] connection detected need for sudo

[node41][DEBUG ] connected to host: node41

[node41][DEBUG ] detect platform information from remote host

[node41][DEBUG ] detect machine type

[node41][DEBUG ] find the location of an executable

[node41][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node41.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node41 monitor is not yet in quorum, tries left: 5

[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying

[node41][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node41.asok mon_status

[ceph_deploy.mon][WARNIN] mon.node41 monitor is not yet in quorum, tries left: 4

[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying

错误的含义是：mon.node40 监视器尚未达到仲裁状态，经过多轮尝试后失败。

网络参考可能原因：

  1. 防火墙：

  2. hosts配置和hostname 不一致

  3. public_network配置问题



    一些其他的文档说明是地址是 public_network如下图：



    [分布式存储]Ceph环境部署失败问题总结

Ceph集群搭建记录的更多相关文章

  1. kafka集群搭建记录

    本文记录搭建kafka搭建过程. 一.硬件机器介绍 192.168.183.195 master-node 192.168.183.194 data-node1 192.168.183.196 dat ...

  2. zookeeper集群搭建记录

    本文仅记录zookeeper集群搭建的过程,留待日后查看.使用. 一.硬件机器: 192.168.183.195 master-node 192.168.183.194 data-node1 192. ...

  3. ceph集群搭建

    CEPH 1.组成部分 1.1 monitor admin节点安装ceph-deploy工具 admin节点安装ceph-deploy 添加源信息 rm -f /etc/yum.repos.d/* w ...

  4. Ceph集群搭建及Kubernetes上实现动态存储（StorageClass）

    集群准备 ceph集群配置说明   节点名称 IP地址 配置 作用 ceph-moni-0 10.10.3.150 centos7.5 4C,16G,200Disk 管理节点,监视器 monitor ...

  5. Centos下Apache+Tomcat集群--搭建记录

    一.目的 利用apache的mod_jk模块,实现tomcat集群服务器的负载均衡以及会话复制,这里用到了<Cluster>. 二.环境 1.基础:3台主机,系统Centos6.5,4G内 ...

  6. Ceph 集群搭建

    1.Ceph部署 1.1 Ceph 安装前准备,环境与拓扑图如下: 主机名 外网IP 内网IP 安装组件 磁盘块 c720181 172.22.69.84 192.168.20.181 ceph-de ...

  7. CEPH集群搭建（CentOS 7）

    以包含四个节点的集群为例,其中包括一个 ceph-deploy 管理节点和一个三节点的Ceph存储集群. 下图中每个节点代表一台机器. 创建一个 Ceph 存储集群,它有一个 Monitor 和两个 ...

  8. ceph-deploy离线部署ceph集群及报错解决FAQ

    ceph-deploy部署ceph集群 环境介绍 主机名 ip地址 操作系统 角色 备注 ceph-node1 10.153.204.13 Centos7.6 mon.osd.mds.mgr.rgw. ...

  9. docker创建ceph集群

    背景 Ceph官方现在提供两类镜像来创建集群,一种是常规的,每一种Ceph组件是单独的一个镜像,如ceph/daemon.ceph/radosgw.ceph/mon.ceph/osd等:另外一种是最新 ...

随机推荐

  1. Android.mk文件如何打日志信息

    1. 在mk文件中添加:$(warning  "xxx="$(变量名)) 2. 执行lunch,选一个分支,此过程中可以打出添加的log.

  2. JavaScript实现简单轮播图动画

    运行效果: 源代码: <!DOCTYPE html> <html lang="zh"> <head> <meta charset=&quo ...

  3. 浅谈ES6中的Async函数

    转载地址:https://www.cnblogs.com/sghy/p/7987640.html 定义:Async函数是一个异步操作函数,本质上,Async函数是Generator函数的语法糖.asy ...

  4. Spring理解1 ioc

    Spring Spring是一个轻量级的控制反转(IOC)和面向切面(AOP)的容器(框架).   需要了解 ioc容器 IOC底层原理 IOC接口 BeanFactory Bean的作用域 IOC操 ...

  5. findmnt、lsblk、mount 命令查看磁盘、目录挂载、挂载点以及文件系统格式等情况

    findmnt 展示出了目标挂载点( TARGET ).源设备( SOURCE ).文件系统类型( FSTYPE )以及相关的挂载选项( OPTIONS ),例如文件系统是否是可读可写或者只读的.根( ...

  6. 基于Vue开发的门户网站展示和后台数据管理系统

    基于Vue的前端框架有很多,这几年随着前端技术的官方应用,总有是学不完的前端知识在等着我们,一个人的精力也是有限,不可能一一掌握,不过我们学习很大程度都会靠兴趣驱动,或者目标导向,最终是可以以点破面, ...

  7. JavaScript学习①

    # 今日内容 1. JavaScript基础 ## JavaScript: * 概念: 一门客户端脚本语言 * 运行在客户端浏览器中的.每一个浏览器都有JavaScript的解析引擎 * 脚本语言:不 ...

  8. 2021.11.11 P4052 [JSOI2007]文本生成器（AC自动机+DP）

    2021.11.11 P4052 [JSOI2007]文本生成器(AC自动机+DP) https://www.luogu.com.cn/problem/P4052 题意: JSOI 交给队员 ZYX ...

  9. 用 getElementsByTagName() 来获取,父元素指定的子元素

    1. html 结构 <ul> <li>知否知否,应是等你好久11</li> <li>知否知否,应是等你好久11</li> <li&g ...

  10. 技术管理进阶——一线Leader怎么做？经理的速成宝典

    原创不易,求分享.求一键三连 本期培训材料关注公众号后回复:经理培训,获得 前段时间有个同学问我有没有一线Leader的速成培训课程,很好的问题,首先我们需要定义一下什么是小Leader: 所谓小Le ...