ceph rgw的使用，dashboard和监控

1.部署RadosGW 服务

将ceph-mgr1、ceph-mgr2 服务器部署为高可用的radosGW 服务

1.1 安装radosgw 服务

root@mgr1:~# apt install radosgw root@mgr2:~# apt install radosgw

1.2 初始化rgw节点

$ ceph-deploy --overwrite-conf rgw create mgr1
$ ceph-deploy --overwrite-conf rgw create mgr2

1.3 验证radosgw 服务状态

$ ceph -s
cluster:
    id:     54ed6318-9830-4152-917c-f1af7fa1002a
    health: HEALTH_OK
    services:     mon: 3 daemons, quorum mon1,mon2,mon3 (age 3d)
    mgr: mgr1(active, since 3d), standbys: mgr2
    mds: 2/2 daemons up, 2 standby
    osd: 20 osds: 20 up (since 3d), 20 in (since 3d)
    rgw: 2 daemons active (2 hosts, 1 zones)

data:
    volumes: 1/1 healthy
    pools:   8 pools, 217 pgs
    bjects: 477 objects, 774 MiB
    usage:   11 GiB used, 1.9 TiB / 2.0 TiB avail
    pgs:     217 active+clean

1.4 验证radosgw 服务进程

root@mgr1:~# ps -ef |grep radosgw
ceph      1261     1  0 10:40 ?        00:00:02 /usr/bin/radosgw -f --cluster ceph --name client.rgw.mgr1 --setuser ceph --setgroup ceph  

root@mgr2:~# ps -ef |grep radosgw
ceph     32197     1  0 10:41 ?        00:00:02 /usr/bin/radosgw -f --cluster ceph --name client.rgw.mgr2 --setuser ceph --setgroup ceph

2.radosgw 服务配置

2.1 自定义端口

2.1.1 配置文件可以在ceph deploy 服务器修改然后统一推送，或者单独修改每个radosgw 服务器的配置为同一配置。

[client.rgw.mgr1] rgw_host = mgr1 rgw_frontends = civetweb port=8080
[client.rgw.mgr2] rgw_host = mgr2 rgw_frontends = civetweb port=8080

2.1.2 同步配置文件到rgw1(mgr1),rgw2(mgr2)

cephuser@ceph-deploy:~/ceph-cluster$ scp ceph.conf root@mgr1:/etc/ceph/
cephuser@ceph-deploy:~/ceph-cluster$ scp ceph.conf root@mgr2:/etc/ceph/

2.1.3 在rgw节点重启rgw服务

root@mgr1:~# systemctl restart ceph-radosgw@rgw.mgr1.service
root@mgr2:~# systemctl restart ceph-radosgw@rgw.mgr2.service

root@mgr2:~# netstat -ntlp |grep radosgw
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address
Foreign Address         State       PID/Program name     tcp        0      0
0.0.0.0:8080            0.0.0.0:*               LISTEN      32197/radosgw

2.2 配置nginx 代理

2.2.1 安装nginx并配置

apt install nginx

rm -f /etc/nginx/sites-enabled/default

vim /etc/nginx/sites-enabled/ceph_rgw.conf
upstream rgw_yanceph {
    server 192.168.2.31:8080;
    server 192.168.2.32:8080;
}

server {
    listen       80;
    server_name  rgw.yanceph.com;
    charset utf-8;

    location / {
        proxy_pass  http://rgw_yanceph;
        proxy_set_header    Host   $host;
        access_log /var/log/nginx/rgw.yanceph.com.log;
    }
} 

# nginx 配置检查

nginx -t  

# nginx 后台启动
nginx

2.2.2 winows测试机上配置hosts

192.168.2.2 rgw.yanceph.com

2.2.3 访问rgw.yanceph.com 测试：

2.2.4 如需https协议， 可以申请域名的ssl证书并在nginx配置即可，配置文件修改如下

listen       443 ssl;
server_name  lrgw.yanceph.com;

ssl_certificate      cert/rgw.yanceph.comt.pem;
ssl_certificate_key  cert/rgw.yanceph.com.key;
charset utf-8;

3.Ceph dashboard

Ceph dashboard 是通过一个web 界面，对已经运行的ceph 集群进行状态查看及功能配置等功能，

3.1 启用dashboard 插件

Ceph mgr 是一个多插件( 模块化) 的组件， 其组件可以单独的启用或关闭

root@mgr1:~# apt install ceph-mgr-dashboard

3.2 查看开启的模块

cephuser@ceph-deploy:~/ceph-cluster$ ceph mgr module ls |head -n 30
{
...
    "enabled_modules": [
        "dashboard",
        "iostat",
        "nfs",
        "restful"
    ],
...
}

3.3 启用dashboard 模块

Ceph dashboard 在mgr 节点进行开启设置，并且可以配置开启或者关闭SSL

# 开启dashboard模块
$ ceph mgr module enable dashboard 

# 开启dashboard模块的ssl （关闭是false）
$ ceph config set mgr mgr/dashboard/ssl true

# 设置监听地址
ceph confi set mgr mgr/dashboard/mgr1/server_addr 192.168.2.31

#设置监听端口
ceph confi set mgr mgr/dashboard/mgr1/server_port 9009

如果集群报错，需要重启mgr服务

cephuser@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     54ed6318-9830-4152-917c-f1af7fa1002a
    health: HEALTH_ERR
            Module 'dashboard' has failed: OSError("Port 8080 not free on '192.168.2.31'",)

root@mgr1:~# systemctl restart ceph-mgr@mgr1.service

3.4 查看dashboard服务状态

cephuser@ceph-deploy:~/ceph-cluster$ ceph mgr services
{
    "dashboard": "http://192.168.2.31:9009/"
}

3.5 设置dashboard账号和密码

$  touch ceph-dashboard-passwd.txt

$ echo admin123123 >> ceph-dashboard-passwd.txt 

$ ceph dashboard set-login-credentials admin -i ceph-dashboard-passwd.txt
******************************************************************
***          WARNING: this command is deprecated.              ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************

3.6 验证并访问dashboard

http://192.168.2.31:9009/

3.7 配置dashboard SSL

# ceph 自签名证书
$ ceph dashboard create-self-signed-cert
Self-signed certificate created

$ ceph config set mgr mgr/dashboard/ssl true

如果不生效需要重启mgr服务

root@mgr1:~# systemctl restart ceph-mgr@mgr1.service

3.8 查看dashboard ssl

cephuser@ceph-deploy:~/ceph-cluster$ ceph mgr services
{
    "dashboard": "https://192.168.2.31:8443/"
}

3.9 使用https访问dashboard

---

4.使用prometheus监控ceph node节点

4.1 部署prometheus

# 创建专用目录
mkdir /apps
cd /apps

# 下载安装包
root@mgr1:/apps# wget https://mirrors.tuna.tsinghua.edu.cn/github-release/prometheus/prometheus/LatestRelease/prometheus-2.29.2.linux-amd64.tar.gz

# 解压安装包
root@mgr1:/apps# tar xf prometheus-2.29.2.linux-amd64.tar.gz

# 创建软连接，主要方便后续版本升级，执行修改软连接即可
root@mgr1:/apps# ln -s /apps/prometheus-2.29.2.linux-amd64 /apps/prometheus

4.2 配置prometheus

# 新建配置文件
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/apps/prometheus/
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target

#启动服务
root@mgr1:/apps/prometheus# systemctl daemon-reload
root@mgr1:/apps/prometheus# systemctl restart prometheus
root@mgr1:/apps/prometheus# systemctl enable prometheus

4.3 浏览器访问prometheus

4.4 部署node_exporter

各node(osd)节点安装node_exporter

# 创建目录
root@osd1:/apps# mkdir /apps && cd !$

# 下载安装包
root@osd1:/apps# wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz

# 解压
root@osd1:/apps# tar xf node_exporter-1.2.2.linux-amd64.tar.gz 

# 为方便后续版本升级创建软连接
root@osd1:/apps# ln -sv /apps/node_exporter-1.2.2.linux-amd64

4.5 配置node_exporter

# 创建配置文件
root@osd1:/apps# vim /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/apps/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

# 启动服务
root@osd1:/apps# systemctl daemon-reload
root@osd1:/apps# systemctl restart node-exporter
root@osd1:/apps# systemctl enable node-exporter

# 查看进程和监听端口号
root@osd1:/apps# ps -ef |grep exporter
root       29396       1  0 16:23 ?        00:00:00 /apps/node_exporter/node_exporter
root       29457   29017  0 16:23 pts/0    00:00:00 grep --color=auto exporter

root@osd1:/apps# netstat -ntlp |grep exporter
tcp6       0      0 :::9100                 :::*                    LISTEN      29396/node_exporter

其他node节点也需安装

4.6 配置prometheus server,增加node节点监控配置

root@mgr1:/apps/prometheus# vim prometheus.yml
  - job_name: 'ceph-node-data'
    static_configs:
            - targets: ['192.168.2.41:9100','192.168.2.42:9100','192.168.2.43:9100','192.168.2.44:9100']
            -
# 重启服务
root@mgr1:/apps/prometheus# systemctl restart prometheus.service

4.7浏览器访问

---

5.通过prometheus 监控ceph 服务

Ceph manager 内部的模块中包含了prometheus 的监控模块,并监听在每个manager 节点的9283 端口，该端口用于将采集到的信息通过http 接口向prometheus 提供数据

5.1 启用ceph prometheus 监控模块

$ ceph mgr module enable prometheus

5.2 prometheus server 验证监听端口

root@mgr1:/apps/prometheus# netstat -ntlp |grep 9283
tcp        0      0 192.168.2.31:9283       0.0.0.0:*               LISTEN      2687/ceph-mgr  

5.3 验证manager 数据

curl "http://192.168.2.31:9283"

5.4 配置prometheus采集数据

root@mgr1:/apps/prometheus# vim prometheus.yml
  - job_name: 'ceph-cluster-data'
    static_configs:
            - targets: ['192.168.2.31:9283']

---

6. 通过grafana显示监控数据

通过granfana 显示对ceph 的集群监控数据及node 数据

6.1 安装grafana

# 安装依赖包
root@mgr1:/apps# apt-get install -y adduser libfontconfig1

# 下载安装包
root@mgr1:/apps# wget https://dl.grafana.com/oss/release/grafana_8.1.2_amd64.deb

# 安装
root@mgr1:/apps# dpkg -i grafana_8.1.2_amd64.deb

# 启动
root@mgr1:/apps# systemctl restart grafana-server.service

# 设置开机自启动
root@mgr1:/apps# systemctl enable grafana-server.service

6.2 浏览器访问

初始账号密码admin:admin

6.3 配置数据源

configuration -> Data souces

6.4 导入模板

create -> import

2842

5346