一 部署高可用kube-controller-manager

1.1 高可用kube-controller-manager介绍

本实验部署一个三实例 kube-controller-manager 的集群,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用时,阻塞的节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。
为保证通信安全,本文档先生成 x509 证书和私钥,kube-controller-manager 在如下两种情况下使用该证书:
  • 与 kube-apiserver 的安全端口通信;
  • 在安全端口(https,10252) 输出 prometheus 格式的 metrics。

1.2 创建kube-controller-manager证书和私钥

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > kube-controller-manager-csr.json <<EOF
3 {
4 "CN": "system:kube-controller-manager",
5 "hosts": [
6 "127.0.0.1",
7 "172.24.8.71",
8 "172.24.8.72",
9 "172.24.8.73"
10 ],
11 "key": {
12 "algo": "rsa",
13 "size": 2048
14 },
15 "names": [
16 {
17 "C": "CN",
18 "ST": "Shanghai",
19 "L": "Shanghai",
20 "O": "system:kube-controller-manager",
21 "OU": "System"
22 }
23 ]
24 }
25 EOF
26 #创建kube-controller-manager的CA证书请求文件
解释:


hosts 列表包含所有 kube-controller-manager 节点 IP;


CN 和 O 均为 system:kube-controller-manager,kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
4 -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager #生成CA密钥(ca-key.pem)和证书(ca.pem)

1.3 分发证书和私钥

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp kube-controller-manager*.pem root@${master_ip}:/etc/kubernetes/cert/
7 done

1.4 创建和分发kubeconfig

kube-controller-manager 使用 kubeconfig 文件访问 apiserver,该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书:
  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \
4 --certificate-authority=/opt/k8s/work/ca.pem \
5 --embed-certs=true \
6 --server=${KUBE_APISERVER} \
7 --kubeconfig=kube-controller-manager.kubeconfig
8
9 [root@k8smaster01 work]# kubectl config set-credentials system:kube-controller-manager \
10 --client-certificate=kube-controller-manager.pem \
11 --client-key=kube-controller-manager-key.pem \
12 --embed-certs=true \
13 --kubeconfig=kube-controller-manager.kubeconfig
14
15 [root@k8smaster01 work]# kubectl config set-context system:kube-controller-manager \
16 --cluster=kubernetes \
17 --user=system:kube-controller-manager \
18 --kubeconfig=kube-controller-manager.kubeconfig
19
20 [root@k8smaster01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
21
22 [root@k8smaster01 ~]# cd /opt/k8s/work
23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
24 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
25 do
26 echo ">>> ${master_ip}"
27 scp kube-controller-manager.kubeconfig root@${master_ip}:/etc/kubernetes/
28 done

1.5 创建kube-controller-manager的systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kube-controller-manager.service.template <<EOF
4 [Unit]
5 Description=Kubernetes Controller Manager
6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
7
8 [Service]
9 WorkingDirectory=${K8S_DIR}/kube-controller-manager
10 ExecStart=/opt/k8s/bin/kube-controller-manager \\
11 --profiling \\
12 --cluster-name=kubernetes \\
13 --controllers=*,bootstrapsigner,tokencleaner \\
14 --kube-api-qps=1000 \\
15 --kube-api-burst=2000 \\
16 --leader-elect \\
17 --use-service-account-credentials\\
18 --concurrent-service-syncs=2 \\
19 --bind-address=##MASTER_IP## \\
20 --secure-port=10252 \\
21 --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
22 --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
23 --port=0 \\
24 --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
25 --client-ca-file=/etc/kubernetes/cert/ca.pem \\
26 --requestheader-allowed-names="" \\
27 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
28 --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
29 --requestheader-group-headers=X-Remote-Group \\
30 --requestheader-username-headers=X-Remote-User \\
31 --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
32 --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
33 --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
34 --experimental-cluster-signing-duration=8760h \\
35 --horizontal-pod-autoscaler-sync-period=10s \\
36 --concurrent-deployment-syncs=10 \\
37 --concurrent-gc-syncs=30 \\
38 --node-cidr-mask-size=24 \\
39 --service-cluster-ip-range=${SERVICE_CIDR} \\
40 --pod-eviction-timeout=6m \\
41 --terminated-pod-gc-threshold=10000 \\
42 --root-ca-file=/etc/kubernetes/cert/ca.pem \\
43 --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
44 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
45 --logtostderr=true \\
46 --v=2
47 Restart=on-failure
48 RestartSec=5
49
50 [Install]
51 WantedBy=multi-user.target
52 EOF

1.6 分发systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ ))
4 do
5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${MASTER_IPS[i]}.service
6 done #修正相应IP
7 [root@k8smaster01 work]# ls kube-controller-manager*.service
8 [root@k8smaster01 ~]# cd /opt/k8s/work
9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
11 do
12 echo ">>> ${master_ip}"
13 scp kube-controller-manager-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-controller-manager.service
14 done #分发system

二 启动并验证

2.1 启动kube-controller-manager 服务

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
7 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
8 done

2.2 检查kube-controller-manager 服务

  1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}
3 do
4 echo ">>> ${master_ip}"
5 ssh root@${master_ip} "systemctl status kube-controller-manager|grep Active"
6 done

2.3 查看输出的 metrics

  1 [root@k8smaster01 ~]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.24.8.71:10252/metrics |head
注意:以上命令在 kube-controller-manager 节点上执行。

2.4 查看权限

  1 [root@k8smaster01 ~]# kubectl describe clusterrole system:kube-controller-manager
ClusteRole system:kube-controller-manager 的权限很小,只能创建 secret、serviceaccount 等资源对象,各 controller 的权限分散到 ClusterRole system:controller:XXX 中。
当在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数,这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。
  1 [root@k8smaster01 ~]# kubectl get clusterrole|grep controller
如deployment controller:
  1 [root@k8smaster01 ~]# kubectl describe clusterrole system:controller:deployment-controller

2.5 查看当前leader

  1 [root@k8smaster01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
kubelet 认证和授权:https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization

010.Kubernetes二进制部署kube-controller-manager的更多相关文章

  1. Kubernetes 二进制部署(一)单节点部署(Master 与 Node 同一机器)

    0. 前言 最近受“新冠肺炎”疫情影响,在家等着,入职暂时延后,在家里办公和学习 尝试通过源码编译二进制的方式在单一节点(Master 与 Node 部署在同一个机器上)上部署一个 k8s 环境,整理 ...

  2. Kubernetes 二进制部署(二)集群部署(多 Master 节点通过 Nginx 负载均衡)

    0. 前言 紧接上一篇,本篇文章我们尝试学习多节点部署 kubernetes 集群 并通过 haproxy+keepalived 实现 Master 节点的负载均衡 1. 实验环境 实验环境主要为 5 ...

  3. kubernetes二进制部署k8s-master集群controller-manager服务unhealthy问题

    一.问题现象 我们使用二进制部署k8s的高可用集群时,在部署多master时,kube-controller-manager服务提示Unhealthy [root@ceph-01 system]# k ...

  4. 003.Kubernetes二进制部署准备

    一 前置准备 1.1 前置条件 相应的充足资源的Linux服务器: 设置相应的主机名,参考命令: hostnamectl set-hostname k8smaster Mac及UUID唯一: 若未关闭 ...

  5. 015.Kubernetes二进制部署所有节点kubelet

    一 部署 kubelet kubelet 运行在每个 worker 节点上,接收 kube-apiserver 发送的请求,管理 Pod 容器,执行交互式命令,如 exec.run.logs 等. k ...

  6. Kubernetes 二进制部署

    目录 1.基础环境 2.部署DNS 3.准备自签证书 4.部署Docker环境 5.私有仓库Harbor部署 6.部署Master节点 6.1.部署Etcd集群 6.2.部署kube-apiserve ...

  7. 005.Kubernetes二进制部署kubectl

    一 部署 kubectl 1.1 安装kubectl [root@k8smaster01 ~]# cd /opt/k8s/work [root@k8smaster01 work]# wget http ...

  8. 008.Kubernetes二进制部署Nginx实现高可用

    一 Nginx代理实现kube-apiserver高可用 1.1 Nginx实现高可用 基于 nginx 代理的 kube-apiserver 高可用方案. 控制节点的 kube-controller ...

  9. 009.Kubernetes二进制部署kube-apiserver

    一 部署master节点 1.1 master节点服务 kubernetes master 节点运行如下组件: kube-apiserver kube-scheduler kube-controlle ...

随机推荐

  1. 流水线机制、滑动窗口协议、GBN、SR

    一.滑动窗口协议 为了解决停等操作的性能问题(发了一个分组之后一直等到确认了这个分组才发下一个),推出了流水线机制,提供资源利用率.就是允许发送方在收到对方的ACK前,发送多个分组 其中窗口是一个范围 ...

  2. Flask中的渲染变量

    Flask中的渲染变量 一.渲染变量 <!DOCTYPE html> <html lang="en"> <head> <meta char ...

  3. Mycat读写分离(一主一从)

    Mycat读写分离(一主一从) 我们一共使用2个虚拟机,每个机器的作用如下: 主机名 IP地址 任务角色 数据库 node1 192.168.1.121 Mycat, master MySQL nod ...

  4. R-plotly|交互式甘特图(Gantt chart)-项目管理/学习计划

    本文首发于“生信补给站”微信公众号,https://mp.weixin.qq.com/s/CGz51qOjFSJ4Wx_qOMzjiw 更多关于R语言,ggplot2绘图,生信分析的内容,敬请关注小号 ...

  5. Roadmap of FE

    未完待补充......

  6. 如何用js做关灯游戏

    编辑器  Sublime Text 3 <!DOCTYPE html><html lang="en"><head> <meta chars ...

  7. linux ln命令 建立文件夹桌面快捷方式

    指令如下: sudo ln -s /root/myhack/ /root/Desktop 以上指令是创建软链接到桌面. 指令解析: ln -s是创建软链接指令,如果不加-s则是创建硬链接.

  8. [JZOJ5185] 【NOIP2017提高组模拟6.30】tty's sequence

    Description

  9. Halcon一日一练:创建AOI

    AOI:Area Of Interesting.感兴趣区域,即你要处理的区域. 一般情况,一整张图像,我们需要对局部进行操作,我们会选择一个我们需要处理的区域,对其进行处理,其他区域进行屏蔽.这个区域 ...

  10. 解决git报错“The file will have its original line endings in your working directory”的方法

    在执行命令 git commit --all -m  '说明' 时报错“The file will have its original line endings in your working dir ...