015.Kubernetes二进制部署所有节点kubelet
一 部署 kubelet
1.1 安装kubelet
1.2 分发kubelet
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 scp kubernetes/server/bin/kubelet root@${all_ip}:/opt/k8s/bin/
7 ssh root@${all_ip} "chmod +x /opt/k8s/bin/*"
8 done
1.3 分发kubeconfig
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_name in ${ALL_NAMES[@]}
4 do
5 echo ">>> ${all_name}"
6
7 # 创建 token
8 export BOOTSTRAP_TOKEN=$(kubeadm token create \
9 --description kubelet-bootstrap-token \
10 --groups system:bootstrappers:${all_name} \
11 --kubeconfig ~/.kube/config)
12
13 # 设置集群参数
14 kubectl config set-cluster kubernetes \
15 --certificate-authority=/etc/kubernetes/cert/ca.pem \
16 --embed-certs=true \
17 --server=${KUBE_APISERVER} \
18 --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig
19
20 # 设置客户端认证参数
21 kubectl config set-credentials kubelet-bootstrap \
22 --token=${BOOTSTRAP_TOKEN} \
23 --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig
24
25 # 设置上下文参数
26 kubectl config set-context default \
27 --cluster=kubernetes \
28 --user=kubelet-bootstrap \
29 --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig
30
31 # 设置默认上下文
32 kubectl config use-context default --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig
33 done
1 [root@k8smaster01 work]# kubeadm token list --kubeconfig ~/.kube/config #查看 kubeadm 为各节点创建的 token
2 [root@k8smaster01 work]# kubectl get secrets -n kube-system|grep bootstrap-token #查看各 token 关联的 Secret
1.5 分发bootstrap kubeconfig
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_name in ${ALL_NAMES[@]}
4 do
5 echo ">>> ${all_name}"
6 scp kubelet-bootstrap-${all_name}.kubeconfig root@${all_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
7 done
1.6 创建kubelet 参数配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kubelet-config.yaml.template <<EOF
4 kind: KubeletConfiguration
5 apiVersion: kubelet.config.k8s.io/v1beta1
6 address: "##ALL_IP##"
7 staticPodPath: ""
8 syncFrequency: 1m
9 fileCheckFrequency: 20s
10 httpCheckFrequency: 20s
11 staticPodURL: ""
12 port: 10250
13 readOnlyPort: 0
14 rotateCertificates: true
15 serverTLSBootstrap: true
16 authentication:
17 anonymous:
18 enabled: false
19 webhook:
20 enabled: true
21 x509:
22 clientCAFile: "/etc/kubernetes/cert/ca.pem"
23 authorization:
24 mode: Webhook
25 registryPullQPS: 0
26 registryBurst: 20
27 eventRecordQPS: 0
28 eventBurst: 20
29 enableDebuggingHandlers: true
30 enableContentionProfiling: true
31 healthzPort: 10248
32 healthzBindAddress: "##ALL_IP##"
33 clusterDomain: "${CLUSTER_DNS_DOMAIN}"
34 clusterDNS:
35 - "${CLUSTER_DNS_SVC_IP}"
36 nodeStatusUpdateFrequency: 10s
37 nodeStatusReportFrequency: 1m
38 imageMinimumGCAge: 2m
39 imageGCHighThresholdPercent: 85
40 imageGCLowThresholdPercent: 80
41 volumeStatsAggPeriod: 1m
42 kubeletCgroups: ""
43 systemCgroups: ""
44 cgroupRoot: ""
45 cgroupsPerQOS: true
46 cgroupDriver: cgroupfs
47 runtimeRequestTimeout: 10m
48 hairpinMode: promiscuous-bridge
49 maxPods: 220
50 podCIDR: "${CLUSTER_CIDR}"
51 podPidsLimit: -1
52 resolvConf: /etc/resolv.conf
53 maxOpenFiles: 1000000
54 kubeAPIQPS: 1000
55 kubeAPIBurst: 2000
56 serializeImagePulls: false
57 evictionHard:
58 memory.available: "100Mi"
59 nodefs.available: "10%"
60 nodefs.inodesFree: "5%"
61 imagefs.available: "15%"
62 evictionSoft: {}
63 enableControllerAttachDetach: true
64 failSwapOn: true
65 containerLogMaxSize: 20Mi
66 containerLogMaxFiles: 10
67 systemReserved: {}
68 kubeReserved: {}
69 systemReservedCgroup: ""
70 kubeReservedCgroup: ""
71 enforceNodeAllocatable: ["pods"]
72 EOF
1.7 分发kubelet 参数配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 sed -e "s/##ALL_IP##/${all_ip}/" kubelet-config.yaml.template > kubelet-config-${all_ip}.yaml.template
7 scp kubelet-config-${all_ip}.yaml.template root@${all_ip}:/etc/kubernetes/kubelet-config.yaml
8 done
1.8 创建kubelet systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kubelet.service.template <<EOF
4 [Unit]
5 Description=Kubernetes Kubelet
6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
7 After=docker.service
8 Requires=docker.service
9
10 [Service]
11 WorkingDirectory=${K8S_DIR}/kubelet
12 ExecStart=/opt/k8s/bin/kubelet \\
13 --allow-privileged=true \\
14 --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
15 --cert-dir=/etc/kubernetes/cert \\
16 --cni-conf-dir=/etc/cni/net.d \\
17 --container-runtime=docker \\
18 --container-runtime-endpoint=unix:///var/run/dockershim.sock \\
19 --root-dir=${K8S_DIR}/kubelet \\
20 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
21 --config=/etc/kubernetes/kubelet-config.yaml \\
22 --hostname-override=##ALL_NAME## \\
23 --pod-infra-container-image=registry.cn-beijing.aliyuncs.com/k8s_images/pause-amd64:3.1 \\
24 --image-pull-progress-deadline=15m \\
25 --volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\
26 --logtostderr=true \\
27 --v=2
28 Restart=always
29 RestartSec=5
30 StartLimitInterval=0
31
32 [Install]
33 WantedBy=multi-user.target
34 EOF
- 如果设置了 --hostname-override 选项,则 kube-proxy 也需要设置该选项,否则会出现找不到 Node 的情况;
- --bootstrap-kubeconfig:指向 bootstrap kubeconfig 文件,kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求;
- K8S approve kubelet 的 csr 请求后,在 --cert-dir 目录创建证书和私钥文件,然后写入 --kubeconfig 文件;
- --pod-infra-container-image 不使用 redhat 的 pod-infrastructure:latest 镜像,它不能回收容器的僵尸。
1.9 分发kubelet systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_name in ${ALL_NAMES[@]}
4 do
5 echo ">>> ${all_name}"
6 sed -e "s/##ALL_NAME##/${all_name}/" kubelet.service.template > kubelet-${all_name}.service
7 scp kubelet-${all_name}.service root@${all_name}:/etc/systemd/system/kubelet.service
8 done
二 启动验证
2.1 授权
1 [root@k8smaster01 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
2.2 启动kubelet
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for all_name in ${ALL_NAMES[@]}
3 do
4 echo ">>> ${all_name}"
5 ssh root@${all_name} "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
6 ssh root@${all_name} "/usr/sbin/swapoff -a"
7 ssh root@${all_name} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
8 done
2.3 查看kubelet服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for all_name in ${ALL_NAMES[@]}
3 do
4 echo ">>> ${all_name}"
5 ssh root@${all_name} "systemctl status kubelet"
6 done
7 [root@k8snode01 ~]# kubectl get csr
8 [root@k8snode01 ~]# kubectl get nodes
三 approve CSR 请求
3.1 自动 approve CSR 请求
1 [root@k8snode01 ~]# cd /opt/k8s/work
2 [root@k8snode01 work]# cat > csr-crb.yaml <<EOF
3 # Approve all CSRs for the group "system:bootstrappers"
4 kind: ClusterRoleBinding
5 apiVersion: rbac.authorization.k8s.io/v1
6 metadata:
7 name: auto-approve-csrs-for-group
8 subjects:
9 - kind: Group
10 name: system:bootstrappers
11 apiGroup: rbac.authorization.k8s.io
12 roleRef:
13 kind: ClusterRole
14 name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
15 apiGroup: rbac.authorization.k8s.io
16 ---
17 # To let a node of the group "system:nodes" renew its own credentials
18 kind: ClusterRoleBinding
19 apiVersion: rbac.authorization.k8s.io/v1
20 metadata:
21 name: node-client-cert-renewal
22 subjects:
23 - kind: Group
24 name: system:nodes
25 apiGroup: rbac.authorization.k8s.io
26 roleRef:
27 kind: ClusterRole
28 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
29 apiGroup: rbac.authorization.k8s.io
30 ---
31 # A ClusterRole which instructs the CSR approver to approve a node requesting a
32 # serving cert matching its client cert.
33 kind: ClusterRole
34 apiVersion: rbac.authorization.k8s.io/v1
35 metadata:
36 name: approve-node-server-renewal-csr
37 rules:
38 - apiGroups: ["certificates.k8s.io"]
39 resources: ["certificatesigningrequests/selfnodeserver"]
40 verbs: ["create"]
41 ---
42 # To let a node of the group "system:nodes" renew its own server credentials
43 kind: ClusterRoleBinding
44 apiVersion: rbac.authorization.k8s.io/v1
45 metadata:
46 name: node-server-cert-renewal
47 subjects:
48 - kind: Group
49 name: system:nodes
50 apiGroup: rbac.authorization.k8s.io
51 roleRef:
52 kind: ClusterRole
53 name: approve-node-server-renewal-csr
54 apiGroup: rbac.authorization.k8s.io
55 EOF
56 [root@k8snode01 work]# kubectl apply -f csr-crb.yaml
3.2 查看 kubelet 的情况
1 [root@k8snode01 ~]# kubectl get csr | grep boot #等待一段时间(1-10 分钟),三个节点的 CSR 都被自动 approved
2 [root@k8snode01 ~]# kubectl get nodes #所有节点均 ready
3 [root@k8snode01 ~]# ls -l /etc/kubernetes/kubelet.kubeconfig
4 [root@k8snode01 ~]# ls -l /etc/kubernetes/cert/|grep kubelet
3.3 手动 approve server cert csr
1 [root@k8smaster01 ~]# kubectl get csr
2 [root@k8smaster01 ~]# kubectl certificate approve csr-2kmtj
3
1 [root@k8smaster01 ~]# ls -l /etc/kubernetes/cert/kubelet-*
四 kubelet API 接口
4.1 kubelet 提供的 API 接口
1 [root@k8smaster01 ~]# sudo netstat -lnpt|grep kubelet #查看kubelet监听端口
- 10248: healthz http 服务;
- 10250: https 服务,访问该端口时需要认证和授权(即使访问 /healthz 也需要);
- 未开启只读端口 10255;
- 从 K8S v1.10 开始,去除了 --cadvisor-port 参数(默认 4194 端口),不支持访问 cAdvisor UI & API。
4.2 kubelet api 认证和授权
- authentication.anonymous.enabled:设置为 false,不允许匿名�访问 10250 端口;
- authentication.x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTPs 证书认证;
- authentication.webhook.enabled=true:开启 HTTPs bearer token 认证。
1 [root@k8smaster01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.24.8.71:10250/metrics
2 Unauthorized[root@k8smaster01 ~]#
3 [root@k8smaster01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer 123456" https://172.24.8.71:10250/metrics
4 Unauthorized
4.3 证书认证和授权
1 [root@k8smaster01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.24.8.71:10250/metrics #默认权限不足
2 Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)
3 curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.24.8.71:10250/metrics|head #使用最高权限的admin
4.4 创建bear token 认证和授权
1 [root@k8smaster01 ~]# kubectl create sa kubelet-api-test
2 [root@k8smaster01 ~]# kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
3 [root@k8smaster01 ~]# SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
4 [root@k8smaster01 ~]# TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
5 [root@k8smaster01 ~]# echo ${TOKEN}
1 [root@k8smaster01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.24.8.71:10250/metrics|head
4.5 cadvisor 和 metrics
015.Kubernetes二进制部署所有节点kubelet的更多相关文章
- 013.Kubernetes二进制部署worker节点Nginx实现高可用
一 Nginx代理实现kube-apiserver高可用 1.1 Nginx实现高可用 基于 nginx 代理的 kube-apiserver 高可用方案. 控制节点的 kube-controller ...
- 016.Kubernetes二进制部署所有节点kube-proxy
一 部署 kube-proxy kube-proxy 运行在所有节点上,它监听 apiserver 中 service 和 endpoint 的变化情况,创建路由规则以提供服务 IP 和负载均衡功能. ...
- 012.Kubernetes二进制部署worker节点Flannel
一 部署flannel 1.1 安装flannel kubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通.flannel 使用 vxlan 技术为各节点创建一 ...
- Kubernetes 二进制部署(一)单节点部署(Master 与 Node 同一机器)
0. 前言 最近受“新冠肺炎”疫情影响,在家等着,入职暂时延后,在家里办公和学习 尝试通过源码编译二进制的方式在单一节点(Master 与 Node 部署在同一个机器上)上部署一个 k8s 环境,整理 ...
- K8s二进制部署单节点 master组件 node组件 ——头悬梁
K8s二进制部署单节点 master组件 node组件 --头悬梁 1.master组件部署 2.node 组件部署 k8s集群搭建: etcd集群 flannel网络插件 搭建maste ...
- K8s二进制部署单节点 etcd集群,flannel网络配置 ——锥刺股
K8s 二进制部署单节点 master --锥刺股 k8s集群搭建: etcd集群 flannel网络插件 搭建master组件 搭建node组件 1.部署etcd集群 2.Flannel 网络 ...
- Kubernetes 二进制部署(二)集群部署(多 Master 节点通过 Nginx 负载均衡)
0. 前言 紧接上一篇,本篇文章我们尝试学习多节点部署 kubernetes 集群 并通过 haproxy+keepalived 实现 Master 节点的负载均衡 1. 实验环境 实验环境主要为 5 ...
- Kubernetes 二进制部署
目录 1.基础环境 2.部署DNS 3.准备自签证书 4.部署Docker环境 5.私有仓库Harbor部署 6.部署Master节点 6.1.部署Etcd集群 6.2.部署kube-apiserve ...
- k8s1.13.0二进制部署-node节点(四)
Master apiserver启用TLS认证后,Node节点kubelet组件想要加入集群,必须使用CA签发的有效证书才能与apiserver通信,当Node节点很多时,签署证书是一件很繁琐的事情, ...
随机推荐
- Qt5教程: (6) 菜单栏、工具栏、状态栏及核心控件
1. 新建QMainWindow工程 取消生成.ui文件 2. 菜单栏 在mainwindow.cpp中添加头文件 和 #include <QMenuBar> #include <Q ...
- C# 委托 (一)—— 委托、 泛型委托与Lambda表达式
C# 委托 (一)—— 委托. 泛型委托与Lambda表达式 2018年08月19日 20:46:47 wnvalentin 阅读数 2992 版权声明:此文乃博主之原创.鄙人才疏,望大侠斧正.此 ...
- Halcon一日一练:图像分割之阈值分割1
先了解什么是阈值,度娘告诉我的是:一个领域或一个系统的界限称为阈,其数值称为阈值.在图像中,我们把图像看成一个由像素灰度值组成的数集,那么阈,就是这个图像中,根据目标与背景灰度值的差异,选取的一个合适 ...
- 【Labview入门】将输入度数转换为3位精度弧度值
Labview版本2015 程序如下: 可以右键输出控件选择属性来调整输出的小数位数: 运行结果:
- opensuse下安装redis
1.安装gcc zypper install gcc 2.解压,安装到指定路径 sudo tar xzf redis-4.0.1.tar.gzcd redis#安装到指定目录中sudo make PR ...
- Go.js 没有中文文档 也没有中文demo 学起来很费劲 给大家整理一个算是详细的文档
<!DOCTYPE html> <html> <head> <meta name="viewport" content="wid ...
- xpath相关用法及技巧
本节讲解网页解析神器----XPath lxml下载 xpath基本用法 xpath插件 Xpath及XML路径语言,它是一门在XML文档查找信息的语言. 一:lxml下载以及安装 首先需要解决lxm ...
- 玩转OneNET物联网平台之MQTT服务③ —— 远程控制LED(设备自注册)
授人以鱼不如授人以渔,目的不是为了教会你具体项目开发,而是学会学习的能力.希望大家分享给你周边需要的朋友或者同学,说不定大神成长之路有博哥的奠基石... QQ技术互动交流群:ESP8266&3 ...
- StreamWriter 相关知识分享
在介绍StreamWriter之前,我们首先来了解一下它的父类TextWriter. 一.TextWriter 1.TextWriter的构造函数和常用属性方法 下面是TextWriter的构造函数: ...
- Vue系列---理解Vue.nextTick使用及源码分析(五)
_ 阅读目录 一. 什么是Vue.nextTick()? 二. Vue.nextTick()方法的应用场景有哪些? 2.1 更改数据后,进行节点DOM操作. 2.2 在created生命周期中进行DO ...