离线方式安装高可用RKE2 (版本: v1.22.13+rke2r1)记录
说明：
- 想要了解RKE2可以到官网(https://docs.rke2.io 或 https://docs.rancher.cn/docs/rke2/_index/)看最新资料
- 用官网给出的离线安装(https://docs.rke2.io/install/airgap/)方式可能有坑, 节点拉起时安装网络插件必须使用到网络,没有网络节点一直时Notready的状态, 有可能也和我部署的版本有关系, 大家也可以跳过给containerd设置的步骤,直接部署试试
rke2-images.linux-amd64.tar.zst,rke2.linux-amd64.tar.gz,sha256sum-amd64.txt下载地址见 https://github.com/rancher/rke2/releases- 如果发现有啥错误,欢迎评论区打扰
初始化节点
配置主机名信息，添加本地hosts解析
hostnamectl set-hostname rke2-master-default-loadblance
hostnamectl set-hostname rke2-master-default-nodepool-1
hostnamectl set-hostname rke2-master-default-nodepool-2
hostnamectl set-hostname rke2-master-default-nodepool-3
hostnamectl set-hostname rke2-node-default-nodepool-1
hostnamectl set-hostname rke2-node-default-nodepool-2
hostnamectl set-hostname rke2-node-default-nodepool-3
hostnamectl set-hostname rke2-node-default-nodepool-4
修改所有节点hosts
172.17.0.50 rke2-master-default-loadblance
172.17.0.51 rke2-master-default-nodepool-1
172.17.0.52 rke2-master-default-nodepool-2
172.17.0.53 rke2-master-default-nodepool-3
172.17.0.54 rke2-node-default-nodepool-1
172.17.0.55 rke2-node-default-nodepool-2
172.17.0.56 rke2-node-default-nodepool-3
172.17.0.57 rke2-node-default-nodepool-4
重新生成机器ID并重启
cp -rf /dev/null /etc/machine-id
systemd-machine-id-setup
reboot
所有节点安装需要的包
systemctl stop firewalld; systemctl disable firewalld; systemctl stop dnsmasq; systemctl disable dnsmasq; systemctl stop ntpd; systemctl disable ntpd; systemctl stop postfix; systemctl disable postfix;
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
swapoff -a; sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab;
setenforce 0
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
yum -y install epel-release; yum -y install chrony curl wget vim sysstat net-tools openssl openssh lsof socat nfs-utils conntrack ipvsadm ipset iptables sysstat libseccomp; systemctl disable rpcbind
调整limit
cat >> /etc/security/limits.conf << EOF
* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535
EOF
调整NetworkManager
cat >> /etc/NetworkManager/conf.d << EOF
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
EOF
升级内核
curl -LO https://storage.corpintra.plus/elrepo/kernel/kernel-lt-5.4.134-1.el7.elrepo.x86_64.rpm
curl -LO https://storage.corpintra.plus/elrepo/kernel/kernel-lt-devel-5.4.134-1.el7.elrepo.x86_64.rpm
yum install kernel-lt-* -y
grub2-set-default 0
grub2-mkconfig -o /etc/grub2.cfg
grubby --default-kernel
reboot
导入ipvs模块
cat > /etc/modules-load.d/ipvs.conf << EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
nf_conntrack
br_netfilter
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
overlay
EOF
systemctl enable --now systemd-modules-load.service
修改内核并重启所有节点
cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.netfilter.nf_conntrack_max = 2310720
fs.inotify.max_user_watches=89100
fs.file-max = 52706963
fs.nr_open = 52706963
vm.swappiness = 0
vm.overcommit_memory=1
kernel.panic=10
EOF
reboot
Haproxy节点配置
yum install haproxy -y
写入以下配置到/etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 6000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
log global
option tcplog
option dontlognull
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats
bind 0.0.0.0:9000
mode http
stats uri /status
stats refresh 30s
stats realm "Haproxy Manager"
stats auth admin:admin
stats hide-version
stats admin if TRUE
frontend rke2-server
bind *:9345
mode tcp
default_backend rke2-server
backend rke2-server
balance roundrobin
mode tcp
server rke2-master-default-nodepool-1 172.17.0.51:9345 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server rke2-master-default-nodepool-2 172.17.0.52:9345 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server rke2-master-default-nodepool-3 172.17.0.53:9345 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
frontend rke2-apiserver
bind *:6443
mode tcp
default_backend rke2-apiserver
backend rke2-apiserver
balance roundrobin
mode tcp
server rke2-master-default-nodepool-1 172.17.0.51:6443 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server rke2-master-default-nodepool-2 172.17.0.52:6443 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server rke2-master-default-nodepool-3 172.17.0.53:6443 check weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
启动haproxy
systemctl enable --now haproxy.service
RKE2-Server初始化
在所有Master节点上执行资源下载以及安装
groupadd -r etcd
useradd -r -g etcd -s /bin/false -d /var/lib/etcd etcd
mkdir -p /etc/rke2/config
curl -L https://storage.corpintra.plus/rke2/audit-policy.yaml -o /etc/rke2/config/audit-policy.yaml
cat > /etc/sysconfig/rke2-server <<EOF
CONTAINERD_HTTP_PROXY=http://192.168.1.10:3128
CONTAINERD_HTTPS_PROXY=http://192.168.1.10:3128
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
EOF
mkdir /root/rke2-artifacts
cd /root/rke2-artifacts/
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/rke2-images.linux-amd64.tar.zst
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/rke2.linux-amd64.tar.gz
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/sha256sum-amd64.txt
curl -sfL https://storage.corpintra.plus/rke2/install.sh | INSTALL_RKE2_TYPE=server INSTALL_RKE2_MIRROR=cn INSTALL_RKE2_ARTIFACT_PATH=/root/rke2-artifacts INSTALL_RKE2_VERSION=v1.22.13+rke2r1 sh -
配置第一台Master节点
常用的参数见: https://docs.rke2.io/install/install_options/server_config/
网络选择: https://docs.rke2.io/install/network_options/
如果你想禁用一些RKE2自带的Chart: https://docs.rke2.io/advanced/#disabling-server-charts
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<EOF
write-kubeconfig-mode: "0644"
#server: https://172.17.0.50:9345 #等三台都起来后把这个配置取消注释，重启下rke2-server
tls-san:
- "127.0.0.1"
- "172.17.0.50"
- "172.17.0.51"
- "172.17.0.52"
- "172.17.0.53"
- "rke2-master-default-loadblance"
- "rke2-master-default-nodepool-1"
- "rke2-master-default-nodepool-2"
- "rke2-master-default-nodepool-3"
cni: "calico"
profile: "cis-1.6"
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
disable-cloud-controller: true
kube-proxy-arg:
- "proxy-mode=ipvs"
kubelet-arg:
- "max-pods=110"
node-taint:
- "node-role.kubernetes.io/control-plane=true:NoSchedule"
audit-policy-file: "/etc/rke2/config/audit-policy.yaml"
etcd-snapshot-schedule-cron: "0 */4 * * *"
etcd-snapshot-retention: "84"
#disable:
#- "rke2-ingress-nginx"
#- "rke2-metrics-server"
EOF
启动第一台节点(需要耐心等一等,很慢，如果遇到启动失败的，重启一次，一般重启一次能解决 )
systemctl enable --now rke2-server.service
配置第二台Master节点
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<EOF
write-kubeconfig-mode: "0644"
token: <token for server node> ##填写第一个server节点的token，通过在第一个master节点查看/var/lib/rancher/rke2/server/token文件获得；
server: https://172.17.0.50:9345
tls-san:
- "127.0.0.1"
- "172.17.0.50"
- "172.17.0.51"
- "172.17.0.52"
- "172.17.0.53"
- "rke2-master-default-loadblance"
- "rke2-master-default-nodepool-1"
- "rke2-master-default-nodepool-2"
- "rke2-master-default-nodepool-3"
cni: "calico"
profile: "cis-1.6"
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
disable-cloud-controller: true
kube-proxy-arg:
- "proxy-mode=ipvs"
kubelet-arg:
- "max-pods=110"
node-taint:
- "node-role.kubernetes.io/control-plane=true:NoSchedule"
audit-policy-file: "/etc/rke2/config/audit-policy.yaml"
etcd-snapshot-schedule-cron: "0 */4 * * *"
etcd-snapshot-retention: "84"
etcd-snapshot-dir: "/var/lib/etcd-snapshots"
#disable:
#- "rke2-ingress-nginx"
#- "rke2-metrics-server"
EOF
启动第二台节点, 比较慢，耐心等等
systemctl enable --now rke2-server.service
配置第三台Master节点
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<EOF
write-kubeconfig-mode: "0644"
token: <token for server node> ##填写第一个server节点的token，通过在第一个master节点查看/var/lib/rancher/rke2/server/token文件获得；
server: https://172.17.0.50:9345
tls-san:
- "127.0.0.1"
- "172.17.0.50"
- "172.17.0.51"
- "172.17.0.52"
- "172.17.0.53"
- "rke2-master-default-loadblance"
- "rke2-master-default-nodepool-1"
- "rke2-master-default-nodepool-2"
- "rke2-master-default-nodepool-3"
cni: "calico"
profile: "cis-1.6"
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
disable-cloud-controller: true
kube-proxy-arg:
- "proxy-mode=ipvs"
kubelet-arg:
- "max-pods=110"
node-taint:
- "node-role.kubernetes.io/control-plane=true:NoSchedule"
audit-policy-file: "/etc/rke2/config/audit-policy.yaml"
etcd-snapshot-schedule-cron: "0 */4 * * *"
etcd-snapshot-retention: "84"
etcd-snapshot-dir: "/var/lib/etcd-snapshots"
#disable:
#- "rke2-ingress-nginx"
#- "rke2-metrics-server"
EOF
启动第三台节点, 比较慢，耐心等等
systemctl enable --now rke2-server.service
等所有Master节点起来后执行下面命令检查
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes -o wide
RKE2-Agent节点安装
挨台安装Agent节点
cat > /etc/sysconfig/rke2-agent <<EOF
CONTAINERD_HTTP_PROXY=http://192.168.1.10:3128
CONTAINERD_HTTPS_PROXY=http://192.168.1.10:3128
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
EOF
mkdir /root/rke2-artifacts
cd /root/rke2-artifacts/
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/rke2-images.linux-amd64.tar.zst
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/rke2.linux-amd64.tar.gz
curl -LO https://storage.corpintra.plus/rke2/v1.22.13+rke2r1/sha256sum-amd64.txt
curl -sfL https://storage.corpintra.plus/rke2/install.sh | INSTALL_RKE2_TYPE=agent INSTALL_RKE2_MIRROR=cn INSTALL_RKE2_ARTIFACT_PATH=/root/rke2-artifacts INSTALL_RKE2_VERSION=v1.22.13+rke2r1 sh -
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<EOF
server: https://172.17.0.50:9345
token: <token for server node> ##填写第一个server节点的token，通过在第一个master节点查看/var/lib/rancher/rke2/server/token文件获得；
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
kube-proxy-arg:
- "proxy-mode=ipvs"
kubelet-arg:
- "max-pods=110"
EOF
systemctl enable rke2-agent.service --now
等所有Node节点起来后执行下面命令检查
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes -o wide
RKE2默认使用containerd作为Runtime,如果想要查看节点上运行的容器,可以使用下面的命令:
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml && /var/lib/rancher/rke2/bin/crictl ps
安装存储插件(可选)
curl -LO https://storage.corpintra.plus/kubernetes/charts/csi-driver-smb-v1.9.0.tgz
helm upgrade csi-driver-smb \
--namespace kube-system \
--create-namespace \
--debug \
--wait \
--install \
--atomic \
--set image.baseRepo="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage" \
--set image.smb.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/smbplugin" \
--set image.smb.tag="v1.9.0" \
--set image.csiProvisioner.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/csi-provisioner" \
--set image.csiProvisioner.tag="v3.2.0" \
--set image.livenessProbe.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/livenessprobe" \
--set image.livenessProbe.tag="v2.7.0" \
--set image.nodeDriverRegistrar.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/csi-node-driver-registrar" \
--set image.nodeDriverRegistrar.tag="v2.5.1" \
--set controller.replicas=2 \
./csi-driver-smb-v1.9.0.tgz
curl -LO https://storage.corpintra.plus/kubernetes/charts/csi-driver-nfs-v4.1.0.tgz
helm upgrade csi-driver-nfs \
--namespace kube-system \
--create-namespace \
--debug \
--wait \
--install \
--atomic \
--set image.nfs.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/nfsplugin" \
--set image.nfs.tag="v4.1.0" \
--set image.csiProvisioner.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/csi-provisioner" \
--set image.csiProvisioner.tag="v3.2.0" \
--set image.livenessProbe.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/livenessprobe" \
--set image.livenessProbe.tag="v2.7.0" \
--set image.nodeDriverRegistrar.repository="registry.cn-hangzhou.aliyuncs.com/kube-sig-storage/csi-node-driver-registrar" \
--set image.nodeDriverRegistrar.tag="v2.5.1" \
--set controller.replicas=2 \
./csi-driver-nfs-v4.1.0.tgz
离线方式安装高可用RKE2 (版本: v1.22.13+rke2r1)记录的更多相关文章
- 企业运维实践-还不会部署高可用的kubernetes集群?使用kubeadm方式安装高可用k8s集群v1.23.7
关注「WeiyiGeek」公众号 设为「特别关注」每天带你玩转网络安全运维.应用开发.物联网IOT学习! 希望各位看友[关注.点赞.评论.收藏.投币],助力每一个梦想. 文章目录: 0x00 前言简述 ...
- kubernetes实战(二十五)：kubeadm 安装 高可用 k8s v1.13.x
1.系统环境 使用kubeadm安装高可用k8s v.13.x较为简单,相比以往的版本省去了很多步骤. kubeadm安装高可用k8s v.11 和 v1.12 点我 主机信息 主机名 IP地址 说明 ...
- 容器云平台No.2~kubeadm创建高可用集群v1.19.1
通过kubernetes构建容器云平台第二篇,最近刚好官方发布了V1.19.0,本文就以最新版来介绍通过kubeadm安装高可用的kubernetes集群. 市面上安装k8s的工具很多,但是用于学习的 ...
- 通过 Kubeadm 安装 K8S 与高可用，版本1.13.4
环境介绍: CentOS: 7.6 Docker: 18.06.1-ce Kubernetes: 1.13.4 Kuberadm: 1.13.4 Kuberlet: 1.13.4 Kuberctl: ...
- kubernetes实战(二十六)：kubeadm 安装 高可用 k8s v1.16.x dashboard 2.x
1.基本配置 基本配置.内核升级.基本服务安装参考https://www.cnblogs.com/dukuan/p/10278637.html,或者参考<再也不踩坑的Kubernetes实战指南 ...
- centos7.4安装高可用(haproxy+keepalived实现)kubernetes1.6.0集群（开启TLS认证）
目录 目录 前言 集群详情 环境说明 安装前准备 提醒 一.创建TLS证书和秘钥 安装CFSSL 创建 CA (Certificate Authority) 创建 CA 配置文件 创建 CA 证书签名 ...
- Kubeadm搭建高可用(k8s)Kubernetes v1.24.0集群
文章转载自:https://i4t.com/5451.html 背景 Kubernetes 1.24新特性 从kubelet中移除dockershim,自1.20版本被弃用之后,dockershim组 ...
- 分布式架构高可用架构篇_03-redis3集群的安装高可用测试
参考文档 Redis 官方集群指南:http://redis.io/topics/cluster-tutorial Redis 官方集群规范:http://redis.io/topics/cluste ...
- Zabbix系列-REHL6.10离线方式安装Zabbix 4.0 LTS
环境 第零步:关闭系统默认防火墙 setenforce 0 sed -i -r "/^SELINUX=/c SELINUX=disabled" /etc/selinux/confi ...
随机推荐
- 教你用VS code 生成vue-cli代码片段
可以自定义设置名字:name.json { "Print to console": { "prefix": "vue", "bod ...
- openssl客户端编程：一个不起眼的函数导致的SSL会话失败问题
我们目前大部分使用的openssl库还是基于TLS1.2协议的1.0.2版本系列,如果要支持更高的TLS1.3协议,就必须使用openssl的1.1.1版本或3.0版本.升级openssl库有可能会导 ...
- python小题目练习（13）
题目:封装用户的上网行为 实现代码: """Author:mllContent:封装用户的上网行为Date:2020-01-19"""def ...
- 无语怎么这么简单！——python实现中文字幕雨+源代码（源码可赠）
大家好鸭,我是小熊猫 最近浏览了很多关于用Python和Pygame实现代码雨的案例,发现很多都是没有深入讲解代码的整个实现过程,从0到1教会你制作中文文字雨(其实啥字好像都可以). 然后在介绍的过程 ...
- centos7解决无法上网的问题
问题:centos7出现无法进行联网,如下图所示,执行该命令: ping qq.com 出现如下情况: 解决方法: 首先cd到需要修改文件的所在目录下: [root@localhost ~]# cd ...
- Docker安装canal、mysql进行简单测试与实现redis和mysql缓存一致性
一.简介 canal [kə'næl],译意为水道/管道/沟渠,主要用途是基于 MySQL 数据库增量日志解析,提供增量数据订阅和消费. 早期阿里巴巴因为杭州和美国双机房部署,存在跨机房同步的业务需求 ...
- java -jar -Xbootclasspath/a:/xxx/config xxx .jar 和 java -jar xxx .jar 的区别
1.如果有用Xbootclasspath的话则config的文件会直接覆盖jar里面的resouces文件,覆盖application.yml ,也会覆盖logback-spring.xml ,比如j ...
- Windows 下如何调试 PowerShell
背景 最近在用 PowerShell 的时候,发现一些地方特别有意思.于是就萌生了看源代码的想法,单看肯定不过瘾,调试起来才有意思.于是就有了这个,记录下. 调试 PowerShell 主要分为两种方 ...
- 【跟着大佬学JavaScript】之数组去重（结果对比）
前言 数组去重在面试和工作中都是比较容易见到的问题. 这篇文章主要是来测试多个方法,对下面这个数组的去重结果进行分析讨论.如果有不对的地方,还请大家指出. const arr = [ 1, 1, &q ...
- 西文字符与中文GBK编码的区别
一般来讲二者读取的时候西文字符的数值是正,而中文字符的数值是负的,此时读取的是中文字符的前一半,需要再读取一个char类型的数据,在大多数运行环境下这个规则都是用. ps:转自算法竞赛的笔记,要注意在 ...