Kubernetes集群搭建（详细）

kubernetes集群搭建（kubeadm方式）

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。这个工具能通过两条指令完成一个kubernetes集群的部署：

# 创建一个 Master 节点
kubeadm init

# 将一个 Node 节点加入到当前集群中
kubeadm join <Master节点的IP和端口 >

• 安装要求
  • 一台或多台机器，操作系统 CentOS7.x-86_x64
  • 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
  • 可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
  • 禁止swap分区
• 准备环境

角色	IP
master	192.168.33.102
node1	192.168.33.103
node2	192.168.33.104

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld # 永久关闭

# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时

# 关闭swap
swapoff -a # 临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久关闭

# 设置主机名
hostnamectl set-hostname xxx

# 在master中添加hosts
vi /etc/hosts

# 允许 iptables 检查桥接流量
lsmod | grep br_netfilter # 查看 br_netfilter 是否被加载
modprobe br_netfilter # 显式加载

# 为了让Linux节点上的 iptables 能够正确地查看桥接流量，确保sysctl 配置中net.bridge.bridge-nf-call-iptables 设置为 1
cat >> /etc/modules-load.d/k8s.conf <<EOF
br_netfilter
EOF

cat >> /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1 # 开启iptables中ipv6的过滤规则
net.bridge.bridge-nf-call-iptables = 1 # 开启ipv4的过滤规则
net.ipv4.ip_forward = 1 # 开启服务器的路由转发功能
EOF

sysctl --system # 生效

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

• 所有节点安装docker：https://blog.csdn.net/weixin_37680513/article/details/116858377
• 所有节点安装kubeadm、kubelet、kubectl

# 添加阿里云yum源的镜像仓库
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 安装kubeadm、kubelet、kubectl
yum list kubeadm --showduplicates # 列出版本
yum install -y kubelet-1.21.1 kubeadm-1.21.1 kubectl-1.21.1 # 安装指定版本
systemctl start kubelet && systemctl enable kubelet

• 部署kubernetes 集群

# 在master节点（192.168.33.102）下载需要的镜像，下面是通用的下载脚本
cat ./pull.sh
for i in `kubeadm config images list`; do
	imageName=${i#k8s.gcr.io/}
	docker pull registry.aliyuncs.com/google_containers/$imageName
	docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
	docker rmi registry.aliyuncs.com/google_containers/$imageName
done;

kubeadm config images list # 查看需要下载的镜像
## 注意：有些镜像版本号带v，有的不带v，coredns这个镜像在k8s.gcr.io中是带v的，但是在registry.aliyuncs.com/google_containers上是不带v的，要手动去掉v后拉取，然后再改成带v的

# 执行kubeadm init
kubeadm init \
--apiserver-advertise-address=192.168.33.102 \
--kubernetes-version v1.21.1 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16

# 根据提示，执行以下命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 根据提示，在其他node节点执行以下命令加入集群
kubeadm join 192.168.33.102:6443 --token da65iu.8yuc73gfpeaoerl6 \
        --discovery-token-ca-cert-hash sha256:5849eded3719b7f52ef26ecef747cbf6d360c519fa5fa30f338de1a034ff6a38

# 有效期24小时，之后需要重新创建token
kubeadm token create --print-join-command

# 在master节点上查看
kubectl get nodes # 已经有master和node节点了，notReady

• 部署网络插件CNI（Container Network Interface）

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 无法访问就在网上找一下内容

# 之后在master查看node一直是notReady，kubectl get pod -n kube-system查看node上的proxy和flannel容器一直没有正常运行，

kubeclt describe pod xxx -n kube-system # 查看pod详细内容

# node节点上拉取镜像失败，手动在node上拉取相关镜像

# 拉取镜像
docker pull quay.io/coreos/flannel
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.21.1
docker pull registry.aliyuncs.com/google_containers/pause:3.4.1

# 修改镜像名字
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.21.1 k8s.gcr.io/kube-proxy:v1.21.1
docker tag registry.aliyuncs.com/google_containers/ k8s.gcr.io/pause:3.4.1 k8s.gcr.io/pause:3.4.1

• 测试集群

## 测试集群，在集群中创建一个pod，验证是否正常运行，web访问
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc

• 命令清单，命令补全：https://blog.csdn.net/weixin_37680513/article/details/116737492

错误

• 正常安装之后可能出现以下错误，是因为/etc/kubernetes/manifests下的kube-controller-manager.yaml和kube-scheduler.yaml设置的默认端口不正确，在文件中注释掉就可以了

=====================未整理=============================

03 k8s集群搭建（二进制方式）

## master 安装kube-apiserver，kube-controller-manager，kube-scheduler，etcd
## worker 安装kubelet，kube-proxy，docker，etcd

## 操作系统初始化，参考kubeadm安装

部署etcd集群

## 部署etcd集群（分布式kv存储系统，半数以上可用集群可用）

##4.1 准备cfssl证书生成工具，相比openssl更方便，任选一台机器执行
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

##4.2 生成etcd证书

## 自签证书颁发机构
# 创建工作目录
mkdir -p ~/TLS{etcd,k8s} # 分别为etcd和k8s准备证书
cd ~/TLS/etcd
# 自签CA，创建文件，添加内容
vi ca-config.json
{
    "signing":{
        "default":{
            "expiry":"87600h"
        },
        "profiles":{
            "www":{
                "expiry":"87600h",
                "usages":[
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

vi ca-csr.json
{
    "CN":"etcd CA",
    "key":{
        "algo":"rsa",
        "size":2048
    },
    "names":[
        {
            "C":"CN",
            "L":"Beijing",
            "ST":"Beijing"
        }
    ]
}
# 生成证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

## 使用自签CA签发etcd https证书
# 创建证书申请文件, 可用多写几个预留ip，是所有etcd节点内部通信ip
vi server-csr.json
{
    "CN":"etcd",
    "hosts":[
        "192.168.31.71",
        "192.168.31.72",
        "192.168.31.73"
    ],
    "key":{
        "algo":"rsa",
        "size":2048
    },
    "names":[
        {
            "C":"CN",
            "L":"BeiJing",
            "ST":"BeiJing"
        }
    ]
}
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

##4.3 从github下载二进制文件https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

##4.4 部署etcd集群，在node1上操作，为方便操作，将node1拷贝到node2

# 创建工作目录，解压，移动
mkdir /opt/etcd/{bin,cfg,ssl} –p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

# 创建etcd配置文件
vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 节点名称
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" # 数据目录
ETCD_LISTEN_PEER_URLS="https://192.168.31.71:2380" # 集群通信监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.31.71:2379" # 客户端访问监听地址
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.71:2380" # 集群通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.71:2379" # 客户端通告地址
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.71:2380,etcd-2=https://192.168.31.72:2380,etcd-3=https://192.168.31.73:2380" # 集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" # 集群Token
ETCD_INITIAL_CLUSTER_STATE="new" # 加入集群的当前状态，new 是新集群，existing 表示加入已有集群

# systemd 管理etcd
vi /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

# 拷贝生成的证书到配置文件中的路径
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

# 启动并设置开机启动
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

# 将上面生成的文件拷贝到其他节点，并在其他节点上修改etcd.conf中的名称和当前服务器ip，然后启动并开机启动
scp -r /opt/etcd/ root@192.168.31.72:/opt/

# 查看集群状态
systemctl status etcd
scp /usr/lib/systemd/system/etcd.service
root@192.168.31.72:/usr/lib/systemd/system/

安装docker

## 安装docker，所有节点都要安装
## 下载地址：https://download.docker.com/linux/static/stable/x86_64/docker-
19.03.9.tgz（二进制安装）
# 解压，移动
tar zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

# systemd 管理docker
vi /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target

# 创建配置文件，配置aliyun镜像加速器
vi /etc/docker/daemon.json
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}

# 启动并设置开机启动
systemctl daemon-reload
systemctl start docker
systemctl enable docker

部署master node

## 部署master node
##6.1 生成kube-apiserver 证书
# 自签证书颁发机构（CA）
vi ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}

vi ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 生成证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
ca-key.pem ca.pem

# 使用自签CA 签发kube-apiserver HTTPS 证书
vi TLS/k8s/server-csr.json #1 创建证书申请文件
{
  "CN": "kubernetes",
  "hosts": [
    "10.0.0.1",
    "127.0.0.1",
    "192.168.31.71",
    "192.168.31.72",
    "192.168.31.73",
    "192.168.31.74",
    "192.168.31.81",
    "192.168.31.82",
    "192.168.31.88",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -
profile=kubernetes server-csr.json | cfssljson -bare server #2 生成证书

##6.2 从Github 下载二进制文件，https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-
1.18.md#v1183 # 链接里有很多包，下一个server就够了，包含master和worker

##6.3 解压，移动
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

##6.4 部署kube-apiserver
# 创建配置文件
vi /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \\ # 启用日志
--v=2 \\ # 日志等级
--log-dir=/opt/kubernetes/logs \\ # 日志目录
--etcdservers=
https://192.168.31.71:2379,https://192.168.31.72:2379,https://192.168.3 # etce集群地址
1.73:2379 \\
--bind-address=192.168.31.71 \\ # 监听地址
--secure-port=6443 \\ # 安全端口
--advertise-address=192.168.31.71 \\ # 集群通告地址
--allow-privileged=true \\ # 启用授权
--service-cluster-ip-range=10.0.0.0/24 \\ # Service 虚拟IP 地址段
--enable-admissionplugins=
NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestric
tion \ \# 准入控制模块
--authorization-mode=RBAC,Node \\ # 认证授权，启用RBAC 授权和节点自管理
--enable-bootstrap-token-auth=true \\ # 启用TLS bootstrap 机制
--token-auth-file=/opt/kubernetes/cfg/token.csv \\ # bootstrap token 文件
--service-node-port-range=30000-32767 \\ # Service nodeport 类型默认分配端口范围
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ # apiserver 访问kubelet 客户端证书
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\ # apiserver https 证书
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\ # 连接Etcd 集群证书
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\ # 审计日志
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
# 拷贝刚才生成的证书到配置文件的路径
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernates/ssl/
# 启用TLS Bootstrapping 机制
vi /opt/kubernetes/cfg/token.csv # 创建token文件，格式：token，用户名，UID，用户组，token 也可自行生成替换：head -c 16 /dev/urandom | od -An -t x | tr -d ' '
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:nodebootstrapper"
# systemd 管理apiserver
vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target#
# 启动并设置开机启动
systemctl daemon-reload
systemctl start kube-apiserver
# 授权kubelet-bootstrap 用户允许请求证书
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
systemctl enable kube-apiserver

##6.5 部署kube-controller-manager

vi /opt/kubernetes/cfg/kube-controller-manager.conf # 创建配置文件
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"

vi /usr/lib/systemd/system/kube-controller-manager.service #systemd 管理controller-manager
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager
\$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

systemctl daemon-reload # 启动并设置开机启动
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

##6.6 部署kube-scheduler

vi /opt/kubernetes/cfg/kube-scheduler.conf # 创建配置文件
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"

vi /usr/lib/systemd/system/kube-scheduler.service # systemd 管理scheduler
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

systemctl daemon-reload # 启动并设置开机启动
systemctl start kube-scheduler
systemctl enable kube-scheduler

kubectl get cs # 查看集群状态

部署worker node

##7.1 创建工作目录并拷贝二进制文件
##7.2 部署kubelet
##7.3 批准kubelet 证书申请并加入集群
##7.4 部署kube-proxy
##7.5 部署CNI 网络
##7.6 授权apiserver 访问kubelet
##7.7 新增加Worker Node