kubeadm 搭建 K8s

本篇主要记录一下使用 kubeadm 搭建 k8s 详细过程，环境使用 VirtualBox 构建的3台虚拟机

1.环境准备

操作系统：Centos7 （CentOS Linux release 7.9.2009）
Master 主节点： 1台虚拟机
Node 计算节点： 2台虚拟机
K8s version：v1.23.5 （选的较新的版本）
Docker version：20.10.14

虚拟机创建我就不详细展开了

2.虚拟机网络配置(保证能访问外网)

虚拟机网卡选择的是 1.网络地址转换NAT （主要用来访问外网的） 2.仅主机网络（主要用来宿主机访问虚拟机的）

把对应的 ip 设置如下

kube-master 192.168.56.20

kube-node-1 192.168.56.21

kube-node-2 192.168.56.22

2.1 3台虚拟机静态ip配置

cat /etc/sysconfig/network-scripts/ifcfg-enp0s3

TYPE="Ethernet"

PROXY_METHOD="none"

BROWSER_ONLY="no"

BOOTPROTO="none" #关闭

DEFROUTE="yes"

IPV4_FAILURE_FATAL="no"

IPV6INIT="yes"

IPV6_AUTOCONF="yes"

IPV6_DEFROUTE="yes"

IPV6_FAILURE_FATAL="no"

IPV6_ADDR_GEN_MODE="stable-privacy"

NAME="enp0s3"

UUID="35eac2d4-234d-4859-848e-ed2a30a64a81"

DEVICE="enp0s3"

#添加部分

IPADDR="10.0.2.20"

PREFIX="24"

GATEWAY="10.0.2.2"

DNS1="10.0.2.3"

DNS2="114.114.114.114"

ONBOOT="yes"

cat /etc/sysconfig/network-scripts/ifcfg-enp0s8

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=static #修改为 static 静态ip

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=enp0s8

UUID=d06bac6d-294f-4bfe-bf08-5cb65c0a431f

DEVICE=enp0s8

#新增部分

IPADDR=192.168.56.20

PREFIX=24

ONBOOT=yes

2.2 虚拟机hosts 配置

在 3台虚拟机 /etc/hosts 中配置如下

192.168.56.20	kube-master

192.168.56.21	kube-node-1

192.168.56.22	kube-node-2

3.k8s 基础环境设置

3.1 yum 换源(可选)

yum -y install wget

cd /etc/yum.repos.d

mv CentOS-Base.repo CentOS-Base.repo.bak

wget -O CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

yum clean all

yum makecache #重新生成缓存

3.2 关闭 selinux

setenforce 0 #临时关闭

sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config #永久关闭

3.3 关闭swap分区或禁用swap文件

swapoff -a

# 注释掉关于swap分区的行

yes | cp /etc/fstab /etc/fstab_bak

cat /etc/fstab_bak |grep -v swap > /etc/fstab

3.4 允许iptables 检查桥接流量

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

br_netfilter

EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

sudo sysctl --system

3.5 关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

3.6 /proc/sys/net/ipv4/ip_forward

echo 1 > /proc/sys/net/ipv4/ip_forward

4.kubectl、kubeadm、kubelet的安装

注意先换源 https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.64be1b11iddYWH 阿里换源文档

4.1 添加Kubernetes的alibaba yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

yum install -y kubelet kubeadm kubectl

4.2 安装 kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl

systemctl enable kubelet && systemctl start kubelet

由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y --nogpgcheck kubelet kubeadm kubectl 安装

4.3 启动kubelet 服务

systemctl enable kubelet && systemctl start kubelet

此时执行systemctl status kubelet查看服务状态，此时肯定有错误（因为我们还未进行初始化操作），错误可使用journalctl -xe查看错误信息。

5.Docker 安装和配置

Docker centos 安装文档地址 https://docs.docker.com/engine/install/centos/

5.1 Docker 安装

sudo yum install -y yum-utils

sudo yum-config-manager \

    --add-repo \

    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce docker-ce-cli containerd.io

# 也可以安装其他版本 yum list docker-ce --showduplicates | sort -r

# sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io

#启动docker

sudo systemctl enable docker && systemctl start docker

5.2 Docker 镜像源更换

sudo tee /etc/docker/daemon.json <<-'EOF'

{

  "registry-mirrors": ["https://9xxxx1clo.mirror.aliyuncs.com"] #请更换自己的 阿里云镜像地址

}

EOF

5.3 Docker 配置cgroup-driver

由于 cgroup-driver ，默认docker 是 cgroupfs ，而k8s 默认是 systemd，所以需要设置 docker 的 cgroup 为 systemd

推荐使用方式二进行修改

#查看cgroup-driver

 docker info | grep -i cgroup

# 方式一 追加 --exec-opt native.cgroupdriver=systemd 参数

 sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

 systemctl daemon-reload # 重新加载服务

 systemctl enable docker # 启用docker服务(开机自起)

 systemctl restart docker # 启动docker服务

# 方式二 修改docker配置文件

 vim /etc/docker/daemon.json

{

  "exec-opts": ["native.cgroupdriver=systemd"]

}

5.4 拉取k8s所需镜像

可以通过

kubeadm config images list 查看所需的镜像

kubeadm config images list 

k8s.gcr.io/kube-apiserver:v1.23.5

k8s.gcr.io/kube-controller-manager:v1.23.5

k8s.gcr.io/kube-scheduler:v1.23.5

k8s.gcr.io/kube-proxy:v1.23.5

k8s.gcr.io/pause:3.6

k8s.gcr.io/etcd:3.5.1-0

k8s.gcr.io/coredns/coredns:v1.8.6

如果拉取不到镜像可以通过去 docker hub 上查找对应的镜像

docker pull k8simage/kube-apiserver:v1.23.5 

docker tag k8simage/kube-apiserver:v1.23.5 k8s.gcr.io/kube-apiserver:v1.23.5 #重新tag改成上面所需的镜像即可， 其他的都按照这样来

Tips: 完成上面操作后你可以通过 docker save 导出镜像,以后再用的时候可以通过 docker image load < 进行离线加载镜像

#导出镜像 ，批量导出所有镜像

docker save $(docker images --format '{{.Repository}}:{{.Tag}}') -o k8s.images.tar

#导入镜像 ，注意 是 < 不是 > 。。 > 后会导致tar 无法使用需要重新上传

docker image load < ...tar/zip

6.kube-master 节点初始化

我这里选的是 flannel 网络插件，先来看看 flannel 的配置文件

6.1 查看flannel 网络范围

flannel.yml https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

不同的网络插件网络范围不同

6.2 kubeadm init 初始化

kubeadm init --pod-network-cidr=10.244.0.0/16 \ #网卡插件的 网络范围  --apiserver-advertise-address=192.168.56.20 #master 的 ip 地址

等一会就会有如下结果

可以把 kubeadm join 命令记下来便于后面使用保存输出中的kubeadm join部分内容，用于添加node节点，或者使用kubeadm token list 和kubeadm token create --print-join-command查看

执行后续步骤，可以看到提示

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

6.3 网络插件flannel 配置

前面提到了flannel 把对应的flannel.yml文件下载到服务器中，可以使用wget

可以看到在没配置 flannel 的时候 coredns 一直是 pending 状态

wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml #可能会下载不到，文章最后我会提供内容，直接copy过去即可

#Tips: 可能需要指定一下网卡 ，在kube-flannel.yml中 - /opt/bin/flanneld args: 后面添加

#- --iface=enp0s8 # 需要指定网卡，否则可能k8s会导致使用错误的网卡 

kubectl apply -f kube-flannel.yml #应用该文件

kubectl get pods --all-namespaces -o wide #查看是否安装成功，它会在 master 和 node 节点上都会自动部署

最后通过 kubectl get nodes -o wide 查看节点

7.kube-node 节点初始化

7.1 查看token

可以通过该命令进行查看 token

kubeadm token create --print-join-command

#kubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7

7.2 node节点上执行 kubeadmin join

kubeadm join kube-master:6443 --token ii5xyw.mywlktar1e8f60cu --discovery-token-ca-cert-hash sha256:358f92418178342f79aa7b049e6c18a423f0fb88186189142459016fe6efb8e7

7.3 在master 上查看结果

watch kubectl get nodes -o wide #可以通过watch 实时监听 加入情况

至此使用 kubeadm 已经成功搭建k8s 集群

总结

本篇主要讲解了如何使用 kubeadm 进行搭建 k8s 集群，搭建的是 k8s 较新版本 v1.23.5 版本，使用kubeadm 搭建相对来说还是比较简单的。

flannel.yml 文件

---

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

  name: psp.flannel.unprivileged

  annotations:

    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default

    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default

    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default

    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default

spec:

  privileged: false

  volumes:

  - configMap

  - secret

  - emptyDir

  - hostPath

  allowedHostPaths:

  - pathPrefix: "/etc/cni/net.d"

  - pathPrefix: "/etc/kube-flannel"

  - pathPrefix: "/run/flannel"

  readOnlyRootFilesystem: false

  # Users and groups

  runAsUser:

    rule: RunAsAny

  supplementalGroups:

    rule: RunAsAny

  fsGroup:

    rule: RunAsAny

  # Privilege Escalation

  allowPrivilegeEscalation: false

  defaultAllowPrivilegeEscalation: false

  # Capabilities

  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']

  defaultAddCapabilities: []

  requiredDropCapabilities: []

  # Host namespaces

  hostPID: false

  hostIPC: false

  hostNetwork: true

  hostPorts:

  - min: 0

    max: 65535

  # SELinux

  seLinux:

    # SELinux is unused in CaaSP

    rule: 'RunAsAny'

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: flannel

rules:

- apiGroups: ['extensions']

  resources: ['podsecuritypolicies']

  verbs: ['use']

  resourceNames: ['psp.flannel.unprivileged']

- apiGroups:

  - ""

  resources:

  - pods

  verbs:

  - get

- apiGroups:

  - ""

  resources:

  - nodes

  verbs:

  - list

  - watch

- apiGroups:

  - ""

  resources:

  - nodes/status

  verbs:

  - patch

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: flannel

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: flannel

subjects:

- kind: ServiceAccount

  name: flannel

  namespace: kube-system

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: flannel

  namespace: kube-system

---

kind: ConfigMap

apiVersion: v1

metadata:

  name: kube-flannel-cfg

  namespace: kube-system

  labels:

    tier: node

    app: flannel

data:

  cni-conf.json: |

    {

      "name": "cbr0",

      "cniVersion": "0.3.1",

      "plugins": [

        {

          "type": "flannel",

          "delegate": {

            "hairpinMode": true,

            "isDefaultGateway": true

          }

        },

        {

          "type": "portmap",

          "capabilities": {

            "portMappings": true

          }

        }

      ]

    }

  net-conf.json: |

    {

      "Network": "10.244.0.0/16",

      "Backend": {

        "Type": "vxlan"

      }

    }

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: kube-flannel-ds

  namespace: kube-system

  labels:

    tier: node

    app: flannel

spec:

  selector:

    matchLabels:

      app: flannel

  template:

    metadata:

      labels:

        tier: node

        app: flannel

    spec:

      affinity:

        nodeAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            nodeSelectorTerms:

            - matchExpressions:

              - key: kubernetes.io/os

                operator: In

                values:

                - linux

      hostNetwork: true

      priorityClassName: system-node-critical

      tolerations:

      - operator: Exists

        effect: NoSchedule

      serviceAccountName: flannel

      initContainers:

      - name: install-cni-plugin

       #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply)

        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1

        command:

        - cp

        args:

        - -f

        - /flannel

        - /opt/cni/bin/flannel

        volumeMounts:

        - name: cni-plugin

          mountPath: /opt/cni/bin

      - name: install-cni

       #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)

        image: rancher/mirrored-flannelcni-flannel:v0.17.0

        command:

        - cp

        args:

        - -f

        - /etc/kube-flannel/cni-conf.json

        - /etc/cni/net.d/10-flannel.conflist

        volumeMounts:

        - name: cni

          mountPath: /etc/cni/net.d

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      containers:

      - name: kube-flannel

       #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)

        image: rancher/mirrored-flannelcni-flannel:v0.17.0

        command:

        - /opt/bin/flanneld

        args:

        #- --iface=enp0s8 # 需要指定网卡，否则可能会导致问题

        - --ip-masq

        - --kube-subnet-mgr

        resources:

          requests:

            cpu: "100m"

            memory: "50Mi"

          limits:

            cpu: "100m"

            memory: "50Mi"

        securityContext:

          privileged: false

          capabilities:

            add: ["NET_ADMIN", "NET_RAW"]

        env:

        - name: POD_NAME

          valueFrom:

            fieldRef:

              fieldPath: metadata.name

        - name: POD_NAMESPACE

          valueFrom:

            fieldRef:

              fieldPath: metadata.namespace

        - name: EVENT_QUEUE_DEPTH

          value: "5000"

        volumeMounts:

        - name: run

          mountPath: /run/flannel

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

        - name: xtables-lock

          mountPath: /run/xtables.lock

      volumes:

      - name: run

        hostPath:

          path: /run/flannel

      - name: cni-plugin

        hostPath:

          path: /opt/cni/bin

      - name: cni

        hostPath:

          path: /etc/cni/net.d

      - name: flannel-cfg

        configMap:

          name: kube-flannel-cfg

      - name: xtables-lock

        hostPath:

          path: /run/xtables.lock

          type: FileOrCreate

欢迎大家访问个人博客 Johnny小屋