kubespray2.11安装kubernetes1.15

关于kubespray

Kubespray是开源的kubernetes部署工具，整合了ansible，可以方便的部署高可用集群环境，官网地址：https://github.com/kubernetes-sigs/kubespray，本文是用kubespray2.11版本部署kubernetes1.15版本的实战；

重要前提

本次实战采用官方推荐的在线安装，因此会去谷歌镜像仓库下载镜像，需要您的网络可以访问谷歌服务；

机器信息

本次实战共计四台机器，它们的主机名、IP地址和作用描述如下：

主机名	IP地址	作用
ansible	192.168.133.134	ansible主机
a001	192.168.133.139	k8s集群的一号工作节点
a002	192.168.133.140	k8s集群的二号工作节点
a003	192.168.133.141	k8s集群的三号工作节点

标准化设置

本次实战的所有机器都要做以下设置：

操作系统：CentOS Linux release 7.7.1908
所以操作都是root账号执行的
关闭防火墙：

systemctl stop firewalld && systemctl disable firewalld

关闭selinux：

setenforce 0

sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

ipv4网络设置：

modprobe br_netfilter

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

sysctl -w net.ipv4.ip_forward=1

ansible主机免密码ssh登录a001、a002、a003

ssh登录ansible主机；
生成ssh公私钥，输入命令ssh-keygen，然后连续四次回车：

[root@ansible ~]# ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:Empen3/RfLndRkS8mKfkq6a2IXtSdqwK7TqKNoHkNEU root@ansible

The key's randomart image is:

+---[RSA 2048]----+

|  .E           . |

|   .            o|

|  .   .       o..|

| +   . .     + o.|

|= . o o S . ooo..|

|.o o ..o + o.oo.o|

|  . .. o=.o  ..o+|

| o.  .o.o=.... .+|

|......o+=o=o.  . |

+----[SHA256]-----+

输入命令ssh-copy-id root@192.168.133.139，将ansible的ssh分发给a001主机，会要求输入yes和a001主机的root账号的密码，完成输入后，以后ansible就可以免密码ssh登录a001主机了：

[root@ansible ~]# ssh-copy-id root@192.168.133.139

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"

The authenticity of host '192.168.133.139 (192.168.133.139)' can't be established.

ECDSA key fingerprint is SHA256:DPE2nldWHiOhC4DB9doy7jPWNZVup6XFZ+sR2i1gqz8.

ECDSA key fingerprint is MD5:fc:21:f7:7f:e8:cd:1a:76:d7:fb:cc:d4:28:91:f3:5a.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@192.168.133.139's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.133.139'"

and check to make sure that only the key(s) you wanted were added.

继续输入命令ssh-copy-id root@192.168.133.140和ssh-copy-id root@192.168.133.141，使得ansible主机可以免密码登录a002和a003；
至此，ansible主机可以用命令ssh root@192.168.133.139、ssh root@192.168.133.140、ssh root@192.168.133.141免密码登录a001、a002、a003了；

ansible主机操作

ssh登录ansible主机；
安装ansible应用：

yum install -y epel-release ansible

安装pip：

easy_install pip

通过pip安装jinja2：

pip2 install jinja2 --upgrade

安装python36：

yum install python36 -y

mkdir /usr/local/kubespray && cd /usr/local/kubespray/

下载kubespray，我这里下载的是v2.11.0版本：

wget https://github.com/kubernetes-sigs/kubespray/archive/v2.11.0.tar.gz

解压：

tar -zxvf v2.11.0.tar.gz

cd kubespray-2.11.0/

安装kubespray所需的应用(注意是pip3)：

pip3 install -r requirements.txt

复制一份demo配置信息到目录inventory/mycluster：

cp -rfp inventory/sample inventory/mycluster

进去看一下，可见mycluster目录下复制了很多文件：

[root@ansible kubespray-2.11.0]# tree inventory/

inventory/

├── local

│   ├── group_vars -> ../sample/group_vars

│   └── hosts.ini

├── mycluster

│   ├── group_vars

│   │   ├── all

│   │   │   ├── all.yml

│   │   │   ├── azure.yml

│   │   │   ├── coreos.yml

│   │   │   ├── docker.yml

│   │   │   ├── oci.yml

│   │   │   └── openstack.yml

│   │   ├── etcd.yml

│   │   └── k8s-cluster

│   │       ├── addons.yml

│   │       ├── k8s-cluster.yml

│   │       ├── k8s-net-calico.yml

│   │       ├── k8s-net-canal.yml

│   │       ├── k8s-net-cilium.yml

│   │       ├── k8s-net-contiv.yml

│   │       ├── k8s-net-flannel.yml

│   │       ├── k8s-net-kube-router.yml

│   │       ├── k8s-net-macvlan.yml

│   │       └── k8s-net-weave.yml

│   └── inventory.ini

设置集群信息(当前目录仍旧是kubespray-2.11.0)：

declare -a IPS=(192.168.133.139 192.168.133.140 192.168.133.141)

配置ansible：

CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

此时kubespray的脚本根据输入的IP信息做好了集群规划，具体信息可见inventory/mycluster/hosts.yml，如下所示，您也可以自行修改此文件：

all:

  hosts:

    node1:

      ansible_host: 192.168.133.139

      ip: 192.168.133.139

      access_ip: 192.168.133.139

    node2:

      ansible_host: 192.168.133.140

      ip: 192.168.133.140

      access_ip: 192.168.133.140

    node3:

      ansible_host: 192.168.133.141

      ip: 192.168.133.141

      access_ip: 192.168.133.141

  children:

    kube-master:

      hosts:

        node1:

        node2:

    kube-node:

      hosts:

        node1:

        node2:

        node3:

    etcd:

      hosts:

        node1:

        node2:

        node3:

    k8s-cluster:

      children:

        kube-master:

        kube-node:

    calico-rr:

      hosts: {}

执行以下命令即可开始安装，在线安装比较耗时请耐心等待：

ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

安装完成时控制台输出类似如下的信息：

PLAY RECAP ********************************************************************************************************************************************************************************

localhost                  : ok=1    changed=0    unreachable=0    failed=0

node1                      : ok=658  changed=95   unreachable=0    failed=0

node2                      : ok=566  changed=77   unreachable=0    failed=0

node3                      : ok=475  changed=66   unreachable=0    failed=0   

Sunday 17 November 2019  17:31:19 +0800 (0:00:00.064)       0:09:56.193 *******

===============================================================================

kubernetes/master : kubeadm | Init other uninitialized masters -------------------------------------------------------------------------------------------------------------------- 94.91s

kubernetes/master : kubeadm | Initialize first master ----------------------------------------------------------------------------------------------------------------------------- 42.95s

etcd : Install | Copy etcdctl binary from docker container ------------------------------------------------------------------------------------------------------------------------ 14.26s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------ 12.87s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------ 12.28s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------ 10.79s

etcd : reload etcd ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.71s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 9.71s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 9.48s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 8.02s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 7.88s

etcd : wait for etcd up ------------------------------------------------------------------------------------------------------------------------------------------------------------ 7.16s

etcd : Gen_certs | Write etcd master certs ----------------------------------------------------------------------------------------------------------------------------------------- 6.39s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 5.75s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 5.53s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 5.42s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 5.41s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 5.06s

download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 4.87s

kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------------------------------------------------------------------------------------ 4.78s

至此，kubernetes集群环境部署完成，接下来简单验证一下环境是否可用；

检查环境

ssh登录a001机器；
查看节点、service、pod：

[root@node1 ~]# kubectl get nodes

NAME    STATUS   ROLES    AGE   VERSION

node1   Ready    master   25m   v1.15.3

node2   Ready    master   23m   v1.15.3

node3   Ready    <none>   23m   v1.15.3

[root@node1 ~]# kubectl get services --all-namespaces

NAMESPACE     NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE

default       kubernetes             ClusterIP   10.233.0.1    <none>        443/TCP                  25m

kube-system   coredns                ClusterIP   10.233.0.3    <none>        53/UDP,53/TCP,9153/TCP   22m

kube-system   kubernetes-dashboard   ClusterIP   10.233.35.1   <none>        443/TCP                  22m

[root@node1 ~]# kubectl get pods --all-namespaces

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE

kube-system   calico-kube-controllers-c6fb79b8b-v24nq   1/1     Running   0          22m

kube-system   calico-node-46s8t                         1/1     Running   0          23m

kube-system   calico-node-mcjfs                         1/1     Running   0          23m

kube-system   calico-node-q989m                         1/1     Running   1          23m

kube-system   coredns-74c9d4d795-4xz6s                  1/1     Running   0          22m

kube-system   coredns-74c9d4d795-kh6vl                  1/1     Running   0          22m

kube-system   dns-autoscaler-7d95989447-gmcrl           1/1     Running   0          22m

kube-system   kube-apiserver-node1                      1/1     Running   0          24m

kube-system   kube-apiserver-node2                      1/1     Running   0          23m

kube-system   kube-controller-manager-node1             1/1     Running   0          24m

kube-system   kube-controller-manager-node2             1/1     Running   0          23m

kube-system   kube-proxy-2zhwn                          1/1     Running   0          23m

kube-system   kube-proxy-59qx8                          1/1     Running   0          23m

kube-system   kube-proxy-fgpx6                          1/1     Running   0          23m

kube-system   kube-scheduler-node1                      1/1     Running   0          24m

kube-system   kube-scheduler-node2                      1/1     Running   0          23m

kube-system   kubernetes-dashboard-7c547b4c64-x7nfq     1/1     Running   0          22m

kube-system   nginx-proxy-node3                         1/1     Running   0          23m

kube-system   nodelocaldns-8khfq                        1/1     Running   0          22m

kube-system   nodelocaldns-pzx2p                        1/1     Running   0          22m

kube-system   nodelocaldns-s5kcd                        1/1     Running   0          22m

访问dashboard

dashboard可以查看kubernetes系统的整体情况，为了访问dashboard页面，需要增加RBAC：

ssh登录a001机器；
执行以下命令，创建文件admin-user.yaml：

tee admin-user.yaml <<-'EOF'

apiVersion: v1

kind: ServiceAccount

metadata:

  name: admin-user

  namespace: kube-system

EOF

执行以下命令，创建文件admin-user-role.yaml：

tee admin-user-role.yaml <<-'EOF'

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: admin-user

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

- kind: ServiceAccount

  name: admin-user

  namespace: kube-system

EOF

创建ServiceAccount和ClusterRoleBinding：

kubectl create -f admin-user.yaml && kubectl create -f admin-user-role.yaml

获取token看，用于登录dashboard页面：

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

下图红框中就是token的内容：

6. 现在通过浏览器访问dashboard页面了，地址是：https://192.168.133.139:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ ，其中192.168.133.139是a001机器的IP地址，也可以换成a002IP地址；

7. 由于不是https协议，因此浏览器可能弹出安全提示，如下图，选择继续前往：

8. 此时页面会让您选择登录方式，选择令牌并输入前面得到的token，即可登录：

9. 登录成功后可以见到系统信息，如下图：

至此，kubespray2.11安装kubernetes1.15完成，希望本文能给您一些参考。

欢迎关注公众号：程序员欣宸