使用kubeadm搭建Kubernetes(1.10.2)集群（国内环境）

目标
准备
- 主机
- 软件
步骤
卸载集群

目标

在您的机器上建立一个安全的Kubernetes集群。
在集群里安装网络插件，以便应用之间可以相互通讯。
在集群上运行一个简单的微服务。

准备

主机

一台或多台运行Ubuntu 16.04+的主机。
最好选至少有 2 GB 内存的双核主机。
集群中完整的网络连接，公网或者私网都可以。

软件

安装Docker

sudo apt-get update

sudo apt-get install -y docker.io

Kubunetes建议使用老版本的docker.io，如果需要使用最新版的docker-ce，可参考上一篇博客：Docker初体验。

禁用swap文件

然后需要禁用swap文件，这是Kubernetes的强制步骤。实现它很简单，编辑/etc/fstab文件，注释掉引用swap的行，保存并重启后输入sudo swapoff -a即可。

对于禁用swap内存，你可能会有点不解，具体原因可以查看Github上的Issue：Kubelet/Kubernetes should work with Swap Enabled。

步骤

(1/4)安装 kubeadm, kubelet and kubectl

kubeadm: 引导启动k8s集群的命令工具。
kubelet: 在群集中的所有计算机上运行的组件, 并用来执行如启动pods和containers等操作。
kubectl: 用于操作运行中的集群的命令行工具。

sudo apt-get update && sudo apt-get install -y apt-transport-https

curl -s http://packages.faasx.com/google/apt/doc/apt-key.gpg | sudo apt-key add -

sudo cat <<EOF >/etc/apt/sources.list.d/kubernetes.list

deb http://mirrors.ustc.edu.cn/kubernetes/apt/ kubernetes-xenial main

EOF

sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

apt-key下载地址使用了国内镜像，官方地址为：https://packages.cloud.google.com/apt/doc/apt-key.gpg。

apt安装包地址使用了中科大的镜像，官方地址为：http://apt.kubernetes.io/。

(2/4)初始化master节点

由于网络原因，我们需要提前拉取k8s初始化需要用到的Images，并添加对应的k8s.gcr.io标签:

## 拉取镜像

docker pull reg.qiniu.com/k8s/kube-apiserver-amd64:v1.10.2

docker pull reg.qiniu.com/k8s/kube-controller-manager-amd64:v1.10.2

docker pull reg.qiniu.com/k8s/kube-scheduler-amd64:v1.10.2

docker pull reg.qiniu.com/k8s/kube-proxy-amd64:v1.10.2

docker pull reg.qiniu.com/k8s/etcd-amd64:3.1.12

docker pull reg.qiniu.com/k8s/pause-amd64:3.1

## 添加Tag

docker tag reg.qiniu.com/k8s/kube-apiserver-amd64:v1.10.2 k8s.gcr.io/kube-apiserver-amd64:v1.10.2

docker tag reg.qiniu.com/k8s/kube-scheduler-amd64:v1.10.2 k8s.gcr.io/kube-scheduler-amd64:v1.10.2

docker tag reg.qiniu.com/k8s/kube-controller-manager-amd64:v1.10.2 k8s.gcr.io/kube-controller-manager-amd64:v1.10.2

docker tag reg.qiniu.com/k8s/kube-proxy-amd64:v1.10.2 k8s.gcr.io/kube-proxy-amd64:v1.10.2

docker tag reg.qiniu.com/k8s/etcd-amd64:3.1.12 k8s.gcr.io/etcd-amd64:3.1.12

docker tag reg.qiniu.com/k8s/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1

## 在Kubernetes 1.10 中，增加了CoreDNS，如果使用CoreDNS(默认关闭)，则不需要下面三个镜像。

docker pull reg.qiniu.com/k8s/k8s-dns-sidecar-amd64:1.14.10

docker pull reg.qiniu.com/k8s/k8s-dns-kube-dns-amd64:1.14.10

docker pull reg.qiniu.com/k8s/k8s-dns-dnsmasq-nanny-amd64:1.14.10

docker tag reg.qiniu.com/k8s/k8s-dns-sidecar-amd64:1.14.10 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10

docker tag reg.qiniu.com/k8s/k8s-dns-kube-dns-amd64:1.14.10 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.10

docker tag reg.qiniu.com/k8s/k8s-dns-dnsmasq-nanny-amd64:1.14.10 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10

据说kubeadm可以自定义镜像Registry，但我并没有实验成功。

Master节点就是运行着控制组件的机器，包括etcd(集群数据库)和API服务(kubectl CLI通讯服务)。

初始化master节点, 只需随便在一台装过kubeadm的机器上运行如下命令:

sudo kubeadm init --kubernetes-version=v1.10.2 --feature-gates=CoreDNS=true --pod-network-cidr=192.168.0.0/16

init 常用主要参数：

--kubernetes-version: 指定Kubenetes版本，如果不指定该参数，会从google网站下载最新的版本信息。
--pod-network-cidr: 指定pod网络的IP地址范围，它的值取决于你在下一步选择的哪个网络网络插件，比如我在本文中使用的是Calico网络，需要指定为192.168.0.0/16。
--apiserver-advertise-address: 指定master服务发布的Ip地址，如果不指定，则会自动检测网络接口，通常是内网IP。
--feature-gates=CoreDNS: 是否使用CoreDNS，值为true/false，CoreDNS插件在1.10中提升到了Beta阶段，最终会成为Kubernetes的缺省选项。

关于kubeadm更详细的的介绍请参考kubeadm官方文档。

最终输出如下：

raining@raining-ubuntu:~$ sudo kubeadm init --kubernetes-version=v1.10.2 --feature-gates=CoreDNS=true --pod-network-cidr=192.168.0.0/16

[sudo] password for raining:

[init] Using Kubernetes version: v1.10.2

[init] Using Authorization modes: [Node RBAC]

[preflight] Running pre-flight checks.

	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.12.1-ce. Max validated version: 17.03

	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'

	[WARNING FileExisting-crictl]: crictl not found in system path

Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

[preflight] Starting the kubelet service

[certificates] Generated ca certificate and key.

[certificates] Generated apiserver certificate and key.

[certificates] apiserver serving cert is signed for DNS names [raining-ubuntu kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.8]

[certificates] Generated apiserver-kubelet-client certificate and key.

[certificates] Generated etcd/ca certificate and key.

[certificates] Generated etcd/server certificate and key.

[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]

[certificates] Generated etcd/peer certificate and key.

[certificates] etcd/peer serving cert is signed for DNS names [raining-ubuntu] and IPs [192.168.0.8]

[certificates] Generated etcd/healthcheck-client certificate and key.

[certificates] Generated apiserver-etcd-client certificate and key.

[certificates] Generated sa key and public key.

[certificates] Generated front-proxy-ca certificate and key.

[certificates] Generated front-proxy-client certificate and key.

[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"

[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"

[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"

[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"

[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"

[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".

[init] This might take a minute or longer if the control plane images have to be pulled.

[apiclient] All control plane components are healthy after 39.501722 seconds

[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[markmaster] Will mark node raining-ubuntu as master by adding a label and a taint

[markmaster] Master raining-ubuntu tainted and labelled with key/value: node-role.kubernetes.io/master=""

[bootstraptoken] Using token: vtyk9m.g4afak37myq3rsdi

[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node

as root:

  kubeadm join 192.168.0.8:6443 --token vtyk9m.g4afak37myq3rsdi --discovery-token-ca-cert-hash sha256:19246ce11ba3fc633fe0b21f2f8aaaebd7df9103ae47138dc0dd615f61a32d99

如果想在非root用户下使用kubectl，可以执行如下命令(也是kubeadm init输出的一部分)：

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubeadm init 输出的token用于master和加入节点间的身份认证，token是机密的，需要保证它的安全，因为拥有此标记的人都可以随意向集群中添加节点。你也可以使用kubeadm命令列出，创建，删除Token，有关详细信息, 请参阅官方引用文档。

我们在浏览器中输入https://<master-ip>:6443来验证一下是否部署成功，返回如下：

{

  "kind": "Status",

  "apiVersion": "v1",

  "metadata": {

  },

  "status": "Failure",

  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",

  "reason": "Forbidden",

  "details": {

  },

  "code": 403

}

(3/4) 安装网络插件

安装一个网络插件是必须的，因为你的pods之间需要彼此通信。

网络部署必须是优先于任何应用的部署，如kube-dns(本文中使用的是coredns)在网络部署成功之前是无法使用的。kubeadm只支持容器网络接口（CNI）的网络类型（不支持kubenet）。

比较常见的network addon有：Calico, Canal, Flannel, Kube-router, Romana, Weave Net等。详细的网络列表可参考插件页面。

使用下列命令来安装网络插件:

kubectl apply -f <add-on.yaml>

在本文中，我使用的是Calico网络，安装如下：

# 使用国内镜像

kubectl apply -f http://mirror.faasx.com/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml

为了Calico可以正常运行，必须在执行kubeadm init时使用 --pod-network-cidr=192.168.0.0/16。

更详细的可以查看Calico官方文档：kubeadm quickstart。

网络插件安装完成后，可以通过检查coredns pod的运行状态来判断网络插件是否正常运行：

kubectl get pods --all-namespaces

# 输出

NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE

kube-system   calico-etcd-zxmvh                         1/1       Running   0          4m

kube-system   calico-kube-controllers-f9d6c4cb6-42w9j   1/1       Running   0          4m

kube-system   calico-node-jq5qb                         2/2       Running   0          4m

kube-system   coredns-7997f8864c-kfswc                  1/1       Running   0          1h

kube-system   coredns-7997f8864c-ttvj2                  1/1       Running   0          1h

kube-system   etcd-raining-ubuntu                       1/1       Running   0          1h

kube-system   kube-apiserver-raining-ubuntu             1/1       Running   0          1h

kube-system   kube-controller-manager-raining-ubuntu    1/1       Running   0          1h

kube-system   kube-proxy-vrjlq                          1/1       Running   0          1h

kube-system   kube-scheduler-raining-ubuntu             1/1       Running   0          1h

等待coredns pod的状态变成Running，就可以继续添加从节点了。

隔离主节点

默认情况下，出于安全的考虑，并不会在主节点上运行pod，如果你想在主节点上运行pod，比如：运行一个单机版的kubernetes集群时，可运行下面的命令：

kubectl taint nodes --all node-role.kubernetes.io/master-

输出类似这样：

node "test-01" untainted

taint key="dedicated" and effect="" not found.

taint key="dedicated" and effect="" not found.

这将移除所有节点的node-role.kubernetes.io/master标志，包括主节点，Scheduler便可以在任何节点上安排运行pod了。

(4/4)加入其他节点

节点就是你的负载（容器和pod等等）运行的地方，往集群里添加节点，只需要在每台机器上执行下列几步：

SSH 登录机器
切换到root (比如 sudo su -)
执行kubeadm init输出的那句命令: kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

执行后输出类似这样:

raining@ubuntu1:~$ sudo kubeadm join 192.168.0.8:6443 --token vtyk9m.g4afak37myq3rsdi --discovery-token-ca-cert-hash sha256:19246ce11ba3fc633fe0b21f2f8aaaebd7df9103ae47138dc0dd615f61a32d99

[preflight] Running pre-flight checks.

	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.12.1-ce. Max validated version: 17.03

	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'

	[WARNING FileExisting-crictl]: crictl not found in system path

Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

[preflight] Starting the kubelet service

[discovery] Trying to connect to API Server "192.168.0.8:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.8:6443"

[discovery] Requesting info from "https://192.168.0.8:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.0.8:6443"

[discovery] Successfully established connection with API Server "192.168.0.8:6443"

This node has joined the cluster:

* Certificate signing request was sent to master and a response

  was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

几秒后，你在主节点上运行kubectl get nodes就可以看到新加的机器了：

NAME             STATUS    ROLES     AGE       VERSION

raining-ubuntu   Ready     master    1h        v1.10.2

ubuntu1          Ready     <none>    2m        v1.10.2

(可选)在非主节点上管理集群

为了可以在其他电脑上使用kubectl来管理你的集群，可以从主节点上复制管理员的kubeconfig文件到你的电脑上：

scp root@<master ip>:/etc/kubernetes/admin.conf .

kubectl --kubeconfig ./admin.conf get nodes

(可选)映射API服务到本地

如果你想从集群外部连接到API服务，可以使用工具kubectl proxy:

scp root@<master ip>:/etc/kubernetes/admin.conf .

kubectl --kubeconfig ./admin.conf proxy

这样就可以在本地这样 http://localhost:8001/api/v1 访问到API服务了。

(可选)部署一个微服务

现在可以测试你新搭建的集群了，Sock Shop就是一个微服务的样本，它体现了在Kubernetes里如何运行和连接一系列的服务。想了解更多关于微服务的内容，请查看GitHub README。

kubectl create namespace sock-shop

kubectl apply -n sock-shop -f "https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true"

可以通过以下命令来查看前端服务是否有开放对应的端口：

kubectl -n sock-shop get svc front-end

输出类似:

NAME        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

front-end   NodePort   10.107.207.35   <none>        80:30001/TCP   31s

可能需要几分钟时间来下载和启用所有的容器，通过kubectl get pods -n sock-shop来获取服务的状态。

输出如下：

raining@raining-ubuntu:~$ kubectl get pods -n sock-shop

NAME                            READY     STATUS    RESTARTS   AGE

carts-6cd457d86c-wdbsg          1/1       Running   0          1m

carts-db-784446fdd6-9gsrs       1/1       Running   0          1m

catalogue-779cd58f9b-nf6n4      1/1       Running   0          1m

catalogue-db-6794f65f5d-kwc2x   1/1       Running   0          1m

front-end-679d7bcb77-4hbjq      1/1       Running   0          1m

orders-755bd9f786-gbspz         1/1       Running   0          1m

orders-db-84bb8f48d6-98wsm      1/1       Running   0          1m

payment-674658f686-xc7gk        1/1       Running   0          1m

queue-master-5f98bbd67-xgqr6    1/1       Running   0          1m

rabbitmq-86d44dd846-nf2g6       1/1       Running   0          1m

shipping-79786fb956-bs7jn       1/1       Running   0          1m

user-6995984547-nvqw4           1/1       Running   0          1m

user-db-fc7b47fb9-zcf5r         1/1       Running   0          1m

然后在你的浏览器里访问集群节点的IP和对应的端口，比如http://<master_ip>/<cluster-ip>:<port>。在这个例子里，可能是30001，但是它可能跟你的不一样。如果有防火墙的话，确保在你访问之前开放了对应的端口。

需要注意的是，如果在多节点部署时，要使用节点的IP进行访问，而不是Master服务器的IP。

最后，卸载socks shop, 只需要在主节点上运行:

kubectl delete namespace sock-shop

卸载集群

想要撤销kubeadm做的事，首先要排除节点，并确保在关闭节点之前要清空节点。

在主节点上运行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets

kubectl delete node <node name>

然后在需要移除的节点上，重置kubeadm的安装状态：

kubeadm reset

如果你想重新配置集群，只需运行kubeadm init或者kubeadm join并使用所需的参数即可。

参考资料