使用 Kubeadm 安装部署 Kubernetes 1.12.1 集群

手工搭建 Kubernetes 集群是一件很繁琐的事情，为了简化这些操作，就产生了很多安装配置工具，如 Kubeadm ，Kubespray，RKE 等组件，我最终选择了官方的 Kubeadm 主要是不同的 Kubernetes 版本都有一些差异，Kubeadm 更新与支持的会好一些。Kubeadm 是 Kubernetes 官方提供的快速安装和初始化 Kubernetes 集群的工具，目前的还处于孵化开发状态，跟随  Kubernetes 每个新版本的发布都会同步更新, 强烈建议先看下官方的文档了解下各个组件与对象的作用。

• https://kubernetes.io/docs/concepts/
• https://kubernetes.io/docs/setup/independent/install-kubeadm/
• https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

关于其他部署方式参考如下：

使用 Kubespray 部署生产可用的Kubernetes集群（1.11.2）

rancher 中国区加速安装 Kubernetes

系统环境配置

准备3台服务器，1 个Master 节点 2个 Node 节点（所有节点均需执行如下步骤）；生产环境建议 3个 Master N 个 Node 节点，好做到扩展迁移与灾备。

系统版本

$ cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

修改主机名

$ sudo hostnamectl set-hostname kubernetes-master
$ sudo hostnamectl set-hostname kubernetes-node-1
$ sudo hostnamectl set-hostname kubernetes-node-2

关闭防火墙

$ systemctl stop firewalld && systemctl disable firewalld

备注: 开放的端口

关闭 elinux

$ setenforce 0
$ sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

关闭 swap

$ swapoff -a

解决路由异常问题

$ echo "net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0" >> /etc/sysctl.d/k8s.conf
$ sysctl -p /etc/sysctl.d/k8s.conf

或

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl –system

问题：[preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

安装 docker（阿里云镜像）

$ curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
$ systemctl enable docker && systemctl start docker

安装 kubelet kubeadm kubectl (阿里云镜像)

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

备注：

可以通过 yum list --showduplicates | grep 'kubeadm\|kubectl\|kubelet'  查询可用的版本安装指定的版本。

查看版本

$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ kubeadm  version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
$ kubelet --version
Kubernetes v1.12.1

拉取镜像

由于 k8s.gcr.io 访问不了原因，国人在 github 上同步一份镜像，可以通过如下 shell 脚本拉取(不同的 kubernetes 版本对应镜像组件版本也不相同 ，如下我已经匹配好了)

$ touch pull_k8s_images.sh

#!/bin/bash
images=(kube-proxy:v1.12.1 kube-scheduler:v1.12.1 kube-controller-manager:v1.12.1
kube-apiserver:v1.12.1 kubernetes-dashboard-amd64:v1.10.0
heapster-amd64:v1.5.4 heapster-grafana-amd64:v5.0.4 heapster-influxdb-amd64:v1.5.2 etcd:3.2.24 coredns:1.2.2 pause:3.1 )
for imageName in ${images[@]} ; do
docker pull anjia0532/google-containers.$imageName
docker tag anjia0532/google-containers.$imageName k8s.gcr.io/$imageName
docker rmi anjia0532/google-containers.$imageName
done

$ sh touch pull_k8s_images.sh

其他同步镜像源:

• https://github.com/mritd/gcrsync

• https://github.com/openthings/kubernetes-tools/blob/master/kubeadm/2-images/
• https://github.com/anjia0532/gcr.io_mirror
• https://www.jianshu.com/p/832bcd89bc07（registry.cn-hangzhou.aliyuncs.com/google_containers）

查看该版本需要的容器镜像版本

$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.12.1
k8s.gcr.io/kube-controller-manager:v1.12.1
k8s.gcr.io/kube-scheduler:v1.12.1
k8s.gcr.io/kube-proxy:v1.12.1
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.2
# 查看已 pull 好的镜像
$ docker images

备注 ：

官方文档中说明，不同的 Kubernetes 版本拉取的镜像也不同，如 1.12 已经不需要指定平台了（amd64, arm, arm64, ppc64le or s390x）,另外新版本 CoreDNS (kube-dns 的替代品) 服务组件也默认包含，无需指定 feature-gates=CoreDNS=true 配置 。

Here v1.10.x means the “latest patch release of the v1.10 branch”.

${ARCH} can be one of: amd64, arm, arm64, ppc64le or s390x.

If you run Kubernetes version 1.10 or earlier, and if you set --feature-gates=CoreDNS=true, you must also use the coredns/coredns image, instead of the three k8s-dns-* images.

In Kubernetes 1.11 and later, you can list and pull the images using the kubeadm config images sub-command:

kubeadm config images list
kubeadm config images pull

Starting with Kubernetes 1.12, the k8s.gcr.io/kube-*, k8s.gcr.io/etcd and k8s.gcr.io/pause images don’t require an -${ARCH} suffix.

Kubeadm 基本命令

# 创建一个 Master 节点
$ kubeadm init
# 将一个 Node 节点加入到当前集群中
$ kubeadm join <Master 节点的 IP 和端口 >

Kubeadm 部署 Kubernetes 集群最关键的两个步骤，kubeadm init 和 kubeadm join。可以定制集群组件的参数，新建 kubeadm.yaml 配置文件。

$ touch kubeadm.yaml

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
controllerManagerExtraArgs:
  horizontal-pod-autoscaler-use-rest-clients: "true"
  horizontal-pod-autoscaler-sync-period: "10s"
  node-monitor-grace-period: "10s"
apiServerExtraArgs:
  runtime-config: "api/all=true"
kubernetesVersion: "v1.12.1"

备注：

如果报如下错误

your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha1". Please use kubeadm v1.11 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

请检查是否是 1.1x 版本，官方建议使用 v1alpha3 版本，具体可以看官方的变更文档（https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file）。

In Kubernetes 1.11 and later, the default configuration can be printed out using the kubeadm config print-default command. It is recommended that you migrate your old v1alpha2 configuration to v1alpha3 using the kubeadm config migrate command, because v1alpha2 will be removed in Kubernetes 1.13.

For more details on each field in the v1alpha3 configuration you can navigate to our API reference pages.

创建 Kubernetes  集群

创建 Master 节点

$ kubeadm init --config kubeadm.yaml

[init] using Kubernetes version: v1.12.1
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [021rjsh216048s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.23.216.48]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [021rjsh216048s localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [021rjsh216048s localhost] and IPs [172.23.216.48 127.0.0.1 ::1]
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 24.503270 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node 021rjsh216048s as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node 021rjsh216048s as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "021rjsh216048s" as an annotation
[bootstraptoken] using token: zbnjyn.d5ntetgw5mpp9blv
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.23.216.48:6443 --token zbnjyn.d5ntetgw5mpp9blv --discovery-token-ca-cert-hash sha256:3dff1b750972001675fb8f5284722733f014f60d4371cdffb36522cbda6acb98

kubeadm join 命令，就是用来给这个 Master 增加更多的 Node 节点，另外 Kubeadm 还会提示第一次使用 Kubernetes 集群需要的配置命令

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Kubernetes 集群默认需要加密方式访问 ，这几条命名就是将集群的安全配置文件保存到当前用户的 .kube/config 目录下，kubectl 默认会使用这个目录下的授权信息访问 Kubernetes 集群。

部署网络插件

Container Network Interface (CNI) 最早是由 CoreOS 发起的容器网络规范，是 Kubernetes 网络插件的基础。其基本思想为：Container Runtime 在创建容器时，先创建好 network namespace，然后调用 CNI 插件为这个 netns 配置网络，其后再启动容器内的进程。现已加入 CNCF，成为 CNCF 主推的网络模型。

• https://kubernetes.io/docs/concepts/cluster-administration/networking/

• https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy

常见的 CNI 网络插件有很多可以选择：

• ACI provides integrated container networking and network security with Cisco ACI.

• Calico is a secure L3 networking and network policy provider.
• Canal unites Flannel and Calico, providing networking and network policy.
• Cilium is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported.
• CNI-Genie enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave.
• Contiv provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully open sourced. The installerprovides both kubeadm and non-kubeadm based installation options.
• Flannel is an overlay network provider that can be used with Kubernetes.
• Knitter is a network solution supporting multiple networking in Kubernetes.
• Multus is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
• NSX-T Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and Openshift.
• Nuage is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring.
• Romana is a Layer 3 networking solution for pod networks that also supports the NetworkPolicy API. Kubeadm add-on installation details available here.
• Weave Net provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.

这里使用 Weave 插件（https://www.weave.works/docs/net/latest/kubernetes/kube-addon/）。

$ kubectl apply -f https://git.io/weave-kube-1.6
或
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
或
$ kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.5.0/weave-daemonset-k8s-1.8.yaml

备注：其他功能：

• https://www.weave.works/docs/net/latest/kubernetes/kube-addon/

• https://www.weave.works/technologies/monitoring-kubernetes-with-prometheus/
• https://www.weave.works/docs/scope/latest/installing/

或者选择 flannel

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

备注CNI-Genie 是华为 PaaS 团队推出的同时支持多种网络插件（支持 calico, canal, romana, weave 等）的 CNI 插件。

查看 Pod 状态

$  kubectl get pods -n kube-system -l name=weave-net -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP              NODE                NOMINATED NODE
weave-net-j9s27   2/2     Running   0          24h   172.23.216.49   kubernetes-node-1   <none>
weave-net-p22s2   2/2     Running   0          24h   172.23.216.50   kubernetes-node-2   <none>
weave-net-vnq7p   2/2     Running   0          24h   172.23.216.48   kubernetes-master   <none>

$ kubectl logs -n kube-system weave-net-j9s27 weave
$ kubectl logs weave-net-j9s27 -n kube-system weave-npc

增加 Node 节点

$ kubeadm join 172.23.216.48:6443 --token zbnjyn.d5ntetgw5mpp9blv --discovery-token-ca-cert-hash sha256:3dff1b750972001675fb8f5284722733f014f60d4371cdffb36522cbda6acb98

如果需要从其它任意节点控制集群，则需要复制 Master 的安全配置信息到每台服务器

$ mkdir -p $HOME/.kube
$ scp root@172.23.216.48:/etc/kubernetes/admin.conf $HOME/.kube/config
$ chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes

查看所有节点

$ kubectl get nodes
NAME             STATUS   ROLES    AGE     VERSION
021rjsh216048s   Ready    master   2d23h   v1.12.1
021rjsh216049s   Ready    <none>   2d23h   v1.12.1
021rjsh216050s   Ready    <none>   2d23h   v1.12.1
$ kubectl get pods -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-576cbf47c7-ps2s2                 1/1     Running   0          2d23h
coredns-576cbf47c7-qsxdx                 1/1     Running   0          2d23h
etcd-021rjsh216048s                      1/1     Running   0          2d23h
heapster-684777c4cb-qzz8f                1/1     Running   0          2d16h
kube-apiserver-021rjsh216048s            1/1     Running   0          2d23h
kube-controller-manager-021rjsh216048s   1/1     Running   1          2d23h
kube-proxy-5fgf9                         1/1     Running   0          2d23h
kube-proxy-hknws                         1/1     Running   0          2d23h
kube-proxy-qc6xj                         1/1     Running   0          2d23h
kube-scheduler-021rjsh216048s            1/1     Running   1          2d23h
kubernetes-dashboard-77fd78f978-pqdvw    1/1     Running   0          2d18h
monitoring-grafana-56b668bccf-tm2cl      1/1     Running   0          2d16h
monitoring-influxdb-5c5bf4949d-85d5c     1/1     Running   0          2d16h
weave-net-5fq89                          2/2     Running   0          2d23h
weave-net-flxgg                          2/2     Running   0          2d23h
weave-net-vvdkq                          2/2     Running   0          2d23h
$ kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE
kube-system   coredns-576cbf47c7-ps2s2                 1/1     Running   0          2d23h   10.32.0.3       021rjsh216048s   <none>
kube-system   coredns-576cbf47c7-qsxdx                 1/1     Running   0          2d23h   10.32.0.2       021rjsh216048s   <none>
kube-system   etcd-021rjsh216048s                      1/1     Running   0          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   heapster-684777c4cb-qzz8f                1/1     Running   0          2d16h   10.44.0.2       021rjsh216049s   <none>
kube-system   kube-apiserver-021rjsh216048s            1/1     Running   0          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   kube-controller-manager-021rjsh216048s   1/1     Running   1          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   kube-proxy-5fgf9                         1/1     Running   0          2d23h   172.23.216.49   021rjsh216049s   <none>
kube-system   kube-proxy-hknws                         1/1     Running   0          2d23h   172.23.216.50   021rjsh216050s   <none>
kube-system   kube-proxy-qc6xj                         1/1     Running   0          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   kube-scheduler-021rjsh216048s            1/1     Running   1          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   kubernetes-dashboard-77fd78f978-pqdvw    1/1     Running   0          2d18h   10.36.0.1       021rjsh216050s   <none>
kube-system   monitoring-grafana-56b668bccf-tm2cl      1/1     Running   0          2d16h   10.44.0.1       021rjsh216049s   <none>
kube-system   monitoring-influxdb-5c5bf4949d-85d5c     1/1     Running   0          2d16h   10.36.0.2       021rjsh216050s   <none>
kube-system   weave-net-5fq89                          2/2     Running   0          2d23h   172.23.216.48   021rjsh216048s   <none>
kube-system   weave-net-flxgg                          2/2     Running   0          2d23h   172.23.216.50   021rjsh216050s   <none>
kube-system   weave-net-vvdkq                          2/2     Running   0          2d23h   172.23.216.49   021rjsh216049s   <none>

查看健康状态

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

其他命令

#查看 master 节点的 token
$ kubeadm token list | grep authentication,signing | awk '{print $1}'
#查看 discovery-token-ca-cert-hash
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

安装 dashboard

查询了官方最新版本是 v1.10.0 版本，上述脚本已经拉取此镜像

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

修改 kubernetes-dashboard.yaml 文件，在 Dashboard Service 中添加 type: NodePort，暴露 Dashboard 服务。

# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

备注：

暴露服务很多种方式：

• 使用 Service.Type=LoadBalancer

• 使用 Service.Type=NodePort
• 使用 Port Proxy
• 部署一个 Service loadbalancer  允许多个 service 之间共享单个IP，并通过 Service Annotations 实现更高级的负载平衡。
• 使用 Ingress  从Kubernetes 集群外部访问集群内部服务的入口，使用 nginx 等开源的反向代理负载均衡器暴露服务。

安装插件

# 安装 Dashboard 插件
$ kubectl create -f kubernetes-dashboard.yaml
# 替换配置
$ kubectl replace --force -f kubernetes-dashboard.yaml

备注 ：使用 proxy 本地访问 ，集群外访需要使用 Ingress , 这里先使用 NodePort 方式。

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
$ kubectl proxy

Now access Dashboard at:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/.

授予 Dashboard 账户集群管理权限

创建一个 kubernetes-dashboard-admin 的 ServiceAccount 并授予集群admin的权限，创建 kubernetes-dashboard-admin.rbac.yaml。

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-admin
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard-admin
  namespace: kube-system

执行

$ kubectl create -f kubernetes-dashboard-admin.rbac.yaml

或

$ kubectl apply -f https://raw.githubusercontent.com/batizhao/dockerfile/master/k8s/kubernetes-dashboard/kubernetes-dashboard-admin.rbac.yaml

查看 Dashboard 服务端口

[root@kubernetes-master ~]# kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP   10.96.0.10    <none>        53/UDP,53/TCP   6h40m
kubernetes-dashboard   NodePort    10.98.73.56   <none>        443:30828/TCP   63m

查看 kubernete-dashboard-admin 的 token

$ kubectl -n kube-system get secret | grep kubernetes-dashboard-admin
kubernetes-dashboard-admin-token-4k82b           kubernetes.io/service-account-token   3      75m
$ kubectl describe -n kube-system secret/kubernetes-dashboard-admin-token-4k82b
Name:         kubernetes-dashboard-admin-token-4k82b
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard-admin
              kubernetes.io/service-account.uid: a904fbf5-d3aa-11e8-945d-0050569f4a19

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi00azgyYiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE5MDRmYmY1LWQzYWEtMTFlOC05NDVkLTAwNTA1NjlmNGExOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.DGajGHRfLmtFpCyoHKn4wS0ZHKALfwMgTUTjmGSzBM3u1rr4hF51KFWBVwBPCkFQ1e1A5v6ENdhCNUQ_b66XohehJqKdgF_OBx5MXe0den_XVquJVlQRHVssL2BW-MjLXccuJ4LrKf4Q7sjGOqr4ivd6D39Bqjv7e6BxFUGO6vRPFzAme5dbJ7u28_DJZ1RGgVz-ylz3wCRZC89bP_3qqd1RK5G-gF2--RPA3atoCfrTIPzynu-y3qLQl6EWtC-hYywGb1oJPRa1it7EqTsLXmuOHqR_9tpDfJwiN9oDcnjU0ZHe6ifLcHWwRRka5tuSnKD6S3iRgaM47xtQe8yn4A

部署 Heapter 插件(统计 Nodes、Pods 的 CPU、内存、负载等功能，看官网说明已废弃,未安装)

mkdir -p ~/k8s/heapster
cd ~/k8s/heapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
kubectl create -f ./

备注：

已过期，Heapter 将在 Kubernetes 1.13 版本中移除（https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md），推荐使用 metrics-server 与 Prometheus。

最后访问 https://172.23.216.48:30828（新版本的谷歌启用 HTTPS 安全性验证，好像不行），使用火狐打开，输入上述 Token 完成。

REFER:
https://kubernetes.io/docs/setup/independent/high-availability/
https://kubernetes.io/docs/setup/independent/install-kubeadm/
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file
https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/
https://opsx.alibaba.com/mirror?lang=zh-CN
https://jimmysong.io/posts/kubernetes-dashboard-upgrade/
https://blog.frognew.com/2018/10/kubeadm-install-kubernetes-1.12.html
https://github.com/opsnull/follow-me-install-kubernetes-cluster
https://github.com/kubernetes/examples
https://github.com/kubernetes/dashboard