kubespray(ansible)自动化安装k8s集群

https://github.com/kubernetes-incubator/kubespray

https://kubernetes.io/docs/setup/pick-right-solution/

kubespray本质是一堆ansible的role文件,通过这种方式,即ansible方式可以自动化的安装高可用k8s集群,目前支持1.9.

安装完成后,k8s所有组件都是通过hyperkube的容器化来运行的.

  • 最佳安装centos7

  • 安装docker(kubespray默认给docker添加的启动参数)

  1. /usr/bin/dockerd
  2. --insecure-registry=10.233.0.0/18
  3. --graph=/var/lib/docker
  4. --log-opt max-size=50m
  5. --log-opt max-file=5
  6. --iptables=false
  7. --dns 10.233.0.3
  8. --dns 114.114.114.114
  9. --dns-search default.svc.cluster.local
  10. --dns-search svc.cluster.local
  11. --dns-opt ndots:2
  12. --dns-opt timeout:2
  13. --dns-opt attempts:2

  2. - 规划3(n1  n2 n3)主2从(n4 n5)
  4. - hosts
  6. 192.168.2.11 n1.ma.com n1
  7. 192.168.2.12 n2.ma.com n2
  8. 192.168.2.13 n3.ma.com n3
  9. 192.168.2.14 n4.ma.com n4
  10. 192.168.2.15 n5.ma.com n5
  11. 192.168.2.16 n6.ma.com n6
  13. - 1.9的ansible-playbook里涉及到这些镜像
  14. gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
  15. gcr.io/google_containers/pause-amd64:3.0
  16. gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
  17. gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
  18. gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
  19. gcr.io/google_containers/elasticsearch:v2.4.1
  20. gcr.io/google_containers/fluentd-elasticsearch:1.22
  21. gcr.io/google_containers/kibana:v4.6.1
  22. gcr.io/kubernetes-helm/tiller:v2.7.2
  23. gcr.io/google_containers/kubernetes-dashboard-amd64:v1.0.1
  24. gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.7.1
  25. quay.io/l23network/k8s-netchecker-agent:v1.0
  26. quay.io/l23network/k8s-netchecker-server:v1.0
  27. quay.io/coreos/etcd:v3.2.4
  28. quay.io/coreos/flannel:v0.9.1
  29. quay.io/coreos/flannel-cni:v0.3.0
  30. quay.io/calico/ctl:v1.6.1
  31. quay.io/calico/node:v2.6.2
  32. quay.io/calico/cni:v1.11.0
  33. quay.io/calico/kube-controllers:v1.0.0
  34. quay.io/calico/routereflector:v0.4.0
  35. quay.io/coreos/hyperkube:v1.9.0_coreos.0
  36. quay.io/ant31/kargo:master
  37. quay.io/external_storage/local-volume-provisioner-bootstrap:v1.0.0
  38. quay.io/external_storage/local-volume-provisioner:v1.0.0
  40. - 1.9镜像dockerhub版
  41. lanny/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.1.1
  42. lanny/gcr.io_google_containers_pause-amd64:3.0
  43. lanny/gcr.io_google_containers_k8s-dns-kube-dns-amd64:1.14.7
  44. lanny/gcr.io_google_containers_k8s-dns-dnsmasq-nanny-amd64:1.14.7
  45. lanny/gcr.io_google_containers_k8s-dns-sidecar-amd64:1.14.7
  46. lanny/gcr.io_google_containers_elasticsearch:v2.4.1
  47. lanny/gcr.io_google_containers_fluentd-elasticsearch:1.22
  48. lanny/gcr.io_google_containers_kibana:v4.6.1
  49. lanny/gcr.io_kubernetes-helm_tiller:v2.7.2
  50. lanny/gcr.io_google_containers_kubernetes-dashboard-init-amd64:v1.0.1
  51. lanny/gcr.io_google_containers_kubernetes-dashboard-amd64:v1.7.1
  52. lanny/quay.io_l23network_k8s-netchecker-agent:v1.0
  53. lanny/quay.io_l23network_k8s-netchecker-server:v1.0
  54. lanny/quay.io_coreos_etcd:v3.2.4
  55. lanny/quay.io_coreos_flannel:v0.9.1
  56. lanny/quay.io_coreos_flannel-cni:v0.3.0
  57. lanny/quay.io_calico_ctl:v1.6.1
  58. lanny/quay.io_calico_node:v2.6.2
  59. lanny/quay.io_calico_cni:v1.11.0
  60. lanny/quay.io_calico_kube-controllers:v1.0.0
  61. lanny/quay.io_calico_routereflector:v0.4.0
  62. lanny/quay.io_coreos_hyperkube:v1.9.0_coreos.0
  63. lanny/quay.io_ant31_kargo:master
  64. lanny/quay.io_external_storage_local-volume-provisioner-bootstrap:v1.0.0
  65. lanny/quay.io_external_storage_local-volume-provisioner:v1.0.0
  67. - 配置文件
  68. kubespray/inventory/group_vars/k8s-cluster.yml  为控制一些基础信息的配置文件。
  69. 修改为flannel
  71. kubespray/inventory/group_vars/all.yml           控制一些需要详细配置的信息
  72. 修改为centos
  74. - 替换flannel的vxlan为host-gw
  75. roles/network_plugin/flannel/defaults/main.yml
  76. 可以通过grep -r 'vxlan' .  这种方式来找到
  78. - 修改kube_api_pwd
  79. vi roles/kubespray-defaults/defaults/main.yaml
  80. kube_api_pwd: xxxx
  82. - 证书时间
  83. kubespray/roles/kubernetes/secrets/files/make-ssl.sh
  85. - 修改image为个人仓库地址
  86. 可以通过grep -r 'gcr.io' .  这种方式来找到
  87. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/download/defaults/main.yml
  88. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
  89. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/kubernetes-apps/ansible/defaults/main.yml
  90. sed -i 's#gcr\.io\/kubernetes-helm\/#lanny/gcr\.io_kubernetes-helm_#g' roles/download/defaults/main.yml
  92. sed -i 's#quay\.io\/coreos\/#lanny/quay\.io_coreos_#g' roles/download/defaults/main.yml
  93. sed -i 's#quay\.io\/calico\/#lanny/quay\.io_calico_#g' roles/download/defaults/main.yml
  94. sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' roles/download/defaults/main.yml
  95. sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' docs/netcheck.md
  96. sed -i 's#quay\.io\/external_storage\/#lanny/quay\.io_external_storage_#g' roles/kubernetes-apps/local_volume_provisioner/defaults/main.yml
  97. sed -i 's#quay\.io\/ant31\/kargo#lanny/quay\.io_ant31_kargo_#g' .gitlab-ci.yml
  99. - 安装完后查看镜像(用到的镜像(calico))
  100. nginx:1.13
  101. quay.io/coreos/hyperkube:v1.9.0_coreos.0
  102. gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
  103. gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
  104. gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
  105. quay.io/calico/node:v2.6.2
  106. gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.1
  107. quay.io/calico/cniv:1.11.0
  108. gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1
  109. quay.io/calico/ctlv:1.6.1
  110. quay.io/calico/routereflectorv:0.4.0
  111. quay.io/coreos/etcdv:3.2.4
  112. gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
  113. gcr.io/google_containers/pause-amd64:3.0
  115. - kubespray生成配置所需的环境(python3 ansible) Ansible v2.4 (or newer) Jinja 2.9 (or newer)
  116. yum install python34 python34-pip python-pip  python-netaddr -y
  117. cd
  118. mkdir .pip
  119. cd .pip
  120. cat > pip.conf <<EOF
  121. [global]
  122. index-url = http://mirrors.aliyun.com/pypi/simple/
  124. [install]
  125. trusted-host=mirrors.aliyun.com
  126. EOF
  128. yum install gcc libffi-devel python-devel openssl-devel -y
  129. pip install  Jinja2-2.10-py2.py3-none-any.whl # https://pypi.python.org/pypi/Jinja2
  130. pip install cryptography
  131. pip install ansible
  133. - 克隆
  134. git clone https://github.com/kubernetes-incubator/kubespray.git
  136. - 修改配置
  138. 1. 使docker能pull gcr的镜像
  139. vim inventory/group_vars/all.yml
  140. 2 bootstrap_os: centos
  141. 95 http_proxy: "http://192.168.1.88:1080/"
  143. 2. 如果vm内存<=1G,如果>=3G,则无需修改
  144. vim roles/kubernetes/preinstall/tasks/verify-settings.yml
  145. 52 - name: Stop if memory is too small for masters
  146. 53   assert:
  147. 54     that: ansible_memtotal_mb <= 1500
  148. 55   ignore_errors: "{{ ignore_assert_errors }}"
  149. 56   when: inventory_hostname in groups['kube-master']
  150. 57
  151. 58 - name: Stop if memory is too small for nodes
  152. 59   assert:
  153. 60     that: ansible_memtotal_mb <= 1024
  154. 61   ignore_errors: "{{ ignore_assert_errors }}"
  156. 3. 修改swap,
  157. vim roles/download/tasks/download_container.yml
  158.  75 - name: Stop if swap enabled
  159.  76   assert:
  160.  77     that: ansible_swaptotal_mb == 0
  161.  78   when: kubelet_fail_swap_on|default(false)
  163. 所有机器执行: 关闭swap
  164. swapoff -a
  165. [root@n1 kubespray]# free -m
  166.               total        used        free      shared  buff/cache   available
  167. Mem:           2796         297        1861           8         637        2206
  168. Swap:             0           0           0  #这栏为0,表示关闭
  170. - 生成 kubespray配置,开始ansible安装k8s之旅(非常用时间,大到1h,小到20min)
  171. cd kubespray
  172. IPS=(192.168.2.11 192.168.2.12 192.168.2.13 192.168.2.14 192.168.2.15)
  173. CONFIG_FILE=inventory/inventory.cfg python3 contrib/inventory_builder/inventory.py ${IPS[@]}
  174. ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v --private-key=~/.ssh/id_rsa
  176. - 修改好的配置
  177. git clone https://github.com/lannyMa/kubespray
  178. run.sh 有启动命令
  180. - 我的inventory.cfg, 我不想让node和master混在一起手动改了下
  181. [root@n1 kubespray]# cat inventory/inventory.cfg
  182. [all]
  183. node1 	 ansible_host=192.168.2.11 ip=192.168.2.11
  184. node2 	 ansible_host=192.168.2.12 ip=192.168.2.12
  185. node3 	 ansible_host=192.168.2.13 ip=192.168.2.13
  186. node4 	 ansible_host=192.168.2.14 ip=192.168.2.14
  187. node5 	 ansible_host=192.168.2.15 ip=192.168.2.15
  189. [kube-master]
  190. node1
  191. node2
  192. node3
  194. [kube-node]
  195. node4
  196. node5 	 
  198. [etcd]
  199. node1
  200. node2
  201. node3 	 
  203. [k8s-cluster:children]
  204. kube-node
  205. kube-master 	 
  207. [calico-rr]

遇到的问题wait for the apiserver to be running

  1. ansible-playbook -i inventory/inventory.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa
  2. ...
  3. <!-- We recommend using snippets services like https://gist.github.com/ etc. -->
  4. RUNNING HANDLER [kubernetes/master : Master | wait for the apiserver to be running] ***
  5. Thursday 23 March 2017  10:46:16 +0800 (0:00:00.468)       0:08:32.094 ********
  6. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (10 retries left).
  7. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (10 retries left).
  8. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (9 retries left).
  9. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (9 retries left).
  10. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (8 retries left).
  11. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (8 retries left).
  12. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (7 retries left).
  13. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (7 retries left).
  14. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (6 retries left).
  15. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (6 retries left).
  16. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (5 retries left).
  17. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (5 retries left).
  18. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (4 retries left).
  19. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (4 retries left).
  20. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (3 retries left).
  21. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (3 retries left).
  22. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (2 retries left).
  23. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (2 retries left).
  24. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (1 retries left).
  25. FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (1 retries left).
  26. fatal: [node1]: FAILED! => {"attempts": 10, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:8080/healthz"}
  27. fatal: [node2]: FAILED! => {"attempts": 10, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:8080/healthz"}
  28.         to retry, use: --limit @/home/dev_dean/kargo/cluster.retry

解决: 所有节点关闭swap

  1. swapoff -a

快捷命令

  1. alias kk='kubectl get pod --all-namespaces -o wide --show-labels'
  2. alias ks='kubectl get svc --all-namespaces -o wide'
  3. alias kss='kubectl get svc --all-namespaces -o wide --show-labels'
  4. alias kd='kubectl get deploy --all-namespaces -o wide'
  5. alias wk='watch kubectl get pod --all-namespaces -o wide --show-labels'
  6. alias kv='kubectl get pv -o wide'
  7. alias kvc='kubectl get pvc -o wide --all-namespaces --show-labels'
  8. alias kbb='kubectl run -it --rm --restart=Never busybox --image=busybox sh'
  9. alias kbbc='kubectl run -it --rm --restart=Never curl --image=appropriate/curl sh'
  10. alias kd='kubectl get deployment --all-namespaces --show-labels'
  11. alias kcm='kubectl get cm --all-namespaces -o wide'
  12. alias kin='kubectl get ingress --all-namespaces -o wide'

kubespray的默认启动参数

  1. ps -ef|egrep  "apiserver|controller-manager|scheduler"
  3. /hyperkube apiserver \
  4. --advertise-address=192.168.2.11 \
  5. --etcd-servers=https://192.168.2.11:2379,https://192.168.2.12:2379,https://192.168.2.13:2379 \
  6. --etcd-quorum-read=true \
  7. --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
  8. --etcd-certfile=/etc/ssl/etcd/ssl/node-node1.pem \
  9. --etcd-keyfile=/etc/ssl/etcd/ssl/node-node1-key.pem \
  10. --insecure-bind-address=127.0.0.1 \
  11. --bind-address=0.0.0.0 \
  12. --apiserver-count=3 \
  13. --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ValidatingAdmissionWebhook,ResourceQuota \
  14. --service-cluster-ip-range=10.233.0.0/18 \
  15. --service-node-port-range=30000-32767 \
  16. --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  17. --profiling=false \
  18. --repair-malformed-updates=false \
  19. --kubelet-client-certificate=/etc/kubernetes/ssl/node-node1.pem \
  20. --kubelet-client-key=/etc/kubernetes/ssl/node-node1-key.pem \
  21. --service-account-lookup=true \
  22. --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \
  23. --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
  24. --proxy-client-cert-file=/etc/kubernetes/ssl/apiserver.pem \
  25. --proxy-client-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
  26. --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
  27. --secure-port=6443 \
  28. --insecure-port=8080
  29. --storage-backend=etcd3 \
  30. --runtime-config=admissionregistration.k8s.io/v1alpha1 --v=2 \
  31. --allow-privileged=true \
  32. --anonymous-auth=False \
  33. --authorization-mode=Node,RBAC \
  34. --feature-gates=Initializers=False \
  35. PersistentLocalVolumes=False
  37. /hyperkube controller-manager \
  38. --kubeconfig=/etc/kubernetes/kube-controller-manager-kubeconfig.yaml \
  39. --leader-elect=true \
  40. --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
  41. --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  42. --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  43. --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  44. --enable-hostpath-provisioner=false \
  45. --node-monitor-grace-period=40s \
  46. --node-monitor-period=5s \
  47. --pod-eviction-timeout=5m0s \
  48. --profiling=false \
  49. --terminated-pod-gc-threshold=12500 \
  50. --v=2 \
  51. --use-service-account-credentials=true \
  52. --feature-gates=Initializers=False \
  53. PersistentLocalVolumes=False
  55. /hyperkube scheduler \
  56. --leader-elect=true \
  57. --kubeconfig=/etc/kubernetes/kube-scheduler-kubeconfig.yaml \
  58. --profiling=false --v=2 \
  59. --feature-gates=Initializers=False \
  60. PersistentLocalVolumes=False

  2. /usr/local/bin/kubelet \
  3. --logtostderr=true --v=2 \
  4. --address=192.168.2.14 \
  5. --node-ip=192.168.2.14 \
  6. --hostname-override=node4 \
  7. --allow-privileged=true \
  8. --pod-manifest-path=/etc/kubernetes/manifests \
  9. --cadvisor-port=0 \
  10. --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
  11. --node-status-update-frequency=10s \
  12. --docker-disable-shared-pid=True \
  13. --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  14. --tls-cert-file=/etc/kubernetes/ssl/node-node4.pem \
  15. --tls-private-key-file=/etc/kubernetes/ssl/node-node4-key.pem \
  16. --anonymous-auth=false \
  17. --cgroup-driver=cgroupfs \
  18. --cgroups-per-qos=True \
  19. --fail-swap-on=True \
  20. --enforce-node-allocatable= \
  21. --cluster-dns=10.233.0.3 \
  22. --cluster-domain=cluster.local \
  23. --resolv-conf=/etc/resolv.conf \
  24. --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml \
  25. --require-kubeconfig \
  26. --kube-reserved cpu=100m,memory=256M \
  27. --node-labels=node-role.kubernetes.io/node=true \
  28. --feature-gates=Initializers=False,PersistentLocalVolumes=False \
  29. --network-plugin=cni --cni-conf-dir=/etc/cni/net.d \
  30. --cni-bin-dir=/opt/cni/bin
  32. ps -ef|grep kube-porxy
  33. hyperkube proxy --v=2 \
  34.     --kubeconfig=/etc/kubernetes/kube-proxy-kubeconfig.yaml\
  35.     --bind-address=192.168.2.14 \
  36.     --cluster-cidr=10.233.64.0/18 \
  37.     --proxy-mode=iptables

我把所需要的镜像v1.9推送到dockerhub了

  1. lanny/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.1.1
  2. lanny/gcr.io_google_containers_pause-amd64:3.0
  3. lanny/gcr.io_google_containers_k8s-dns-kube-dns-amd64:1.14.7
  4. lanny/gcr.io_google_containers_k8s-dns-dnsmasq-nanny-amd64:1.14.7
  5. lanny/gcr.io_google_containers_k8s-dns-sidecar-amd64:1.14.7
  6. lanny/gcr.io_google_containers_elasticsearch:v2.4.1
  7. lanny/gcr.io_google_containers_fluentd-elasticsearch:1.22
  8. lanny/gcr.io_google_containers_kibana:v4.6.1
  9. lanny/gcr.io_kubernetes-helm_tiller:v2.7.2
  10. lanny/gcr.io_google_containers_kubernetes-dashboard-init-amd64:v1.0.1
  11. lanny/gcr.io_google_containers_kubernetes-dashboard-amd64:v1.7.1
  12. lanny/quay.io_l23network_k8s-netchecker-agent:v1.0
  13. lanny/quay.io_l23network_k8s-netchecker-server:v1.0
  14. lanny/quay.io_coreos_etcd:v3.2.4
  15. lanny/quay.io_coreos_flannel:v0.9.1
  16. lanny/quay.io_coreos_flannel-cni:v0.3.0
  17. lanny/quay.io_calico_ctl:v1.6.1
  18. lanny/quay.io_calico_node:v2.6.2
  19. lanny/quay.io_calico_cni:v1.11.0
  20. lanny/quay.io_calico_kube-controllers:v1.0.0
  21. lanny/quay.io_calico_routereflector:v0.4.0
  22. lanny/quay.io_coreos_hyperkube:v1.9.0_coreos.0
  23. lanny/quay.io_ant31_kargo:master
  24. lanny/quay.io_external_storage_local-volume-provisioner-bootstrap:v1.0.0
  25. lanny/quay.io_external_storage_local-volume-provisioner:v1.0.0
  27. gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
  28. gcr.io/google_containers/pause-amd64:3.0
  29. gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
  30. gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
  31. gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
  32. gcr.io/google_containers/elasticsearch:v2.4.1
  33. gcr.io/google_containers/fluentd-elasticsearch:1.22
  34. gcr.io/google_containers/kibana:v4.6.1
  35. gcr.io/kubernetes-helm/tiller:v2.7.2
  36. gcr.io/google_containers/kubernetes-dashboard-amd64:v1.0.1
  37. gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.7.1
  38. quay.io/l23network/k8s-netchecker-agent:v1.0
  39. quay.io/l23network/k8s-netchecker-server:v1.0
  40. quay.io/coreos/etcd:v3.2.4
  41. quay.io/coreos/flannel:v0.9.1
  42. quay.io/coreos/flannel-cni:v0.3.0
  43. quay.io/calico/ctl:v1.6.1
  44. quay.io/calico/node:v2.6.2
  45. quay.io/calico/cni:v1.11.0
  46. quay.io/calico/kube-controllers:v1.0.0
  47. quay.io/calico/routereflector:v0.4.0
  48. quay.io/coreos/hyperkube:v1.9.0_coreos.0
  49. quay.io/ant31/kargo:master
  50. quay.io/external_storage/local-volume-provisioner-bootstrap:v1.0.0
  51. quay.io/external_storage/local-volume-provisioner:v1.0.0
  53. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/download/defaults/main.yml
  54. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
  55. sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/kubernetes-apps/ansible/defaults/main.yml
  56. sed -i 's#gcr\.io\/kubernetes-helm\/#lanny/gcr\.io_kubernetes-helm_#g' roles/download/defaults/main.yml
  58. sed -i 's#quay\.io\/coreos\/#lanny/quay\.io_coreos_#g' roles/download/defaults/main.yml
  59. sed -i 's#quay\.io\/calico\/#lanny/quay\.io_calico_#g' roles/download/defaults/main.yml
  60. sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' roles/download/defaults/main.yml
  61. sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' docs/netcheck.md
  62. sed -i 's#quay\.io\/external_storage\/#lanny/quay\.io_external_storage_#g' roles/kubernetes-apps/local_volume_provisioner/defaults/main.yml
  63. sed -i 's#quay\.io\/ant31\/kargo#lanny/quay\.io_ant31_kargo_#g' .gitlab-ci.yml

参考: https://jicki.me/2017/12/08/kubernetes-kubespray-1.8.4/

构建etcd容器集群

  1. /usr/bin/docker run
  2.     --restart=on-failure:5
  3.     --env-file=/etc/etcd.env
  4.     --net=host
  5.     -v /etc/ssl/certs:/etc/ssl/certs:ro
  6.     -v /etc/ssl/etcd/ssl:/etc/ssl/etcd/ssl:ro
  7.     -v /var/lib/etcd:/var/lib/etcd:rw
  8.     --memory=512M
  9.     --oom-kill-disable
  10.     --blkio-weight=1000
  11.     --name=etcd1
  12.     lanny/quay.io_coreos_etcd:v3.2.4
  13.     /usr/local/bin/etcd
  15. [root@node1 ~]# cat /etc/etcd.env
  16. ETCD_DATA_DIR=/var/lib/etcd
  17. ETCD_ADVERTISE_CLIENT_URLS=https://192.168.2.11:2379
  18. ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.2.11:2380
  19. ETCD_INITIAL_CLUSTER_STATE=existing
  20. ETCD_METRICS=basic
  21. ETCD_LISTEN_CLIENT_URLS=https://192.168.2.11:2379,https://127.0.0.1:2379
  22. ETCD_ELECTION_TIMEOUT=5000
  23. ETCD_HEARTBEAT_INTERVAL=250
  24. ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
  25. ETCD_LISTEN_PEER_URLS=https://192.168.2.11:2380
  26. ETCD_NAME=etcd1
  27. ETCD_PROXY=off
  28. ETCD_INITIAL_CLUSTER=etcd1=https://192.168.2.11:2380,etcd2=https://192.168.2.12:2380,etcd3=https://192.168.2.13:2380
  29. ETCD_AUTO_COMPACTION_RETENTION=8
  31. # TLS settings
  32. ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
  33. ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
  34. ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
  35. ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
  36. ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
  37. ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
  38. ETCD_PEER_CLIENT_CERT_AUTH=true
  40. [root@node1 ssl]# docker exec b1159a1c6209 env|grep -i etcd
  41. ETCD_DATA_DIR=/var/lib/etcd
  42. ETCD_ADVERTISE_CLIENT_URLS=https://192.168.2.11:2379
  43. ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.2.11:2380
  44. ETCD_INITIAL_CLUSTER_STATE=existing
  45. ETCD_METRICS=basic
  46. ETCD_LISTEN_CLIENT_URLS=https://192.168.2.11:2379,https://127.0.0.1:2379
  47. ETCD_ELECTION_TIMEOUT=5000
  48. ETCD_HEARTBEAT_INTERVAL=250
  49. ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
  50. ETCD_LISTEN_PEER_URLS=https://192.168.2.11:2380
  51. ETCD_NAME=etcd1
  52. ETCD_PROXY=off
  53. ETCD_INITIAL_CLUSTER=etcd1=https://192.168.2.11:2380,etcd2=https://192.168.2.12:2380,etcd3=https://192.168.2.13:2380
  54. ETCD_AUTO_COMPACTION_RETENTION=8
  55. ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
  56. ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
  57. ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
  58. ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
  59. ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
  60. ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
  61. ETCD_PEER_CLIENT_CERT_AUTH=true
  63. [root@node1 ssl]# tree /etc/ssl/etcd/ssl
  64. /etc/ssl/etcd/ssl
  65. ├── admin-node1-key.pem
  66. ├── admin-node1.pem
  67. ├── admin-node2-key.pem
  68. ├── admin-node2.pem
  69. ├── admin-node3-key.pem
  70. ├── admin-node3.pem
  71. ├── ca-key.pem
  72. ├── ca.pem
  73. ├── member-node1-key.pem
  74. ├── member-node1.pem
  75. ├── member-node2-key.pem
  76. ├── member-node2.pem
  77. ├── member-node3-key.pem
  78. ├── member-node3.pem
  79. ├── node-node1-key.pem
  80. ├── node-node1.pem
  81. ├── node-node2-key.pem
  82. ├── node-node2.pem
  83. ├── node-node3-key.pem
  84. ├── node-node3.pem
  85. ├── node-node4-key.pem
  86. ├── node-node4.pem
  87. ├── node-node5-key.pem
  88. └── node-node5.pem
  90. [root@n2 ~]# tree /etc/ssl/etcd/ssl
  91. /etc/ssl/etcd/ssl
  92. ├── admin-node2-key.pem
  93. ├── admin-node2.pem
  94. ├── ca-key.pem
  95. ├── ca.pem
  96. ├── member-node2-key.pem
  97. ├── member-node2.pem
  98. ├── node-node1-key.pem
  99. ├── node-node1.pem
  100. ├── node-node2-key.pem
  101. ├── node-node2.pem
  102. ├── node-node3-key.pem
  103. ├── node-node3.pem
  104. ├── node-node4-key.pem
  105. ├── node-node4.pem
  106. ├── node-node5-key.pem
  107. └── node-node5.pem
  109. admin-node1.crt
  110. CN = etcd-admin-node1
  112. DNS Name=localhost
  113. DNS Name=node1
  114. DNS Name=node2
  115. DNS Name=node3
  116. IP Address=192.168.2.11
  117. IP Address=192.168.2.11
  118. IP Address=192.168.2.12
  119. IP Address=192.168.2.12
  120. IP Address=192.168.2.13
  121. IP Address=192.168.2.13
  122. IP Address=127.0.0.1
  124. member-node1.crt
  125. CN = etcd-member-node1
  127. DNS Name=localhost
  128. DNS Name=node1
  129. DNS Name=node2
  130. DNS Name=node3
  131. IP Address=192.168.2.11
  132. IP Address=192.168.2.11
  133. IP Address=192.168.2.12
  134. IP Address=192.168.2.12
  135. IP Address=192.168.2.13
  136. IP Address=192.168.2.13
  137. IP Address=127.0.0.1
  139. node-node1.crt
  140. CN = etcd-node-node1
  142. DNS Name=localhost
  143. DNS Name=node1
  144. DNS Name=node2
  145. DNS Name=node3
  146. IP Address=192.168.2.11
  147. IP Address=192.168.2.11
  148. IP Address=192.168.2.12
  149. IP Address=192.168.2.12
  150. IP Address=192.168.2.13
  151. IP Address=192.168.2.13
  152. IP Address=127.0.0.1
  154. [root@node1 bin]# tree
  155. .
  156. ├── etcd
  157. ├── etcdctl
  158. ├── etcd-scripts
  159. │   └── make-ssl-etcd.sh
  160. ├── kubectl
  161. ├── kubelet
  162. └── kubernetes-scripts
  163.     ├── kube-gen-token.sh
  164.     └── make-ssl.sh

[k8s]kubespray(ansible)自动化安装k8s集群的更多相关文章

  1. Redis自动化安装以及集群实现

    Redis实例安装 安装说明:自动解压缩安装包,按照指定路径编译安装,复制配置文件模板到Redis实例路的数据径下,根据端口号修改配置文件模板 三个必须文件:1,配置文件,2,当前shell脚本,3, ...

  2. 利用ansible书写playbook在华为云上批量配置管理工具自动化安装ceph集群

    首先在华为云上购买搭建ceph集群所需云主机: 然后购买ceph所需存储磁盘 将购买的磁盘挂载到用来搭建ceph的云主机上 在跳板机上安装ansible 查看ansible版本,检验ansible是否 ...

  3. Ansible自动化部署K8S集群

    Ansible自动化部署K8S集群 1.1 Ansible介绍 Ansible是一种IT自动化工具.它可以配置系统,部署软件以及协调更高级的IT任务,例如持续部署,滚动更新.Ansible适用于管理企 ...

  4. Kubernetes(K8s) 安装（使用kubeadm安装Kubernetes集群）

    背景: 由于工作发生了一些变动,很长时间没有写博客了. 概述: 这篇文章是为了介绍使用kubeadm安装Kubernetes集群(可以用于生产级别).使用了Centos 7系统. 一.Centos7 ...

  5. k8s中安装rabbitmq集群

    官方文档地址:https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html 要求 1.k8s版本要1.18及其以上 2.能 ...

  6. 利用ansible进行自动化构建etcd集群

    上一篇进行了手动安装etcd集群,此篇利用自动化工具ansible为三个节点构建etcd集群 环境: master:192.168.101.14,node1:192.168.101.15,node2: ...

  7. 简单了解一下K8S，并搭建自己的集群

    距离上次更新已经有一个月了,主要是最近工作上的变动有点频繁,现在才暂时稳定下来.这篇博客的本意是带大家从零开始搭建K8S集群的.但是我后面一想,如果是我看了这篇文章,会收获什么?就是跟着步骤一步一走吗 ...

  8. lvs+keepalived部署k8s v1.16.4高可用集群

    一.部署环境 1.1 主机列表 主机名 Centos版本 ip docker version flannel version Keepalived version 主机配置 备注 lvs-keepal ...

  9. Centos7.6部署k8s v1.16.4高可用集群(主备模式)

    一.部署环境 主机列表: 主机名 Centos版本 ip docker version flannel version Keepalived version 主机配置 备注 master01 7.6. ...

随机推荐

  1. JavaScript 之 截取字符串函数

    一.函数:split() 功能:使用一个指定的分隔符把一个字符串分割存储到数组 例子: str=”jpg|bmp|gif|ico|png”; arr=theString.split(”|”); //a ...

  2. ubuntu 软件包管理工具 dpkg,apt-get,aptitude 区别

    ubuntu 软件包管理工具 dpkg,apt-get,aptitude 区别 一:dpkg dpkg 是一种比较低层的软件包安装管理工具,在安装时,不会安装软件包的依赖关系:只能安装所要求的软件包: ...

  3. gradle 项目转成maven项目

    找到一个个子项目目录下的build.gradle文件,在文件开头添加以下内容: apply plugin: 'java' apply plugin: 'maven' compileJava.optio ...

  4. 谈谈Boost网络编程（2）—— 新系统的设计

    写文章之前.我们一般会想要採用何种方式,是"开门见山",还是"疑问式开头".写代码也有些类似.在编码之前我们须要考虑系统总体方案,这也就是各种设计文档的作用.在 ...

  5. MySQL的各种SHOW

    . SHOW语法 13.5.4.1. SHOW CHARACTER SET语法 13.5.4.2. SHOW COLLATION语法 13.5.4.3. SHOW COLUMNS语法 13.5.4.4 ...

  6. Oracle 11g 分区拆分与合并

    时间范围分区拆分create table emp (id number(6) not null,hire_date date not null)partition by range(hire_date ...

  7. 〖Linux〗apt-get wait for another apt process

    #!/bin/bash i= tput sc >& || \ >&; do )) in ) j="-" ;; ) j="\\" ;; ...

  8. Linux下通用线程池的创建与使用

    线程池:简单地说,线程池 就是预先创建好一批线程,方便.快速地处理收到的业务.比起传统的到来一个任务,即时创建一个线程来处理,节省了线程的创建和回收的开销,响应更快,效率更高. 在linux中,使用的 ...

  9. WinForm如何调用Web Service

    参考地址 今天看了李天平关于WinForm调用Web Service的代码,我自己模仿做一个代码基本都是复制粘贴的,结果不好使.郁闷的是,又碰到那个该死的GET调用Web Service,我想肯定又是 ...

  10. HDUOJ--1874 畅通工程续

    畅通工程续 Time Limit: 3000/1000 MS (Java/Others)    Memory Limit: 32768/32768 K (Java/Others)Total Submi ...