实验环境介绍:

本次实验环境是5个节点 3台master 2台node节点:

k8smaster01 192.168.111.128 软件:etcd k8smaster haproxy keepalived
k8smaster02 192.168.111.129 软件:etcd k8smaster haproxy keepalived
k8smaster03 192.168.111.130 软件:etcd k8smaster haproxy keepalived
k8snode01 192.168.111.131 软件:k8snode
k8snode02 192.168.111.132 软件:k8snode VIP: 192.168.111.100

系统优化(在所有节点上操作)

关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

关闭SELINUX和swap,优化内核参数

 sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0 # 临时关闭swap
# 永久关闭 注释/etc/fstab文件里swap相关的行
swapoff -a # 配置转发相关参数,否则可能会出错
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system # 加载ipvs相关内核模块
# 如果重新开机,需要重新加载
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs

配置yum源

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=
gpgcheck=
repo_gpgcheck=
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF yum clean all && yum makecache sudo yum install -y yum-utils device-mapper-persistent-data lvm2

配置hosts解析

192.168.111.128 k8smaster01
192.168.111.129 k8smaster02
192.168.111.130 k8smaster03
192.168.111.131 k8snode01
192.168.111.132 k8snode02

安装docker

v1.11.1版本推荐使用docker v17.03,v1.11,v1.12,v1.13, 也可以使用,再高版本官网不推荐使用,但是可以忽略。

这里安装18.06.0-ce

yum -y install docker-ce
systemctl enable docker && systemctl restart docker

安装 kubeadm, kubelet 和 kubectl(所有节点)

yum install -y kubelet kubeadm kubectl ipvsadm
systemctl enable kubelet && systemctl start kubelet

配置haproxy代理和keepalived(如下操作在所有master节点上操作)

# 拉取haproxy镜像
docker pull haproxy:1.7.-alpine
cat >/etc/haproxy/haproxy.cfg<<EOF
global
log 127.0.0.1 local0 err
maxconn
uid
gid
#daemon
nbproc
pidfile haproxy.pid defaults
mode http
log 127.0.0.1 local0 err
maxconn
retries
timeout connect 5s
timeout client 30s
timeout server 30s
timeout check 2s listen admin_stats
mode http
bind 0.0.0.0:
log 127.0.0.1 local0 err
stats refresh 30s
stats uri /haproxy-status
stats realm Haproxy\ Statistics
stats auth will:will
stats hide-version
stats admin if TRUE frontend k8s-https
bind 0.0.0.0:
mode tcp
#maxconn
default_backend k8s-https backend k8s-https
mode tcp
balance roundrobin
server k8smaster01 192.168.111.128: weight maxconn check inter rise fall
server k8smaster02 192.168.111.129: weight maxconn check inter rise fall
server k8smaster03 192.168.111.130: weight maxconn check inter rise fall
EOF
# 启动haproxy
docker run -d --name my-haproxy \
-v /etc/haproxy:/usr/local/etc/haproxy:ro \
-p : \
-p : \
--restart always \
haproxy:1.7.-alpine
# 拉取keepalived镜像
docker pull osixia/keepalived:1.4. # 启动
# 载入内核相关模块
lsmod | grep ip_vs
modprobe ip_vs # 启动keepalived
# ens33为本次实验192.168.111./24网段的所在网卡
docker run --net=host --cap-add=NET_ADMIN \
-e KEEPALIVED_INTERFACE=ens33 \
-e KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['192.168.111.100']" \
-e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.111.128','192.168.111.129','192.168.111.130']" \
-e KEEPALIVED_PASSWORD=hello \
--name k8s-keepalived \
--restart always \
-d osixia/keepalived:1.4. # 此时会配置 192.168.111.100 到其中一台机器
# ping测试
ping 192.168.111.100 # 如果失败后清理后,重新实验
#docker rm -f k8s-keepalived
#ip a del 192.168.111.100/ dev ens33

配置kubelet(所有节点操作)

# 配置kubelet使用国内pause镜像
# 配置kubelet的cgroups cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
EOF # 启动
systemctl daemon-reload
systemctl enable kubelet && systemctl restart kubelet

配置k8smaster01(192.168.111.128上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1" api:
advertiseAddress: 192.168.111.128
controlPlaneEndpoint: 192.168.111.100: etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.128:2379"
advertise-client-urls: "https://192.168.111.128:2379"
listen-peer-urls: "https://192.168.111.128:2380"
initial-advertise-peer-urls: "https://192.168.111.128:2380"
initial-cluster: "k8smaster01=https://192.168.111.128:2380"
serverCertSANs:
- k8smaster01
- 192.168.111.128
peerCertSANs:
- k8smaster01
- 192.168.111.128 controllerManagerExtraArgs:
node-monitor-grace-period: 10s
pod-eviction-timeout: 10s networking:
podSubnet: 10.244.0.0/ kubeProxy:
config:
mode: ipvs
# mode: iptables
EOF # 提前拉取镜像
# 如果执行失败 可以多次执行
kubeadm config images pull --config kubeadm-master.config # 初始化
# 注意保存返回的 join 命令
kubeadm init --config kubeadm-master.config # 初始化失败时使用
#kubeadm reset # 将ca相关文件传至其他master节点
cd /etc/kubernetes/pki/
USER=root
CONTROL_PLANE_IPS="k8smaster02 k8smaster03"
for host in ${CONTROL_PLANE_IPS}; do
   ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
scp ca.crt ca.key  sa.key  sa.pub front-proxy-ca.crt front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/
scp etcd/ca.crt etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/
  scp ../admin.conf "${USER}"@$host:/etc/kubernetes/ done

kubeadm init失败解决:

将阿里云image tag成官方的image,即可解决init失败问题。(v1.11.0有此问题)

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.2. k8s.gcr.io/etcd-amd64:3.2.
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.1. k8s.gcr.io/coredns:1.1.
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1

配置k8smaster02(192.168.111.129上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1" api:
advertiseAddress: 192.168.111.129
controlPlaneEndpoint: 192.168.111.100: etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.129:2379"
advertise-client-urls: "https://192.168.111.129:2379"
listen-peer-urls: "https://192.168.111.129:2380"
initial-advertise-peer-urls: "https://192.168.111.129:2380"
initial-cluster: "k8smaster01=https://192.168.111.128:2380,k8smaster02=https://192.168.111.129:2380"
initial-cluster-state: existing
serverCertSANs:
- k8smaster02
- 192.168.111.129
peerCertSANs:
- k8smaster02
- 192.168.111.129 controllerManagerExtraArgs:
node-monitor-grace-period: 10s
pod-eviction-timeout: 10s networking:
podSubnet: 10.244.0.0/ kubeProxy:
config:
mode: ipvs
# mode: iptables
EOF # 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet # 添加etcd到集群中
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-k8smaster01 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://192.168.111.128:2379 member add k8smaster02 https://192.168.111.129:2380
kubeadm alpha phase etcd local --config kubeadm-master.config
# 提前拉取镜像
kubeadm config images pull --config kubeadm-master.config # 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

配置k8smaster03(192.168.111.130上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1" api:
advertiseAddress: 192.168.111.130
controlPlaneEndpoint: 192.168.111.100: etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.130:2379"
advertise-client-urls: "https://192.168.111.130:2379"
listen-peer-urls: "https://192.168.111.130:2380"
initial-advertise-peer-urls: "https://192.168.111.130:2380"
initial-cluster: "k8smaster01=https://192.168.111.128:2380,k8smaster02=https://192.168.111.129:2380,k8smaster03=https://192.168.111.130:2380"
initial-cluster-state: existing
serverCertSANs:
- k8smaster03
- 192.168.111.130
peerCertSANs:
- k8smaster03
- 192.168.111.130 controllerManagerExtraArgs:
node-monitor-grace-period: 10s
pod-eviction-timeout: 10s networking:
podSubnet: 10.244.0.0/ kubeProxy:
config:
mode: ipvs
# mode: iptables
EOF
# 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet # 添加etcd到集群中
KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-k8smaster01 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://192.168.111.128:2379 member add k8smaster03 https://192.168.111.130:2380
kubeadm alpha phase etcd local --config kubeadm-master.config
# 提前拉取镜像
kubeadm config images pull --config kubeadm-master.config # 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

配置使用kubectl (master 任意节点执行)

rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 查看node节点
 kubectl get nodes
 
# 只有网络插件也安装配置完成之后,才能会显示为ready状态
# 设置master允许部署应用pod,参与工作负载,现在可以部署其他系统组件

配置使用网络插件(任意master节点上操作)

# 下载配置
cd /etc/kubernetes
mkdir flannel && cd flannel
wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml # 修改配置
# 此处的ip配置要与上面kubeadm的pod-network一致
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
} # 修改镜像
image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64 # 如果Node有多个网卡的话,参考flannel issues ,
# https://github.com/kubernetes/kubernetes/issues/39701
# 目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称,
# 否则可能会出现dns无法解析。容器无法通信的情况,需要将kube-flannel.yml下载到本地,
# flanneld启动参数加上--iface=<iface-name>
containers:
- name: kube-flannel
image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=ens33 # 启动
kubectl apply -f kube-flannel.yml # 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

配置node节点加入集群(所有的node节点上操作)

以下上master生成的,与你环境可能不符合  
kubeadm join 192.168.111.100: --token uf9oul.7k4csgxe5p7upvdb --discovery-token-ca-cert-hash sha256:36bc173b46eb0545fc30dd5db2d27dab70a257bd406fd791647d991a69454595

node节点报错处理办法:

tail -f /var/log/message
Jul 19 07:52:21 localhost kubelet: E0726 07:52:21.336281   10018 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

在kubelet配置文件追加以下配置

/etc/sysconfig/kubelet

# Append configuration in Kubelet
--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

这样一个集群环境配置完成里,其余的是自己添加附件吧。

Centos7使用kubeadm 安装多主高可用kubernets:v.1.11集群的更多相关文章

  1. 通过 Kubeadm 安装 K8S 与高可用,版本1.13.4

    环境介绍: CentOS: 7.6 Docker: 18.06.1-ce Kubernetes: 1.13.4 Kuberadm: 1.13.4 Kuberlet: 1.13.4 Kuberctl: ...

  2. 高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群

    高性能Linux服务器 第11章 构建高可用的LVS负载均衡集群 libnet软件包<-依赖-heartbeat(包含ldirectord插件(需要perl-MailTools的rpm包)) l ...

  3. 高可用,多路冗余GFS2集群文件系统搭建详解

    高可用,多路冗余GFS2集群文件系统搭建详解 2014.06 标签:GFS2 multipath 集群文件系统 cmirror 实验拓扑图: 实验原理: 实验目的:通过RHCS集群套件搭建GFS2集群 ...

  4. 使用kubeadm搭建高可用k8s v1.16.3集群

    目录 1.部署环境说明 2.集群架构及部署准备工作 2.1.集群架构说明 2.2.修改hosts及hostname 2.3.其他准备 3.部署keepalived 3.1.安装 3.2.配置 3.3. ...

  5. CentOS7.4下安装部署HAProxy高可用群集

    目录第一部分 实验环境第二部分 搭建配置web服务器第三部分 安装配置haproxy服务器第四部分 测试验证第五部分 haproxy配置相关详细解释 第一部分 实验环境1.一台harpoxy调度服务器 ...

  6. 使用Nginx+Keepalived组建高可用负载平衡Web server集群

    一,首先说明一下网络拓扑结构: 1,Nginx 反向代理Server(HA):     ①Nginx master:192.168.1.157     ②Nginx backup:192.168.1. ...

  7. LVS + Keepalived 实现高可用、负载均衡 Web 集群

    简介: LVS 是 Linux Virtual Server 的简写,Linux 虚拟服务器的意思,是一个虚拟的服务器集群系统,此项目由章文嵩博士于 1998 年 5 月成立,是中国最早出现的自由软件 ...

  8. 高可用Hadoop平台-HBase集群搭建

    1.概述 今天补充一篇HBase集群的搭建,这个是高可用系列遗漏的一篇博客,今天抽时间补上,今天给大家介绍的主要内容目录如下所示: 基础软件的准备 HBase介绍 HBase集群搭建 单点问题验证 截 ...

  9. 华为云:实现高可用的负载均衡web集群

    华为云: 2台云主机做负载均衡调度 >>申请一个虚拟浮动ip,并绑定一个弹性公网ip >>将两台云主机绑定到虚拟浮动ip上 3台web服务器 1台云服务器做jumpserver ...

随机推荐

  1. 使用binlog,实现MySQL数据恢复

    mysql的binlog日志,用于记录数据库的增.删.改等修改操作,默认处于关闭状态.使用binlog实现数据恢复的条件为 1.binlog日志功能已开启 2.若binlog在数据库创建一段时候后开启 ...

  2. 炎黄流程中改流程节点颜色的js

  3. 数据库 -- mysql支持的数据类型

    mysql支持的数据类型 数值类型 MySQL支持所有标准SQL数值数据类型. 这些类型包括严格数值数据类型(INTEGER.SMALLINT.DECIMAL和NUMERIC),以及近似数值数据类型( ...

  4. LGP2801 教主的魔法

    题目链接 : P2801 教主的魔法 这是第一次A分块的题 就是模板题了 每个块内排序 每个整块仅需维护整块的修改量 询问操作: 对于边缘块 直接暴力找在[l, r]内 且比给定值大的有几个 对于整块 ...

  5. Android undefined intent constructor错误?

    本文选自StackOverflow(简称:SOF)精选问答汇总系列文章之一,本系列文章将为读者分享国外最优质的精彩问与答,供读者学习和了解国外最新技术.在Android中启动Service时出现&qu ...

  6. cf1073D Berland Fair (二分答案+树状数组)

    用一个树状数组维护前缀和,每次我二分地找一个位置,使得我能一路买过去 但这个买不了 那以后肯定也都买不了了,就把它改成0,再从头二分地找下一个位置,直到这一圈我可以跑下来 然后就看跑这一圈要花多少钱. ...

  7. NOIP2013华容道(BFS+乱搞)

    n<=30 * m<=30 的地图上,0表示墙壁,1表示可以放箱子的空地.q<=500次询问,每次问:当空地上唯一没有放箱子的空格子在(ex,ey)时,把位于(sx,sy)的箱子移动 ...

  8. linux(fedora) 第三课

    树形打印所有进程名:pstree pstree | grep pstree -A2 -B2(查看pstree前后两行) NI的值[-20,20) nice(改变NI的值):改变程序优先级 nice - ...

  9. LCOW —— 单一Docker引擎下可同时运行Linux和Windows容器啦!

    https://blog.csdn.net/m2l0zgssvc7r69efdtj/article/details/79251059 就在上周,Docker官方的master分支上新增了LCOW(Li ...

  10. (1)Phonics自然拼读 英语动画 Fun with Phonics 国际主流英语教学法

    Phonics(拼音英语)是目前国际主流的英语教学法,我国香港和台湾地区2000年就已引进此教学法,并已进入大规模推广和普及阶段.它之所以风靡全球,是因为这种教学法简单高效,符合人类学习语言的规律,尤 ...