Ingress-Controller高可用方案及多租户场景(21)
一、Ingress-controller高可用
Ingress Controller 是集群流量的接入层，对它做高可用非常重要，可以基于 keepalive 实现 nginx-ingress-controller 高可用，具体实现如下：
Ingress-controller 根据 Deployment+ nodeSeletor+pod 反亲和性方式部署在 k8s 指定的两个 work 节点，nginx-ingress-controller 这个 pod 共享宿主机 ip，然后通过 keepalive+lvs 实现 nginx-ingress-controller 高可用
参考：https://github.com/kubernetes/ingress-nginx
https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/baremetal
#节点添加标签：
[root@master ingress]# kubectl label node node1 kubernetes.io/ingress=nginx
node/node1 labeled
[root@master ingress]# kubectl label node node3 kubernetes.io/ingress=nginx
node/node3 labeled # node1和node3节点导入镜像
[root@node3 ~]# docker load -i ingress-nginx-controllerv1.1.0.tar.gz
Loaded image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
[root@node3 ~]# docker load -i kube-webhook-certgen-v1.1.0.tar.gz
Loaded image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1 # ingress-deploy.yaml 官方下载，deploy但是添加了反亲和性
[root@master ingress]# kubectl apply -f ingress-deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created [root@master ingress]# kubectl get pods -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-5hwxc 0/1 Completed 0 2m48s 10.244.135.2 node3 <none> <none>
ingress-nginx-admission-patch-7r7nf 0/1 Completed 1 2m48s 10.244.135.1 node3 <none> <none>
ingress-nginx-controller-6c8ffbbfcf-bpllf 1/1 Running 0 2m48s 192.168.10.13 node3 <none> <none>
ingress-nginx-controller-6c8ffbbfcf-mzvzf 1/1 Running 0 2m48s 192.168.10.11 node1 <none> <none>
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx ---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: NodePort
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv4
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
hostNetwork: true
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
topologyKey: kubernetes.io/hostname
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: controller
image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
ingress-deploy.yaml
二、通过keepalive+nginx实现nginx-ingress-controller高可用
2.1 安装keepalived+nginx
#1.安装nginx主备
在node1和node3节点安装nginx和keepalived
yum install nginx keepalived nginx-mod-stream -y #2.修改nginx配置文件，注意下upstream
upstream k8s-apiserver {
server 192.168.10.11:80; # Master1 APISERVER IP:PORT
server 192.168.10.13:80; # Master2 APISERVER IP:PORT
} server {
listen 30080;
proxy_pass k8s-apiserver;
}
注意：nginx 监听端口变成大于 30000 的端口，比方说 30080,这样访问域名:30080 就可以了，必须是满足大于 30000 以上，才能代理 ingress-controller #3.keepalive配置
[root@node1 keepalived]# cat keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER # 备服改为：NGINX_BACKEND
} vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
} vrrp_instance VI_1 {
state MASTER # 备用修改为BACKUP
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.10.199/24
}
track_script {
check_nginx
}
} #vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
#virtual_ipaddress：虚拟IP（VIP） [root@node1 keepalived]# cat check_nginx.sh
#!/bin/bash
# 1.判断nginx是否存活
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]
then
# 2.如果不存活尝试启动nginx
service nginx start
sleep 2
# 3.等待2s后再次获取nginx状态
counter=`ps -C nginx --no-header | wc -l`
# 4.再次进行判断，如果nginx还不存活就停止keepalived,进行地址迁移
if [ $counter -eq 0 ]
then
service keepalived stop
fi
fi # 4.启动nginx和keepalived
systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived # 5.测试vip是否绑定成功
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:31:c4:c5 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.11/24 brd 192.168.10.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.10.199/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::22ce:531b:494:83c9/64 scope link noprefixroute
valid_lft forever preferred_lft forever # 6.测试keepalived
[root@node1 keepalived]# systemctl stop keepalived
[root@node3 keepalived]# ip addr
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:7e:66:de brd ff:ff:ff:ff:ff:ff
inet 192.168.10.13/24 brd 192.168.10.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.10.199/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::713e:adde:728a:e1ee/64 scope link noprefixroute
valid_lft forever preferred_lft forever [root@node1 keepalived]# systemctl start keepalived
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:31:c4:c5 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.11/24 brd 192.168.10.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.10.199/24 scope global secondary ens33
valid_lft forever preferred_lft forever
2.2 测试Ingress HTTP代理k8s内部站点
# 1.后端部署tomcat服务
[root@master ingress]# cat ingress-tomcat.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
spec:
selector:
app: tomcat
release: canary
ports:
- name: http
targetPort: 8080
port: 8080
- name: ajp
targetPort: 8009
port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deploy
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: tomcat
release: canary
template:
metadata:
labels:
app: tomcat
release: canary
spec:
containers:
- name: tomcat
image: tomcat:8.5.34-jre8-alpine
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
name: ajp
containerPort: 8009 [root@master ingress]# kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat-deploy-66b67fcf7b-jmlr4 1/1 Running 0 3m38s
tomcat-deploy-66b67fcf7b-qvzwt 1/1 Running 0 3m38s # 2.编写ingress规则
[root@master ingress]# cat ingress-myapp.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules: #定义后端转发的规则
- host: tomcat.lucky.com #通过域名进行转发
http:
paths:
- path: / #配置访问路径，如果通过 url 进行转发，需要修改；空默认为访问的路径为"/"
pathType: Prefix
backend: #配置后端服务
service:
name: tomcat
port:
number: 8080 [root@master ingress]# kubectl apply -f ingress-myapp.yaml
ingress.networking.k8s.io/ingress-myapp created
[root@master ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> tomcat.lucky.com 192.168.10.11,192.168.10.13 80 41s # 修改本地hosts,添加域名解析
192.168.10.199 tomcat.lucky.com
http://tomcat.lucky.com
总结：
通过 deployment+nodeSelector+pod 反亲和性实现 ingress-controller 在 node1和 node3调度
Keeplaive+nginx 实现 ingress-controller 高可用
测试 ingress 七层代理是否正常
三、Ingress Controller多租户场景使用
3.1 同一个k8s搭建多套Ingress-controller
ingress 可以简单理解为 service 的 service，他通过独立的 ingress 对象来制定请求转发的规则， 把请求路由到一个或多个 service 中。这样就把服务与请求规则解耦了，可以从业务维度统一考虑业务的 暴露，而不用为每个 service 单独考虑。
在同一个 k8s 集群里，部署两个 ingress nginx。一个 deploy 部署给 A 的 API 网关项目用。另一 个 daemonset 部署给其它项目作域名访问用。这两个项目的更新频率和用法不一致，暂时不用合成一 个。
为了满足多租户场景，需要在 k8s 集群部署多个 ingress-controller，给不同用户不同环境使用。
#主要参数设置：
containers:
- name: nginx-ingress-controller
image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
args:
- /nginx-ingress-controller
- --ingress-class=ngx-ds 注意：--ingress-class 设置该 Ingress Controller 可监听的目标 Ingress Class 标识；注意：同一个集群中不同套 Ingress Controller 监听的 Ingress Class 标识必须唯一，且不能设置为 nginx 关键字
（其是集群默认 Ingress Controller 的监听标识）；
创建Ingress规则
[root@master ingress]# cat ingress-myapp-2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx-ds" # 注意：这里要设置为你前面配置的`controller.ingressClass`唯一标识
spec:
rules: #定义后端转发的规则
- host: tomcat.lucky.com #通过域名进行转发
http:
paths:
- path: / #配置访问路径，如果通过 url 进行转发，需要修改；空默认为访问的路径为"/"
pathType: Prefix
backend: #配置后端服务
service:
name: tomcat
port:
number: 8080
四、Ingress通过Header实现流量切分--灰度发布
https://www.cnblogs.com/yangmeichong/p/16649906.html
Ingress-Controller高可用方案及多租户场景(21)的更多相关文章
- MySQL高可用方案
高可用架构对于互联网服务基本是标配,无论是应用服务还是数据库服务都需要做到高可用.虽然互联网服务号称7*24小时不间断服务,但多多少少有一些时候服务不可用,比如某些时候网页打不开,百度不能搜索或者无法 ...
- [转]MYSQL高可用方案探究(总结)
前言 http://blog.chinaunix.net/uid-20639775-id-3337432.htmlLvs+Keepalived+Mysql单点写入主主同步高可用方案 http://bl ...
- Sentinel-Redis高可用方案（一）：主从复制
引言 大概是因为Redis是个人开发的产品,所以Redis的高可用方案是被分成了几块来实现:主从复制.主从切换以及虚拟IP或客户端方案. 从Redis 2.8开始加入对Sentinel机制从而实现了服 ...
- Redis之高可用方案
Redis之高可用方案 Redis以其高效的访问速度著称.但由于官方还未发布redis-cluster,而redis的replica又有诸多不便:比如一组master-slave的机器,如果之间有 ...
- mysql高可用方案总结性说明
MySQL的各种高可用方案,大多是基于以下几种基础来部署的(也可参考:Mysql优化系列(0)--总结性梳理 该文后面有提到)1)基于主从复制:2)基于Galera协议(PXC):3)基于NDB引 ...
- 大型网站系统架构实践（五）深入探讨web应用高可用方案
从上篇文章到这篇文章,中间用了一段时间准备,主要是想把东西讲透,同时希望大家给与一些批评和建议,这样我才能有所进步,也希望喜欢我文章的朋友,给个赞,这样我才能更有激情,呵呵. 由于本篇要写的内容有点多 ...
- [转载] MySQL高可用方案选型参考
原文: http://imysql.com/2015/09/14/solutions-of-mysql-ha.shtml?hmsr=toutiao.io&utm_medium=toutiao. ...
- 分布式数据存储 - MySQL主从复制高可用方案
前面几篇文章说道MySQL数据库的高可用方案主从复制.主从复制的延迟产生原因.延迟检测及延迟解决方案(并未从根本上解决),这种主从复制方案保证数据的冗余的同时可以做读写分离来分担系统压力但是并非是高可 ...
- (转)基于Redis Sentinel的Redis集群(主从&Sharding)高可用方案
转载自:http://warm-breeze.iteye.com/blog/2020413 本文主要介绍一种通过Jedis&Sentinel实现Redis集群高可用方案,该方案需要使用Jedi ...
- mysql高可用方案MHA介绍
mysql高可用方案MHA介绍 概述 MHA是一位日本MySQL大牛用Perl写的一套MySQL故障切换方案,来保证数据库系统的高可用.在宕机的时间内(通常10-30秒内),完成故障切换,部署MHA, ...
随机推荐
- 做easyexcel遇到的问题数据库采用的mybatis-plus
导入坐标 <!-- easyexcel依赖--><dependency> <groupId>com.alibaba</groupId> <arti ...
- PS的流水灯设计分析
PS的流水灯设计分析 1.实验原理 PS设计流水灯就是采用ARM的硬核实现流水灯设计.ARM内核采用的就是一个可以执行软件程序的平台.这里采用C语言设计.基于前面构建的GPIO外设的硬件平台,这里实现 ...
- KingbaseES 如何查看表的创建时间
前言 在oracle数据库中,我们可以查看数据字典dba_objects得到表的创建时间.在Kingbase中如何查看表的创建时间呢?Kingbase数据库中无法通过数据字典查看有关信息,但可以通过其 ...
- KingbaseES V8R6集群运维案例之---sys_rewind应用分析
案例说明: sys_rewind是用于在数据库cluster的时间线分叉以后,同步一个 KingbaseES 数据库cluster 和同一数据库cluster另一份拷贝的工具.一种典型的场景是在失 ...
- List和ObservableCollection的转换
1.我们后台查询全部List数据的时候,前台需要ObservableCollection展示 这个时候List需要转换成ObservableCollection public static Obser ...
- C++设计模式 - 桥模式（Bridge）
单一职责模式: 在软件组件的设计中,如果责任划分的不清晰,使用继承得到的结果往往是随着需求的变化,子类急剧膨胀,同时充斥着重复代码,这时候的关键是划清责任. 典型模式 Decorator Bridge ...
- 鸿蒙HarmonyOS实战-ArkUI组件（Button）
一.Button Button(按钮)是一种常见的用户界面控件,通常用于触发操作或提交数据.Button 拥有文本标签和一个可点击的区域,用户点击该区域即可触发相应的操作或事件. Button 的主要 ...
- Jmeter线程组-上
线程组 线程组作为JMeter测试计划的核心组件之一,对于模拟并发用户的行为至关重要.线程组元件是整个测试计划的入口,所有的取样器和控制器必须放置在线程组下. 可以将线程组视为一个虚拟用户池,其中每个 ...
- #并查集#JZOJ 4223 旅游
题目 多次询问有多少个无序点对\((x,y)\), 满足至少有一条最大边权\(\leq d\)的路径 分析 离线询问,用并查集加边,每次产生的贡献为\(2*siz[x]*siz[y]\) 代码 #in ...
- #最大流#WOJ 124 Football Coach
题目 有\(n\)支球队,互相之间已经进行了一些比赛.现在还有\(m\)场比赛未进行, 每场比赛胜者得2分,平局各得1分,负者不得分. 问是否存在一种方法使得球队\(n\)的得分比其他\(n-1\)支 ...