k8s 新版本 部署 Ingress-nginx controller

本篇主要记录一下 k8s 新版本 1.23.5 中如何搭建 ingress controller 以及里面的注意项

新版本和老版本 区别有不少 ，坑很多，只能从官网一点点撸

1.环境准备

  • 操作系统：Centos7 （CentOS Linux release 7.9.2009）
  • Master 主节点： 1台 虚拟机
  • Node 计算节点： 2台 虚拟机
  • K8s version：v1.23.5 （选的较新的版本）
  • Docker version：20.10.14

环境和上一篇中 kubeadm 搭建 k8s 的一致

2.概述

下面从 官网摘取的对于 Ingress 的介绍

2.1 什么是 Ingress

Ingress 是对集群中服务的外部访问进行管理的 API 对象，典型的访问方式是 HTTP。Ingress 可以提供负载均衡

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。

下面是一个将所有流量都发送到同一 Service 的简单 Ingress 示例：

但是 仅创建 Ingress 资源本身没有任何效果，需要有对应的 Ingress 控制器 ，你可能需要部署 Ingress 控制器，例如 ingress-nginx。 你可以从许多 Ingress 控制器 中进行选择。

2.2 什么是 Ingress controller

为了让 Ingress 资源工作，集群必须有一个正在运行的 Ingress 控制器。k8s官网维护了 3 个Ingress控制器

目前支持和维护 AWS、 GCE 和 Nginx Ingress 控制器 ，本篇就拿 Ingress-nginx 作为控制器为例 讲解一下如何部署

还有一些 社区等提供的 控制器

3.部署 Ingress-nginx controller

本篇就拿 最常用的 Ingress-nginx 作为 Ingress的控制器，实现k8s 把请求重定向到集群内部（Cluster Ip）服务去

3.1 deploy.yaml 坑点

Ingress-nginx 官网 https://kubernetes.github.io/ingress-nginx/ 提到了 deploy.yaml 文件

Ingress-nginx 新版本的 depoly.yaml 有些不同，需要拉取下面2个镜像

k8s.gcr.io/ingress-nginx/controller:v1.1.2

k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1

多半是下载不到的，所以需要 自己替换一下 ，可以去docker hub 上找到对应的 镜像文件

总结 坑点：

  • 新版本中 提供了 IngressClass ，需要在编写 Ingress 的时候指定
  • Image 加载不到，需要手动去 docker hub 上找其他的 并且修改 deploy.yaml文件
  • 把 ingress-nginx-controller 使用 hostNetwork: true 进行部署 比 NodePort 减少一层转发，但是需要指定 选择打了标签的 node nodeSelector: app: ingress

3.2 deploy.yaml 样例（我自己修改后的 可以参考）

  1. #GENERATED FOR K8S 1.20
  2. apiVersion: v1
  3. kind: Namespace
  4. metadata:
  5.   labels:
  6.     app.kubernetes.io/instance: ingress-nginx
  7.     app.kubernetes.io/name: ingress-nginx
  8.   name: ingress-nginx
  9. ---
  10. apiVersion: v1
  11. automountServiceAccountToken: true
  12. kind: ServiceAccount
  13. metadata:
  14.   labels:
  15.     app.kubernetes.io/component: controller
  16.     app.kubernetes.io/instance: ingress-nginx
  17.     app.kubernetes.io/managed-by: Helm
  18.     app.kubernetes.io/name: ingress-nginx
  19.     app.kubernetes.io/part-of: ingress-nginx
  20.     app.kubernetes.io/version: 1.1.2
  21.     helm.sh/chart: ingress-nginx-4.0.18
  22.   name: ingress-nginx
  23.   namespace: ingress-nginx
  24. ---
  25. apiVersion: v1
  26. kind: ServiceAccount
  27. metadata:
  28.   annotations:
  29.     helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
  30.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  31.   labels:
  32.     app.kubernetes.io/component: admission-webhook
  33.     app.kubernetes.io/instance: ingress-nginx
  34.     app.kubernetes.io/managed-by: Helm
  35.     app.kubernetes.io/name: ingress-nginx
  36.     app.kubernetes.io/part-of: ingress-nginx
  37.     app.kubernetes.io/version: 1.1.2
  38.     helm.sh/chart: ingress-nginx-4.0.18
  39.   name: ingress-nginx-admission
  40.   namespace: ingress-nginx
  41. ---
  42. apiVersion: rbac.authorization.k8s.io/v1
  43. kind: Role
  44. metadata:
  45.   labels:
  46.     app.kubernetes.io/component: controller
  47.     app.kubernetes.io/instance: ingress-nginx
  48.     app.kubernetes.io/managed-by: Helm
  49.     app.kubernetes.io/name: ingress-nginx
  50.     app.kubernetes.io/part-of: ingress-nginx
  51.     app.kubernetes.io/version: 1.1.2
  52.     helm.sh/chart: ingress-nginx-4.0.18
  53.   name: ingress-nginx
  54.   namespace: ingress-nginx
  55. rules:
  56.   - apiGroups:
  57.       - ""
  58.     resources:
  59.       - namespaces
  60.     verbs:
  61.       - get
  62.   - apiGroups:
  63.       - ""
  64.     resources:
  65.       - configmaps
  66.       - pods
  67.       - secrets
  68.       - endpoints
  69.     verbs:
  70.       - get
  71.       - list
  72.       - watch
  73.   - apiGroups:
  74.       - ""
  75.     resources:
  76.       - services
  77.     verbs:
  78.       - get
  79.       - list
  80.       - watch
  81.   - apiGroups:
  82.       - networking.k8s.io
  83.     resources:
  84.       - ingresses
  85.     verbs:
  86.       - get
  87.       - list
  88.       - watch
  89.   - apiGroups:
  90.       - networking.k8s.io
  91.     resources:
  92.       - ingresses/status
  93.     verbs:
  94.       - update
  95.   - apiGroups:
  96.       - networking.k8s.io
  97.     resources:
  98.       - ingressclasses
  99.     verbs:
  100.       - get
  101.       - list
  102.       - watch
  103.   - apiGroups:
  104.       - ""
  105.     resourceNames:
  106.       - ingress-controller-leader
  107.     resources:
  108.       - configmaps
  109.     verbs:
  110.       - get
  111.       - update
  112.   - apiGroups:
  113.       - ""
  114.     resources:
  115.       - configmaps
  116.     verbs:
  117.       - create
  118.   - apiGroups:
  119.       - ""
  120.     resources:
  121.       - events
  122.     verbs:
  123.       - create
  124.       - patch
  125. ---
  126. apiVersion: rbac.authorization.k8s.io/v1
  127. kind: Role
  128. metadata:
  129.   annotations:
  130.     helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
  131.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  132.   labels:
  133.     app.kubernetes.io/component: admission-webhook
  134.     app.kubernetes.io/instance: ingress-nginx
  135.     app.kubernetes.io/managed-by: Helm
  136.     app.kubernetes.io/name: ingress-nginx
  137.     app.kubernetes.io/part-of: ingress-nginx
  138.     app.kubernetes.io/version: 1.1.2
  139.     helm.sh/chart: ingress-nginx-4.0.18
  140.   name: ingress-nginx-admission
  141.   namespace: ingress-nginx
  142. rules:
  143.   - apiGroups:
  144.       - ""
  145.     resources:
  146.       - secrets
  147.     verbs:
  148.       - get
  149.       - create
  150. ---
  151. apiVersion: rbac.authorization.k8s.io/v1
  152. kind: ClusterRole
  153. metadata:
  154.   labels:
  155.     app.kubernetes.io/instance: ingress-nginx
  156.     app.kubernetes.io/managed-by: Helm
  157.     app.kubernetes.io/name: ingress-nginx
  158.     app.kubernetes.io/part-of: ingress-nginx
  159.     app.kubernetes.io/version: 1.1.2
  160.     helm.sh/chart: ingress-nginx-4.0.18
  161.   name: ingress-nginx
  162. rules:
  163.   - apiGroups:
  164.       - ""
  165.     resources:
  166.       - configmaps
  167.       - endpoints
  168.       - nodes
  169.       - pods
  170.       - secrets
  171.       - namespaces
  172.     verbs:
  173.       - list
  174.       - watch
  175.   - apiGroups:
  176.       - ""
  177.     resources:
  178.       - nodes
  179.     verbs:
  180.       - get
  181.   - apiGroups:
  182.       - ""
  183.     resources:
  184.       - services
  185.     verbs:
  186.       - get
  187.       - list
  188.       - watch
  189.   - apiGroups:
  190.       - networking.k8s.io
  191.     resources:
  192.       - ingresses
  193.     verbs:
  194.       - get
  195.       - list
  196.       - watch
  197.   - apiGroups:
  198.       - ""
  199.     resources:
  200.       - events
  201.     verbs:
  202.       - create
  203.       - patch
  204.   - apiGroups:
  205.       - networking.k8s.io
  206.     resources:
  207.       - ingresses/status
  208.     verbs:
  209.       - update
  210.   - apiGroups:
  211.       - networking.k8s.io
  212.     resources:
  213.       - ingressclasses
  214.     verbs:
  215.       - get
  216.       - list
  217.       - watch
  218. ---
  219. apiVersion: rbac.authorization.k8s.io/v1
  220. kind: ClusterRole
  221. metadata:
  222.   annotations:
  223.     helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
  224.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  225.   labels:
  226.     app.kubernetes.io/component: admission-webhook
  227.     app.kubernetes.io/instance: ingress-nginx
  228.     app.kubernetes.io/managed-by: Helm
  229.     app.kubernetes.io/name: ingress-nginx
  230.     app.kubernetes.io/part-of: ingress-nginx
  231.     app.kubernetes.io/version: 1.1.2
  232.     helm.sh/chart: ingress-nginx-4.0.18
  233.   name: ingress-nginx-admission
  234. rules:
  235.   - apiGroups:
  236.       - admissionregistration.k8s.io
  237.     resources:
  238.       - validatingwebhookconfigurations
  239.     verbs:
  240.       - get
  241.       - update
  242. ---
  243. apiVersion: rbac.authorization.k8s.io/v1
  244. kind: RoleBinding
  245. metadata:
  246.   labels:
  247.     app.kubernetes.io/component: controller
  248.     app.kubernetes.io/instance: ingress-nginx
  249.     app.kubernetes.io/managed-by: Helm
  250.     app.kubernetes.io/name: ingress-nginx
  251.     app.kubernetes.io/part-of: ingress-nginx
  252.     app.kubernetes.io/version: 1.1.2
  253.     helm.sh/chart: ingress-nginx-4.0.18
  254.   name: ingress-nginx
  255.   namespace: ingress-nginx
  256. roleRef:
  257.   apiGroup: rbac.authorization.k8s.io
  258.   kind: Role
  259.   name: ingress-nginx
  260. subjects:
  261.   - kind: ServiceAccount
  262.     name: ingress-nginx
  263.     namespace: ingress-nginx
  264. ---
  265. apiVersion: rbac.authorization.k8s.io/v1
  266. kind: RoleBinding
  267. metadata:
  268.   annotations:
  269.     helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
  270.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  271.   labels:
  272.     app.kubernetes.io/component: admission-webhook
  273.     app.kubernetes.io/instance: ingress-nginx
  274.     app.kubernetes.io/managed-by: Helm
  275.     app.kubernetes.io/name: ingress-nginx
  276.     app.kubernetes.io/part-of: ingress-nginx
  277.     app.kubernetes.io/version: 1.1.2
  278.     helm.sh/chart: ingress-nginx-4.0.18
  279.   name: ingress-nginx-admission
  280.   namespace: ingress-nginx
  281. roleRef:
  282.   apiGroup: rbac.authorization.k8s.io
  283.   kind: Role
  284.   name: ingress-nginx-admission
  285. subjects:
  286.   - kind: ServiceAccount
  287.     name: ingress-nginx-admission
  288.     namespace: ingress-nginx
  289. ---
  290. apiVersion: rbac.authorization.k8s.io/v1
  291. kind: ClusterRoleBinding
  292. metadata:
  293.   labels:
  294.     app.kubernetes.io/instance: ingress-nginx
  295.     app.kubernetes.io/managed-by: Helm
  296.     app.kubernetes.io/name: ingress-nginx
  297.     app.kubernetes.io/part-of: ingress-nginx
  298.     app.kubernetes.io/version: 1.1.2
  299.     helm.sh/chart: ingress-nginx-4.0.18
  300.   name: ingress-nginx
  301. roleRef:
  302.   apiGroup: rbac.authorization.k8s.io
  303.   kind: ClusterRole
  304.   name: ingress-nginx
  305. subjects:
  306.   - kind: ServiceAccount
  307.     name: ingress-nginx
  308.     namespace: ingress-nginx
  309. ---
  310. apiVersion: rbac.authorization.k8s.io/v1
  311. kind: ClusterRoleBinding
  312. metadata:
  313.   annotations:
  314.     helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
  315.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  316.   labels:
  317.     app.kubernetes.io/component: admission-webhook
  318.     app.kubernetes.io/instance: ingress-nginx
  319.     app.kubernetes.io/managed-by: Helm
  320.     app.kubernetes.io/name: ingress-nginx
  321.     app.kubernetes.io/part-of: ingress-nginx
  322.     app.kubernetes.io/version: 1.1.2
  323.     helm.sh/chart: ingress-nginx-4.0.18
  324.   name: ingress-nginx-admission
  325. roleRef:
  326.   apiGroup: rbac.authorization.k8s.io
  327.   kind: ClusterRole
  328.   name: ingress-nginx-admission
  329. subjects:
  330.   - kind: ServiceAccount
  331.     name: ingress-nginx-admission
  332.     namespace: ingress-nginx
  333. ---
  334. apiVersion: v1
  335. data:
  336.   allow-snippet-annotations: "true"
  337. kind: ConfigMap
  338. metadata:
  339.   labels:
  340.     app.kubernetes.io/component: controller
  341.     app.kubernetes.io/instance: ingress-nginx
  342.     app.kubernetes.io/managed-by: Helm
  343.     app.kubernetes.io/name: ingress-nginx
  344.     app.kubernetes.io/part-of: ingress-nginx
  345.     app.kubernetes.io/version: 1.1.2
  346.     helm.sh/chart: ingress-nginx-4.0.18
  347.   name: ingress-nginx-controller
  348.   namespace: ingress-nginx
  349. ---
  350. apiVersion: v1
  351. kind: Service
  352. metadata:
  353.   labels:
  354.     app.kubernetes.io/component: controller
  355.     app.kubernetes.io/instance: ingress-nginx
  356.     app.kubernetes.io/managed-by: Helm
  357.     app.kubernetes.io/name: ingress-nginx
  358.     app.kubernetes.io/part-of: ingress-nginx
  359.     app.kubernetes.io/version: 1.1.2
  360.     helm.sh/chart: ingress-nginx-4.0.18
  361.   name: ingress-nginx-controller
  362.   namespace: ingress-nginx
  363. spec:
  364.   externalTrafficPolicy: Local
  365.   ipFamilies:
  366.     - IPv4
  367.   ipFamilyPolicy: SingleStack
  368.   ports:
  369.     - appProtocol: http
  370.       name: http
  371.       port: 80
  372.       protocol: TCP
  373.       targetPort: http
  374.     - appProtocol: https
  375.       name: https
  376.       port: 443
  377.       protocol: TCP
  378.       targetPort: https
  379.   selector:
  380.     app.kubernetes.io/component: controller
  381.     app.kubernetes.io/instance: ingress-nginx
  382.     app.kubernetes.io/name: ingress-nginx
  383.   type: LoadBalancer
  384. ---
  385. apiVersion: v1
  386. kind: Service
  387. metadata:
  388.   labels:
  389.     app.kubernetes.io/component: controller
  390.     app.kubernetes.io/instance: ingress-nginx
  391.     app.kubernetes.io/managed-by: Helm
  392.     app.kubernetes.io/name: ingress-nginx
  393.     app.kubernetes.io/part-of: ingress-nginx
  394.     app.kubernetes.io/version: 1.1.2
  395.     helm.sh/chart: ingress-nginx-4.0.18
  396.   name: ingress-nginx-controller-admission
  397.   namespace: ingress-nginx
  398. spec:
  399.   ports:
  400.     - appProtocol: https
  401.       name: https-webhook
  402.       port: 443
  403.       targetPort: webhook
  404.   selector:
  405.     app.kubernetes.io/component: controller
  406.     app.kubernetes.io/instance: ingress-nginx
  407.     app.kubernetes.io/name: ingress-nginx
  408.   type: ClusterIP
  409. ---
  410. apiVersion: apps/v1
  411. kind: Deployment
  412. metadata:
  413.   labels:
  414.     app.kubernetes.io/component: controller
  415.     app.kubernetes.io/instance: ingress-nginx
  416.     app.kubernetes.io/managed-by: Helm
  417.     app.kubernetes.io/name: ingress-nginx
  418.     app.kubernetes.io/part-of: ingress-nginx
  419.     app.kubernetes.io/version: 1.1.2
  420.     helm.sh/chart: ingress-nginx-4.0.18
  421.   name: ingress-nginx-controller
  422.   namespace: ingress-nginx
  423. spec:
  424.   minReadySeconds: 0
  425.   revisionHistoryLimit: 10
  426.   selector:
  427.     matchLabels:
  428.       app.kubernetes.io/component: controller
  429.       app.kubernetes.io/instance: ingress-nginx
  430.       app.kubernetes.io/name: ingress-nginx
  431.   template:
  432.     metadata:
  433.       labels:
  434.         app.kubernetes.io/component: controller
  435.         app.kubernetes.io/instance: ingress-nginx
  436.         app.kubernetes.io/name: ingress-nginx
  437.     spec:
  438.       hostNetwork: true #修改  ingress-nginx-controller 为 hostNetwork模式
  439.       nodeSelector: #选择 node label 中有 app=ingress的节点进行部署
  440.         app: ingress
  441.       containers:
  442.         - args:
  443.             - /nginx-ingress-controller
  444.             - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
  445.             - --election-id=ingress-controller-leader
  446.             - --controller-class=k8s.io/ingress-nginx
  447.             - --ingress-class=nginx
  448.             - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
  449.             - --validating-webhook=:8443
  450.             - --validating-webhook-certificate=/usr/local/certificates/cert
  451.             - --validating-webhook-key=/usr/local/certificates/key
  452.           env:
  453.             - name: POD_NAME
  454.               valueFrom:
  455.                 fieldRef:
  456.                   fieldPath: metadata.name
  457.             - name: POD_NAMESPACE
  458.               valueFrom:
  459.                 fieldRef:
  460.                   fieldPath: metadata.namespace
  461.             - name: LD_PRELOAD
  462.               value: /usr/local/lib/libmimalloc.so
  463.           image: k8s.gcr.io/ingress-nginx/controller:v1.1.2 #修改镜像地址
  464.           imagePullPolicy: IfNotPresent
  465.           lifecycle:
  466.             preStop:
  467.               exec:
  468.                 command:
  469.                   - /wait-shutdown
  470.           livenessProbe:
  471.             failureThreshold: 5
  472.             httpGet:
  473.               path: /healthz
  474.               port: 10254
  475.               scheme: HTTP
  476.             initialDelaySeconds: 10
  477.             periodSeconds: 10
  478.             successThreshold: 1
  479.             timeoutSeconds: 1
  480.           name: controller
  481.           ports:
  482.             - containerPort: 80
  483.               name: http
  484.               protocol: TCP
  485.             - containerPort: 443
  486.               name: https
  487.               protocol: TCP
  488.             - containerPort: 8443
  489.               name: webhook
  490.               protocol: TCP
  491.           readinessProbe:
  492.             failureThreshold: 3
  493.             httpGet:
  494.               path: /healthz
  495.               port: 10254
  496.               scheme: HTTP
  497.             initialDelaySeconds: 10
  498.             periodSeconds: 10
  499.             successThreshold: 1
  500.             timeoutSeconds: 1
  501.           resources:
  502.             requests:
  503.               cpu: 100m
  504.               memory: 90Mi
  505.           securityContext:
  506.             allowPrivilegeEscalation: true
  507.             capabilities:
  508.               add:
  509.                 - NET_BIND_SERVICE
  510.               drop:
  511.                 - ALL
  512.             runAsUser: 101
  513.           volumeMounts:
  514.             - mountPath: /usr/local/certificates/
  515.               name: webhook-cert
  516.               readOnly: true
  517.       dnsPolicy: ClusterFirst
  518.       nodeSelector:
  519.         kubernetes.io/os: linux
  520.       serviceAccountName: ingress-nginx
  521.       terminationGracePeriodSeconds: 300
  522.       volumes:
  523.         - name: webhook-cert
  524.           secret:
  525.             secretName: ingress-nginx-admission
  526. ---
  527. apiVersion: batch/v1
  528. kind: Job
  529. metadata:
  530.   annotations:
  531.     helm.sh/hook: pre-install,pre-upgrade
  532.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  533.   labels:
  534.     app.kubernetes.io/component: admission-webhook
  535.     app.kubernetes.io/instance: ingress-nginx
  536.     app.kubernetes.io/managed-by: Helm
  537.     app.kubernetes.io/name: ingress-nginx
  538.     app.kubernetes.io/part-of: ingress-nginx
  539.     app.kubernetes.io/version: 1.1.2
  540.     helm.sh/chart: ingress-nginx-4.0.18
  541.   name: ingress-nginx-admission-create
  542.   namespace: ingress-nginx
  543. spec:
  544.   template:
  545.     metadata:
  546.       labels:
  547.         app.kubernetes.io/component: admission-webhook
  548.         app.kubernetes.io/instance: ingress-nginx
  549.         app.kubernetes.io/managed-by: Helm
  550.         app.kubernetes.io/name: ingress-nginx
  551.         app.kubernetes.io/part-of: ingress-nginx
  552.         app.kubernetes.io/version: 1.1.2
  553.         helm.sh/chart: ingress-nginx-4.0.18
  554.       name: ingress-nginx-admission-create
  555.     spec:
  556.       containers:
  557.         - args:
  558.             - create
  559.             - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
  560.             - --namespace=$(POD_NAMESPACE)
  561.             - --secret-name=ingress-nginx-admission
  562.           env:
  563.             - name: POD_NAMESPACE
  564.               valueFrom:
  565.                 fieldRef:
  566.                   fieldPath: metadata.namespace
  567.           image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 #修改镜像地址
  568.           imagePullPolicy: IfNotPresent
  569.           name: create
  570.           securityContext:
  571.             allowPrivilegeEscalation: false
  572.       nodeSelector:
  573.         kubernetes.io/os: linux
  574.       restartPolicy: OnFailure
  575.       securityContext:
  576.         fsGroup: 2000
  577.         runAsNonRoot: true
  578.         runAsUser: 2000
  579.       serviceAccountName: ingress-nginx-admission
  580. ---
  581. apiVersion: batch/v1
  582. kind: Job
  583. metadata:
  584.   annotations:
  585.     helm.sh/hook: post-install,post-upgrade
  586.     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  587.   labels:
  588.     app.kubernetes.io/component: admission-webhook
  589.     app.kubernetes.io/instance: ingress-nginx
  590.     app.kubernetes.io/managed-by: Helm
  591.     app.kubernetes.io/name: ingress-nginx
  592.     app.kubernetes.io/part-of: ingress-nginx
  593.     app.kubernetes.io/version: 1.1.2
  594.     helm.sh/chart: ingress-nginx-4.0.18
  595.   name: ingress-nginx-admission-patch
  596.   namespace: ingress-nginx
  597. spec:
  598.   template:
  599.     metadata:
  600.       labels:
  601.         app.kubernetes.io/component: admission-webhook
  602.         app.kubernetes.io/instance: ingress-nginx
  603.         app.kubernetes.io/managed-by: Helm
  604.         app.kubernetes.io/name: ingress-nginx
  605.         app.kubernetes.io/part-of: ingress-nginx
  606.         app.kubernetes.io/version: 1.1.2
  607.         helm.sh/chart: ingress-nginx-4.0.18
  608.       name: ingress-nginx-admission-patch
  609.     spec:
  610.       containers:
  611.         - args:
  612.             - patch
  613.             - --webhook-name=ingress-nginx-admission
  614.             - --namespace=$(POD_NAMESPACE)
  615.             - --patch-mutating=false
  616.             - --secret-name=ingress-nginx-admission
  617.             - --patch-failure-policy=Fail
  618.           env:
  619.             - name: POD_NAMESPACE
  620.               valueFrom:
  621.                 fieldRef:
  622.                   fieldPath: metadata.namespace
  623.           image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 #修改镜像地址
  624.           imagePullPolicy: IfNotPresent
  625.           name: patch
  626.           securityContext:
  627.             allowPrivilegeEscalation: false
  628.       nodeSelector:
  629.         kubernetes.io/os: linux
  630.       restartPolicy: OnFailure
  631.       securityContext:
  632.         fsGroup: 2000
  633.         runAsNonRoot: true
  634.         runAsUser: 2000
  635.       serviceAccountName: ingress-nginx-admission
  636. ---
  637. apiVersion: networking.k8s.io/v1
  638. kind: IngressClass
  639. metadata:
  640.   labels:
  641.     app.kubernetes.io/component: controller
  642.     app.kubernetes.io/instance: ingress-nginx
  643.     app.kubernetes.io/managed-by: Helm
  644.     app.kubernetes.io/name: ingress-nginx
  645.     app.kubernetes.io/part-of: ingress-nginx
  646.     app.kubernetes.io/version: 1.1.2
  647.     helm.sh/chart: ingress-nginx-4.0.18
  648.   name: nginx
  649. spec:
  650.   controller: k8s.io/ingress-nginx
  651. ---
  652. apiVersion: admissionregistration.k8s.io/v1
  653. kind: ValidatingWebhookConfiguration
  654. metadata:
  655.   labels:
  656.     app.kubernetes.io/component: admission-webhook
  657.     app.kubernetes.io/instance: ingress-nginx
  658.     app.kubernetes.io/managed-by: Helm
  659.     app.kubernetes.io/name: ingress-nginx
  660.     app.kubernetes.io/part-of: ingress-nginx
  661.     app.kubernetes.io/version: 1.1.2
  662.     helm.sh/chart: ingress-nginx-4.0.18
  663.   name: ingress-nginx-admission
  664. webhooks:
  665.   - admissionReviewVersions:
  666.       - v1
  667.     clientConfig:
  668.       service:
  669.         name: ingress-nginx-controller-admission
  670.         namespace: ingress-nginx
  671.         path: /networking/v1/ingresses
  672.     failurePolicy: Fail
  673.     matchPolicy: Equivalent
  674.     name: validate.nginx.ingress.kubernetes.io
  675.     rules:
  676.       - apiGroups:
  677.           - networking.k8s.io
  678.         apiVersions:
  679.           - v1
  680.         operations:
  681.           - CREATE
  682.           - UPDATE
  683.         resources:
  684.           - ingresses
  685.     sideEffects: None

4.部署 Ingress-nginx

4.1 准备工作

给 node2 节点打了app=ingress标签，因为上面的ingress-nginx-controller 使用的是 hostNetwork 模式（只会放pod真实pod 的 端口） + nodeSelector

  1. kubectl label node kubec-node-2 app=ingress

4.2 部署 kubect apply -f

  1. kubectl apply -f deploy.yaml #通过 kubectl apply 命令进行部署 ，前提是镜像准备好，否则GG

4.3 查看状态

  1. kubectl get all -n ingress-nginx #查看 ingress-nginx namespace的 部署情况

  1. kubectl logs -f ingress-nginx-controller-744f6d5bdf-tl6g4 -n ingress-nginx # 查看 ingress-nginx-controller 的 日志情况

4.4 测试访问

直接访问 kube-node-2的 ip 即可，因为 ingress-nginx-controller 默认是 监听 80端口，由于上面的 nodeSelector: #选择 node label 中有 app=ingress的节点进行部署 ,而 kube-node-2 是被打了标签的节点node

可以看到 其实就是一个 nginx

5.部署一个 tomcat 测试 Ingress-nginx

通过部署一个tomcat ，测试Ingress-nginx的代理 是否生效

5.1 编写 deploy-tomcat.yaml

主要编写

  • Deployment 部署tomcat:8.0-alpine，

  • Service 暴露 tomcat pod

  • 一个 Ingress 资源它规定 访问 tomcat.demo.com 这个域名的 所有请求 / 都转发到 tomcat-demo Service 上

    IngressClass 新版本提供的资源 ，用于在定义 Ingress资源的时候 指定，在集群中有多个 Ingress controller 的时候很有用处

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: tomcat-demo
  5. spec:
  6.   selector:
  7.     matchLabels:
  8.       app: tomcat-demo
  9.   replicas: 1
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: tomcat-demo
  14.     spec:
  15.       containers:
  16.       - name: tomcat-demo
  17.         image: tomcat:8.0-alpine
  18.         ports:
  19.         - containerPort: 8080
  20. ---
  22. apiVersion: v1
  23. kind: Service
  24. metadata:
  25.   name: tomcat-demo
  26. spec:
  27.   selector:
  28.     app: tomcat-demo #选择 tomcat-demo pod
  29.   ports:
  30.   - port: 80 #对外暴露 80 端口
  31.     protocol: TCP
  32.     targetPort: 8080 # tomcat 端口
  34. ---
  36. apiVersion: networking.k8s.io/v1
  37. kind: Ingress
  38. metadata:
  39.   name: tomcat-demo
  40. spec:
  41.   defaultBackend:
  42.     service:
  43.       name: default-http-backend #！！！ 指定 默认的backend服务
  44.       port:
  45.       	number: 80
  46.   ingressClassName: nginx #！！！重点 需要指定 哪个 IngressClass 可以看上面的 deploy.yaml 最后定义的
  47.   rules:
  48.   - host: tomcat.demo.com #所有的  tomcat.demo.com请求都转发到  Service tomcat-demo
  49.     http:
  50.       paths:
  51.       - pathType: Prefix
  52.         path: "/"
  53.         backend:
  54.           service:
  55.             name: tomcat-demo
  56.             port:
  57.               number: 80
  59. ---
  60. #定义一个 default-http-backend 当没有被Ingress规定的请求 负载给 它
  61. apiVersion: apps/v1
  62. kind: Deployment
  63. metadata:
  64.   name: default-http-backend
  65.   labels:
  66.     app: default-http-backend
  67. spec:
  68.   replicas: 1
  69.   selector:
  70.     matchLabels:
  71.       app: default-http-backend
  72.   template:
  73.     metadata:
  74.       labels:
  75.         app: default-http-backend
  76.     spec:
  77.       terminationGracePeriodSeconds: 60
  78.       containers:
  79.       - name: default-http-backend
  80.         # Any image is permissible as long as:
  81.         # 1. It serves a 404 page at /
  82.         # 2. It serves 200 on a /healthz endpoint
  83.         image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4
  84.         livenessProbe:
  85.           httpGet:
  86.             path: /healthz
  87.             port: 8080
  88.             scheme: HTTP
  89.           initialDelaySeconds: 30
  90.           timeoutSeconds: 5
  91.         ports:
  92.         - containerPort: 8080
  93.         resources:
  94.           limits:
  95.             cpu: 10m
  96.             memory: 20Mi
  97.           requests:
  98.             cpu: 10m
  99.             memory: 20Mi
  100. ---
  102. apiVersion: v1
  103. kind: Service
  104. metadata:
  105.   name: default-http-backend
  106.   labels:
  107.     app: default-http-backend
  108. spec:
  109.   ports:
  110.   - port: 80
  111.     targetPort: 8080
  112.   selector:
  113.     app: default-http-backend

5.2 部署 tomcat + ingress + default-http-backend

部署上面的文件 即可

查看 部署情况 可以看到都是Running 了

5.3 测试 通过Ingress-nginx 能否访问到tomcat

由于我们 ingress 资源配置的 域名 是 tomcat.demo.com ，所以我需要把它添加到 宿主机的hosts 文件中，如下

那么按照预期，当我访问 tomcat.demo.com 的时候其实就是访问 192.168.56.22 上的 ingress-nginx-controller 这个nginx，那么根据 ingress 的 资源的设定 它会把请求 转发到 Service tomcat-demo 上，从而访问到 tomcat 界面

当我访问 api.demo.com 的时候 由于没有对这个域名进行处理 那么会默认把请求转发到 default-http-backend 上

访问 tomcat.demo.com

访问 api.demo.com

总结

至此在k8s 1.23.5上 已经成功部署了 Ingress-nginx ，并且通过部署一个tomcat服务测试了 Ingress-nginx 已经代理成功了，由于不管是 k8s 新版本和 Ingress-nginx 新版本都有些变化 所有很些坑，是我从官网不断摸索的，下面总结一下坑点

  • image 镜像需要自己从docker hub 上准备好 ，或者直接修改Ingress-nginx 的 deploy.yaml

    k8s.gcr.io/ingress-nginx/controller:v1.1.2

    k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1

  • 重要！新版本中 提供了 IngressClass ，需要在编写 Ingress 的时候指定

  • 把 ingress-nginx-controller 使用 hostNetwork: true 进行部署 比 NodePort 减少一层转发，但是需要指定 选择打了标签的 node nodeSelector: app: ingress

  • 通过 Ingress spec.defaultBackend 指定 默认服务，并且需要保持在同一个 namespace

    1. spec:
    2. 	defaultBackend:
    3.     service:
    4.       name: default-http-backend #！！！ 指定 默认的backend服务
    5.       port:
    6.       	number: 80

  • vscode k8s 插件 快捷生成Depoyment 的时候 自定添加了资源限制，导致部署tomcat的时候 总是启动不起来..

    导致tomcat pod 状态 CrashLoopBackOff，从而导致 通过 Ingress-nginx 无法访问到tomcat

    1. resources:
    2.   limits:
    3.     memory: "128Mi" #不够部署tomcat 内存
    4.     cpu: "500m"

欢迎大家访问 个人博客 Johnny小屋

k8s 新版本 部署 Ingress-nginx controller的更多相关文章

  1. 通过重新上传修改后的docker镜像来在kubeapps上实现k8s上部署的nginx版本更新，回退等

    docker操作:制作自定义镜像 # docker下载官方nginx镜像 docker pull nginx # 基于该镜像运行一个容器 docker run -it -d --name nginx_ ...

  2. 云原生之旅 - 8）云原生时代的网关 Ingress Nginx

    前言 当我们在Kubernetes部署的服务需要暴露给外部用户使用时,有三种选择:LoadBalancer,NodePort, Ingress. LoadBalancer类型得结合各个Cloud Pr ...

  3. 基于k8s集群部署prometheus监控ingress nginx

    目录 基于k8s集群部署prometheus监控ingress nginx 1.背景和环境概述 2.修改prometheus配置 3.检查是否生效 4.配置grafana图形 基于k8s集群部署pro ...

  4. 【原创】一层Nginx反向代理K8S化部署实践

    目录: 1)背景介绍 2)方案分析 3)实现细节 4)监控告警 5)日志收集 6)测试 一.背景介绍     如下图所示,传统方式部署一层Nginx,随着业务扩大,维护管理变得复杂,繁琐,耗时耗力和易 ...

  5. K8S中部署apisix(非ingress)

    不使用pvc的方式在K8S中部署apisix-gateway 简介 因为公司项目准备重构,现在做技术储备,之前公司项目使用的ocelot做网关,ocelot是.net平台下的一个网关,也是很不错,但是 ...

  6. [转帖]在 k8s 中通过 Ingress 配置域名访问

    在 k8s 中通过 Ingress 配置域名访问 https://juejin.im/post/5db8da4b6fb9a0204520b310 在上篇文章中我们已经使用 k8s 部署了第一个应用,此 ...

  7. k8s 中的 ingress 使用细节

    k8s中的ingress 什么是ingress Ingress 如何使用 ingress 使用细节 参考 k8s中的ingress 什么是ingress k8s 中使用 Service 为相同业务的 ...

  8. 【转帖】kubernetes 部署ingress

    kubernetes 部署ingress https://www.cnblogs.com/dingbin/p/9754993.html 明天尝试一下 之前的文档里面一直没有提 需要改host文件 我有 ...

  9. kubernetes 部署ingress

    kubernetes Ingess 是有2部分组成,Ingress Controller 和Ingress服务组成,常用的Ingress Controller 是ingress-nginx,工作的原理 ...

随机推荐

  1. sql高级手工注入

    非常重要:首先在网站找到管理入口,否则,呵呵就算有用户名和密码,找不到入口,也是白玩.. 注入时,注意通过改变大小写.编码.转换等方式躲过系统检查,顺利执行语句!!! (一)数字型注入 正常步骤: 1 ...

  2. nacos底层原理

    Nacos   为什么选择NacosNacos 致力于帮助您发现.配置和管理微服务.Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现.服务配置.服务元数据及流量管理. Nacos 帮 ...

  3. 谈一谈 Kafka 的再均衡？

    在Kafka中,当有新消费者加入或者订阅的topic数发生变化时,会触发Rebalance(再均衡:在同一个消费者组当中,分区的所有权从一个消费者转移到另外一个消费者)机制,Rebalance顾名思义 ...

  4. cornerstone 忽略不必要文件

    转:https://www.jianshu.com/p/f48207baa0cd

  5. Lambda8 表达式

    Lambda 表达式 Lambda 表达式是 JDK8 的一个新特性,可以取代大部分的匿名内部类,写出更优雅的 Java 代码,尤其在集合的遍历和其他集合操作中,可以极大地优化代码结构. JDK 也提 ...

  6. Python中module文件夹里__init__.py的功能

    怎么引用模块 环境:win7 + python3.5.2文档结构: -project -data -src  -filterCorpus.py  -translateMonolingual.py 问题 ...

  7. APICloud案例源码、模块源码、考试源码、开发工具大集合！赶快收藏

    APICloud专注于APP开发定制技术,多年来不停为开发者奉献更多的资源.此次,APICloud将以往的的资源进行更新.整合,以合集的形式分享给广大的用户. APICloud应用案例源码合集 API ...

  8. 动态添加HTML时onclick函数参数传递

    onclick函数动态传参 1.参数为数值类型时: var tmp = 123; var strHTML = "<div onclick=func(" + tmp + &qu ...

  9. 【uniapp 开发】uni-app 资源在线升级/热更新

    注:本文为前端代码资源热更新.如果是整包升级,另见文档 https://ask.dcloud.net.cn/article/34972 HBuilderX 1.6.5 起,uni-app 支持生成 A ...

  10. vue和react给我的感受

    以下纯属个人使用两个框架的感想和体会: 不知道你们是否有这种感觉~ 我vue和react都用过一段时间,但是vue给我感觉就是经常会忘记语法,需要对照文档才知道怎么写( 难不成是我没喝六个核桃的原因吗 ...