K8S容器探针

容器探针

探针是由 kubelet对容器执行的定期诊断。要执行诊断， kubelet 调用由容器实现的    Handler 。有三种类型的处理程序：
   ExecAction ：在容器内执行指定命令。如果命令退出时返回码为 0 则认为诊断成功。
   TCPSocketAction ：对指定端口上的容器的 IP 地址进行 TCP 检查。如果端口打开，则诊断被认为是成功的。
   HTTPGetAction ：对指定的端口和路径上的容器的 IP 地址执行 HTTP Get 请求。如果响应的状态码大于等于 200 且小于 400 ，则诊断被认为是成功的
每次探测都将获得以下三种结果之一：
   成功：容器通过了诊断。
   失败：容器未通过诊断。
   未知：诊断失败，因此不会采取任何行动

探针的方式

livenessProbe ：指示容器是否正在运行。如果存活探测失败，则 kubelet 会杀死容器，并且容器将受到其 重启策略 的影响。如果容器不提供存活探针，则默认状态为 Success
readinessProbe ：指示容器是否准备好服务请求。如果就绪探测失败，端点控制器将从与 Pod 匹配的所有 Service 的端点中删除该 Pod 的 IP 地址。初始延迟之前的就绪状态默认为 Failure 。如果容器不提供就绪探针，则默认状态为 Success。

测试

检测探针 - 就绪检测

read.yaml

[root@k8s-master mnt]# cat read.yaml
apiVersion: v1
kind: Pod
metadata:
  name: readiness-httpget-pod
  namespace: default
spec:
  containers:
  - name: readiness-httpget-container
    image: wangyanglinux/myapp:v1
    imagePullPolicy: IfNotPresent
    readinessProbe:
      httpGet:
        port: 80
        path: /index1.html
      initialDelaySeconds: 1
      periodSeconds: 3
[root@k8s-master mnt]#

[root@k8s-master mnt]# vim read.yaml
[root@k8s-master mnt]# kubectl create -f read.yaml
pod/readiness-httpget-pod created
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
myapp-pod               1/1     Running   0          70m
readiness-httpget-pod   0/1     Running   0          17s
[root@k8s-master mnt]# kubectl describe pod readiness-httpget-pod
Name:         readiness-httpget-pod
Namespace:    default
Priority:     0
Node:         k8s-node01/192.168.180.133
Start Time:   Wed, 18 Dec 2019 23:12:59 +0800
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.2.10
IPs:
  IP:  10.244.2.10
Containers:
  readiness-httpget-container:
    Container ID:   docker://566ff6cdcf44daaba316b796fb8bf6f9563ddd44000c9ae9f572fd0a6719684c
    Image:          wangyanglinux/myapp:v1
    Image ID:       docker-pullable://wangyanglinux/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 18 Dec 2019 23:13:01 +0800
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:80/index1.html delay=1s timeout=1s period=3s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gx2h8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-gx2h8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gx2h8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                 Message
  ----     ------     ----               ----                 -------
  Normal   Scheduled  <unknown>          default-scheduler    Successfully assigned default/readiness-httpget-pod to k8s-node01
  Normal   Pulled     66s                kubelet, k8s-node01  Container image "wangyanglinux/myapp:v1" already present on machine
  Normal   Created    66s                kubelet, k8s-node01  Created container readiness-httpget-container
  Normal   Started    66s                kubelet, k8s-node01  Started container readiness-httpget-container
  Warning  Unhealthy  0s (x22 over 63s)  kubelet, k8s-node01  Readiness probe failed: HTTP probe failed with statuscode: 404
[root@k8s-master mnt]# kubectl exec readiness-httpget-pod -it /bin/sh
/ # ls
bin    dev    etc    home   lib    media  mnt    proc   root   run    sbin   srv    sys    tmp    usr    var
/ # cd /usr/share/nginx
/usr/share/nginx # ls -l
total 0
drwxr-xr-x    1 root     root            24 Feb 25  2018 html
/usr/share/nginx # cd html/
/usr/share/nginx/html # ls -l
total 8
-rw-r--r--    1 root     root           537 Jan 10  2018 50x.html
-rw-r--r--    1 root     root            65 Mar  2  2018 index.html
/usr/share/nginx/html # cat index.html
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
/usr/share/nginx/html # echo "123" >> index1.html
/usr/share/nginx/html # exit
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
myapp-pod               1/1     Running   1          73m
readiness-httpget-pod   1/1     Running   0          3m41s

说明：由于index1.html不存在，会导致他重启，手动创建后就正常了。

检测探针 - 存活检测

[root@k8s-master mnt]# cat live-exec.yaml
apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec-pod
  namespace: default
spec:
  containers:
  - name: liveness-exec-container
    image: busybox
    imagePullPolicy: IfNotPresent
    command: ["/bin/sh","-c","touch /tmp/live ; sleep 60; rm -rf /tmp/live; sleep 3600"]
    livenessProbe:
      exec:
        command: ["test","-e","/tmp/live"]
      initialDelaySeconds: 1
      periodSeconds: 3
[root@k8s-master mnt]#

[root@k8s-master mnt]# vim live-exec.yaml
[root@k8s-master mnt]# kubectl create -f live-exec.yaml
pod/liveness-exec-pod created
[root@k8s-master mnt]# kubectl get pod -w
NAME                    READY   STATUS    RESTARTS   AGE
liveness-exec-pod       1/1     Running   0          25s
myapp-pod               1/1     Running   1          81m
readiness-httpget-pod   1/1     Running   0          11m
liveness-exec-pod       1/1     Running   1          101s

liveness-exec-pod       1/1     Running   2          3m19s
^Z
[1]+  已停止               kubectl get pod -w

说明：由于/tmp/live不存在，会一直重启

[root@k8s-master mnt]# cat live-http.yaml
apiVersion: v1
kind: Pod
metadata:
  name: liveness-httpget-pod
  namespace: default
spec:
  containers:
  - name: liveness-httpget-container
    image: wangyanglinux/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 1
      periodSeconds: 3
      timeoutSeconds: 10
[root@k8s-master mnt]#

[root@k8s-master mnt]# kubectl create -f live-http.yaml
pod/liveness-httpget-pod created
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   0          14s
myapp-pod               1/1     Running   1          90m
readiness-httpget-pod   1/1     Running   0          20m
[root@k8s-master mnt]# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
liveness-httpget-pod    1/1     Running   0          26s   10.244.2.12   k8s-node01   <none>           <none>
myapp-pod               1/1     Running   1          90m   10.244.1.9    k8s-node02   <none>           <none>
readiness-httpget-pod   1/1     Running   0          20m   10.244.2.10   k8s-node01   <none>           <none>
[root@k8s-master mnt]# curl 10.244.2.12
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@k8s-master mnt]# curl 10.244.2.12/index.html
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@k8s-master mnt]# kubectl exec liveness-httpget-pod -it -- /bin/sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls -l
total 8
-rw-r--r--    1 root     root           537 Jan 10  2018 50x.html
-rw-r--r--    1 root     root            65 Mar  2  2018 index.html
/usr/share/nginx/html # rm -rf index.html
/usr/share/nginx/html # exit
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   0          2m24s
myapp-pod               1/1     Running   1          92m
readiness-httpget-pod   1/1     Running   0          22m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          2m41s
myapp-pod               1/1     Running   1          92m
readiness-httpget-pod   1/1     Running   0          22m

说明：删除Html,会发现Pod开始重启了。

[root@k8s-master mnt]# cat live-tcp.yaml
apiVersion: v1
kind: Pod
metadata:
  name: probe-tcp
spec:
  containers:
  - name: nginx
    image: wangyanglinux/myapp:v1
    livenessProbe:
      initialDelaySeconds: 5
      timeoutSeconds: 1
      tcpSocket:
        port: 8080
      periodSeconds: 3
[root@k8s-master mnt]#

[root@k8s-master mnt]# vim live-tcp.yaml
[root@k8s-master mnt]# kubectl create -f live-tcp.yaml
pod/probe-tcp created
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          9m24s
myapp-pod               1/1     Running   1          99m
probe-tcp               1/1     Running   0          5s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          9m37s
myapp-pod               1/1     Running   1          99m
probe-tcp               1/1     Running   1          18s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          9m41s
myapp-pod               1/1     Running   1          99m
probe-tcp               1/1     Running   1          22s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          9m43s
myapp-pod               1/1     Running   1          99m
probe-tcp               1/1     Running   1          24s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          9m44s
myapp-pod               1/1     Running   1          99m
probe-tcp               1/1     Running   1          25s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
liveness-httpget-pod    1/1     Running   1          10m
myapp-pod               1/1     Running   1          100m
probe-tcp               1/1     Running   3          47s
readiness-httpget-pod   1/1     Running   0          29m
[root@k8s-master mnt]# kubectl delete -f live-tcp.yaml
pod "probe-tcp" deleted
[root@k8s-master mnt]#

说明：删除Html,会发现Pod开始重启了。

Pod hook

Pod hook （钩子）是由 Kubernetes 管理的 kubelet 发起的，当容器中的进程启动前或者容器中的进
程终止之前运行，这是包含在容器的生命周期之中。可以同时为 Pod 中的所有容器都配置 hook
Hook 的类型包括两种：
  exec ：执行一段命令
  HTTP ：发送 HTTP 请求

重启策略

PodSpec 中有一个 restartPolicy 字段，可能的值为 Always 、 OnFailure 和 Never 。默认为
Always 。 restartPolicy 适用于 Pod 中的所有容器。 restartPolicy 仅指通过同一节点上的
kubelet 重新启动容器。失败的容器由 kubelet 以五分钟为上限的指数退避延迟（ 10 秒， 20 秒， 40
秒 ... ）重新启动，并在成功执行十分钟后重置。如 Pod 文档 中所述，一旦绑定到一个节点， Pod 将
永远不会重新绑定到另一个节点。

 Pod phase

Pod 的 status 字段是一个 PodStatus 对象， PodStatus 中有一个 phase 字段。
Pod 的相位（ phase ）是 Pod 在其生命周期中的简单宏观概述。该阶段并不是对容器或 Pod 的综合汇总，也不是为了做为综合状态机
Pod 相位的数量和含义是严格指定的。除了本文档中列举的状态外，不应该再假定 Pod 有其他的phase 值
几种常见的值

• 挂起（ Pending ）： Pod 已被 Kubernetes 系统接受，但有一个或者多个容器镜像尚未创建。等待时间包括调度 Pod 的时间和通过网络下载镜像的时间，这可能需要花点时间
• 运行中（ Running ）：该 Pod 已经绑定到了一个节点上， Pod 中所有的容器都已被创建。至少有一个容器正在运行，或者正处于启动或重启状态
• 成功（ Succeeded ）： Pod 中的所有容器都被成功终止，并且不会再重启
• 失败（ Failed ）： Pod 中的所有容器都已终止了，并且至少有一个容器是因为失败终止。也就是说，容器以非 0 状态退出或者被系统终止
• 未知（ Unknown ）：因为某些原因无法取得 Pod 的状态，通常是因为与 Pod 所在主机通信失败

[root@k8s-master mnt]# vim post.yaml
[root@k8s-master mnt]# kubectl create -f post.yaml
pod/lifecycle-demo created
[root@k8s-master mnt]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
lifecycle-demo          1/1     Running   0          9s
liveness-httpget-pod    1/1     Running   1          40m
myapp-pod               1/1     Running   1          130m
readiness-httpget-pod   1/1     Running   0          60m
[root@k8s-master mnt]# kubectl exec lifecycle-demo -it -- /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
command terminated with exit code 126
[root@k8s-master mnt]# kubectl exec lifecycle-demo -it -- /bin/sh
/ # cd /usr/share/message
/bin/sh: cd: can't cd to /usr/share/message
/ # cat /usr/share/message
Hello from the postStart handler
/ # exit
[root@k8s-master mnt]# cat post.yaml
apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: wangyanglinux/myapp:v1
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the poststop handler > /usr/share/message"]
[root@k8s-master mnt]#