处理K8S PVC删除后pod报错

报错如下

Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.381558    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.581422    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.781432    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.981401    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.181612    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.381434    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.581538    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.781372    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.981466    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.182079    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.381529    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted

处理办法

查找到该pod详情，将控制器删除即可

[root@master ~]# kubectl get po -A|grep gr8333e7
672f06d5992f4b4580ae04289e33dde4   gr8333e7-0

[root@master ~]# kubectl describe po gr8333e7-0  -n 672f06d5992f4b4580ae04289e33dde4
Name:         gr8333e7-0
Namespace:    672f06d5992f4b4580ae04289e33dde4
Priority:     0
Node:         node1/172.31.200.68
Start Time:   Fri, 19 Jun 2020 14:54:34 +0800
Labels:       controller-revision-hash=gr8333e7-595ff46986
              creater_id=1592549674520418745
Annotations:  rainbond.com/tolerate-unready-endpoints: true
Status:       Running
IP:           10.244.3.196
IPs:
  IP:           10.244.3.196
Controlled By:  StatefulSet/gr8333e7
Containers:
  f6719b6d0f2adace1d930dc5f48333e7:
    Container ID:   docker://2258ee0b766f2ce261563de4bda331f8bcb172ec474f0c78a9e0627eb6dbe708
    Image:          goodrain.me/f6719b6d0f2adace1d930dc5f48333e7:20200619145017
    Image ID:       docker-pullable://goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab@sha256:8e4eca55761ebadacc6503acced877fa69689389b27c56a15ba165810e563e31
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 19 Jun 2020 14:54:36 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1280m
      memory:  1Gi
    Requests:
      cpu:      240m
      memory:   1Gi
    Readiness:  tcp-socket :3306 delay=4s timeout=5s period=3s #success=1 #failure=3
    Environment:
      LOGGER_DRIVER_NAME:      streamlog
      REVERSE_DEPEND_SERVICE:  gr512123:28d93ce6688d13325dc7986169512123,gr58ee27:599b46254ee0690d3ee750b5ab58ee27,gr5efe93:9c69edab427540f0aecc9bd0bb5efe93
      DB_HOST:                 127.0.0.1
      DB_PORT:                 3306
      TENANT_ID:               672f06d5992f4b4580ae04289e33dde4
      SERVICE_ID:              f6719b6d0f2adace1d930dc5f48333e7
      MEMORY_SIZE:             large
      SERVICE_NAME:            gr8333e7
      SERVICE_POD_NUM:         1
      HOST_IP:                  (v1:status.hostIP)
      POD_IP:                   (v1:status.podIP)
    Mounts:
      /var/lib/mysql from manual165 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lctjg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  manual165:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  manual165-gr8333e7-0
    ReadOnly:   false
  default-token-lctjg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-lctjg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[root@master ~]# grctl service get gr8333e7 -t npwrtv4l
Namespace:      	672f06d5992f4b4580ae04289e33dde4
ServiceID:      	f6719b6d0f2adace1d930dc5f48333e7
ReplicationType:	statefulset
ReplicationID:  	gr8333e7
Status:         	running
------------Service------------
+---------------------+----------------+------------+
| Name                | IP             | Port       |
+---------------------+----------------+------------+
| gr8333e7            | None           | (TCP:3306) |
| service-392-3306    | 10.108.223.74  | (TCP:3306) |
| service-392-3306out | 10.105.125.132 | (TCP:3306) |
+---------------------+----------------+------------+

------------Ingress------------
+------+------+
| Name | Host |
+------+------+
+------+------+

-------------------Pod_1-----------------------
PodName:      	gr8333e7-0
PodStatus:    	Initialized : True  Ready : True  ContainersReady : True  PodScheduled : True
PodIP:        	10.244.3.196
PodHostIP:    	172.31.200.68
PodHostName:  	node1
PodVolumePath:
PodStratTime: 	2020-06-19T14:54:34+08:00
Containers:
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
| ID           | Name                             | Image                                                       | State                              |
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
| 2258ee0b766f | f6719b6d0f2adace1d930dc5f48333e7 | goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab:20200424215058 | Running(2020-06-19T14:54:36+08:00) |
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+

[root@master ~]# kubectl delete sts gr8333e7 -n 672f06d5992f4b4580ae04289e33dde4
statefulset.apps "gr8333e7" deleted

查看日志即正常