0

I have a simple storage setup:

[ GKE [ NFS Server -> PV -> PVC -> Deployment (Min 2 pods-HPA) ] ]

GKE version: 1.19.12-gke.2101

The GKE cluster has 4 nodes before and I resize down to 3 nodes then the deployment can't mount the NFS volume to the second pods which works fine for the first one.

It looks like this, stuck there forever:

namespace   pods-name-768dd46757-t5l2d              1/1     Running                 0          19m
namespace   pods-name-768dd46757-zqr9l              0/1     ContainerCreating       0          18m
Unable to attach or mount volumes: unmounted volumes=[my-nfs-volume]: timed out waiting for the condition

Here is my PV, PVC, nothing special, accessModes is set to RWX

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-nfs-volume
  namespace: my-namespace
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Mi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: my-nfs-volume
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-volume
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 100Mi
  nfs:
    path: /
    server: my-nfs-server.my-namespace.svc.cluster.local
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

When I resize the cluster back to 4 nodes and restart deployment, everything is back to normal again. That's odd. Any ideas?

2
  • I saw likely the same issue long back, it seems like an issue with your persistentVolumeReclaimPolicy: Retain. It should be persistentVolumeReclaimPolicy: delete if you want that PVC will get delete when you reduce the number of replication counts. Sep 14 at 18:46
  • Hi @ArvindRawat, Thank you for your suggestion, I tried but the problem still exists :(( Sep 15 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.