I have a simple storage setup:
[ GKE [ NFS Server -> PV -> PVC -> Deployment (Min 2 pods-HPA) ] ]
GKE version: 1.19.12-gke.2101
The GKE cluster has 4 nodes before and I resize down to 3 nodes then the deployment can't mount the NFS volume to the second pods which works fine for the first one.
It looks like this, stuck there forever:
namespace pods-name-768dd46757-t5l2d 1/1 Running 0 19m namespace pods-name-768dd46757-zqr9l 0/1 ContainerCreating 0 18m
Unable to attach or mount volumes: unmounted volumes=[my-nfs-volume]: timed out waiting for the condition
Here is my PV, PVC, nothing special,
accessModes is set to RWX
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-nfs-volume namespace: my-namespace spec: accessModes: - ReadWriteMany resources: requests: storage: 100Mi storageClassName: "" volumeMode: Filesystem volumeName: my-nfs-volume --- apiVersion: v1 kind: PersistentVolume metadata: name: my-nfs-volume spec: accessModes: - ReadWriteMany capacity: storage: 100Mi nfs: path: / server: my-nfs-server.my-namespace.svc.cluster.local persistentVolumeReclaimPolicy: Retain volumeMode: Filesystem
When I resize the cluster back to 4 nodes and restart deployment, everything is back to normal again. That's odd. Any ideas?