GKE - Persistent Volume not able to mount to the if cluster have less than 4 nodes

Question

I have a simple storage setup:

[ GKE [ NFS Server -> PV -> PVC -> Deployment (Min 2 pods-HPA) ] ]

GKE version: 1.19.12-gke.2101

The GKE cluster has 4 nodes before and I resize down to 3 nodes then the deployment can't mount the NFS volume to the second pods which works fine for the first one.

It looks like this, stuck there forever:

namespace   pods-name-768dd46757-t5l2d              1/1     Running                 0          19m
namespace   pods-name-768dd46757-zqr9l              0/1     ContainerCreating       0          18m

Unable to attach or mount volumes: unmounted volumes=[my-nfs-volume]: timed out waiting for the condition

Here is my PV, PVC, nothing special, accessModes is set to RWX

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-nfs-volume
  namespace: my-namespace
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Mi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: my-nfs-volume
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-volume
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 100Mi
  nfs:
    path: /
    server: my-nfs-server.my-namespace.svc.cluster.local
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

When I resize the cluster back to 4 nodes and restart deployment, everything is back to normal again. That's odd. Any ideas?