Pod to pod communication GKE

Question

I have this terraform-kubernetes script terraform1

so prometheus scraping list of pods, and hitting /metrics based on custom resource Prometheus and PodMonitor rules

it's working fine locally using minikube,

but in GKE, it doesn't work,

podmonitor works normally:

but scraping failed (this part is green when on local minikube):

normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:

does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on 10.63.1.4 can scrape other pods in the target list (10.63.1.252, 10.63.1.5, 10.63.3.5, 10.63.4.210, 10.63.1.133, etc)

Answer 1

Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific Host: header, that's why it's 403 instead of timeout or something else.