I have this terraform-kubernetes script terraform1

so prometheus scraping list of pods, and hitting /metrics based on custom resource Prometheus and PodMonitor rules

it's working fine locally using minikube,

but in GKE, it doesn't work,

podmonitor works normally:

service discovery works

but scraping failed (this part is green when on local minikube):

scraping targets

normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:

metrics exists using port-forwrding

does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on can scrape other pods in the target list (,,,,, etc)

1 Answer 1


Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific Host: header, that's why it's 403 instead of timeout or something else.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.