I have a Kubernetes cluster with the kube-prometheus-stack Helm chart (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) installed. NGINX is used as the reverse proxy. We are using Kubernetes 1.24.
I am running into a problem when I attempt to access the Grafana dashboard. I am getting the "too many redirects" error. It can even be seen in the browser URL:
I have tried to find what could be causing this and a solution to correct the problem. I found some info at the following site that seems to be very much related, but from what I can tell, what is discussed is specific to a more direct installation, rather than within the context of Kubernetes, Helm charts and container abstractions.
and
https://grafana.com/tutorials/run-grafana-behind-a-proxy/
I think I can see a parallel between some of what's mentioned in the above articles (they have a lot of the same information) and my Helm configuration (values.yaml). However, I have no idea what is being referred to regarding NGINX configuration. I'm not sure where to look to see the NGINX config or how to modify it. There is mention of setting up an NGINX rewrite directive, but I have no idea what that is, how it can be done, or if I even need it.
Here is my Helm chart values.yaml if it's of any help.
kube-prometheus-stack:
defaultRules:
create: true
rules:
alertmanager: true
etcd: true
configReloaders: true
general: true
k8s: true
kubeApiserverAvailability: true
kubeApiserverBurnrate: true
kubeApiserverHistogram: true
kubeApiserverSlos: true
kubeControllerManager: true
kubelet: true
kubeProxy: true
kubePrometheusGeneral: true
kubePrometheusNodeRecording: true
kubernetesApps: true
kubernetesResources: true
kubernetesStorage: true
kubernetesSystem: true
kubeSchedulerAlerting: true
kubeSchedulerRecording: true
kubeStateMetrics: true
network: true
node: true
nodeExporterAlerting: true
nodeExporterRecording: true
prometheus: true
prometheusOperator: true
prometheus:
enabled: true
ingress:
enabled: true
ingressClassName: nginx
hosts:
- prometheus.nnpi.beast-code.com
paths:
- /
pathType: ImplementationSpecific
grafana:
enabled: true
ingress:
enabled: true
ingressClassName: nginx
hosts:
- grafana.nnpi.beast-code.com
path: /
persistence:
enabled: true
# size: 10Gi
grafana.ini:
server:
root_url: grafana.nnpi.beast-code.com
alertmanager:
enabled: true
ingress:
enabled: true
ingressClassName: nginx
hosts:
- alerts.nnpi.beast-code.com
paths:
- /
pathType: ImplementationSpecific
config:
global:
slack_api_url: <path:arc/data/alertmanager#SLACK_API_URL>
route:
receiver: "slack-default"
group_by:
- alertname
- cluster
- service
group_wait: 30s # 30s
group_interval: 5m # 5m
repeat_interval: 1h # 4h
routes:
- receiver: "null"
matchers:
- alertname =~ "InfoInhibitor|Watchdog"
- receiver: "slack-warn-critical"
matchers:
- severity =~ "warning|critical"
continue: true
receivers:
- name: "null"
- name: "slack-default"
slack_configs:
- send_resolved: true # false
channel: "#notify-arc-alerts"
text: |-
{{range .Alerts}}
Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
Summary: {{.Annotations.summary}}
Description: {{.Annotations.description}}
{{end}}
- name: "slack-warn-critical"
slack_configs:
- send_resolved: true # false
channel: "#notify-arc-alerts"
text: |-
{{range .Alerts}}
Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
Summary: {{.Annotations.summary}}
Description: {{.Annotations.description}}
{{end}}
Here is the generated ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-grafana
namespace: prometheus
uid: df852ca5-897e-4d5c-bd6b-09b4a4a45910
resourceVersion: '111566747'
generation: 1
creationTimestamp: '2022-11-09T19:27:17Z'
labels:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 9.3.1
helm.sh/chart: grafana-6.48.2
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"prometheus","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"grafana","app.kubernetes.io/version":"9.3.1","helm.sh/chart":"grafana-6.48.2"},"name":"prometheus-grafana","namespace":"prometheus"},"spec":{"ingressClassName":"nginx","rules":[{"host":"grafana.nnpi.beast-code.com","http":{"paths":[{"backend":{"service":{"name":"prometheus-grafana","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
managedFields:
- manager: argocd-application-controller
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-11-09T19:27:17Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:app.kubernetes.io/version: {}
f:helm.sh/chart: {}
f:spec:
f:ingressClassName: {}
f:rules: {}
- manager: nginx-ingress-controller
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-11-09T19:27:56Z'
fieldsType: FieldsV1
fieldsV1:
f:status:
f:loadBalancer:
f:ingress: {}
subresource: status
selfLink: >-
/apis/networking.k8s.io/v1/namespaces/prometheus/ingresses/prometheus-grafana
status:
loadBalancer:
ingress:
- ip: 10.107.160.141
spec:
ingressClassName: nginx
rules:
- host: grafana.nnpi.beast-code.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-grafana
port:
number: 80
I have also included a section of the nginx.conf file from one of the nginx controller pods running in the cluster. I do not understand this at all, but I'm guessing that a combination of ArgoCD, Helm and Kubernetes is taking care of setting this up.
## start server grafana.nnpi.beast-code.com
server {
server_name grafana.nnpi.beast-code.com ;
listen 80 ;
listen [::]:80 ;
listen 443 ssl http2 ;
listen [::]:443 ssl http2 ;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location / {
set $namespace "prometheus";
set $ingress_name "prometheus-grafana";
set $service_name "prometheus-grafana";
set $service_port "80";
set $location_path "/";
set $global_rate_limit_exceeding n;
rewrite_by_lua_block {
lua_ingress.rewrite({
force_ssl_redirect = true,
ssl_redirect = true,
force_no_ssl_redirect = false,
preserve_trailing_slash = false,
use_port_in_redirects = false,
global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
})
balancer.rewrite()
plugins.run()
}
# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
# other authentication method such as basic auth or external auth useless - all requests will be allowed.
#access_by_lua_block {
#}
header_filter_by_lua_block {
lua_ingress.header()
plugins.run()
}
body_filter_by_lua_block {
plugins.run()
}
log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}
port_in_redirect off;
set $balancer_ewma_score -1;
set $proxy_upstream_name "prometheus-prometheus-grafana-80";
set $proxy_host $proxy_upstream_name;
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
set $proxy_alternative_upstream_name "";
client_max_body_size 0;
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Forwarded-Scheme $pass_access_scheme;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 64k;
proxy_buffers 4 64k;
proxy_max_temp_file_size 1024m;
proxy_request_buffering on;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
}
## end server grafana.nnpi.beast-code.com
I would assume (and hope) that it would not be necessary to somehow modify this directly and that instead, modifying some upstream config (probably in the Helm chart values.yaml) would be propagated downward.
