I have been trying for a few days to set up a multi-target MySQL exporter (MSE) deployed with Helm inside a k8s cluster. The chart that I am using is the https://artifacthub.io/packages/helm/prometheus-community/prometheus-mysql-exporter/2.2.0 (latest as of today).
It works when I try to connect directly to the primary DB without using the multi-target config in values.yaml/serviceMonitor : I can see the data in my grafana.
I see in the documentation that I can specify multiple targets for my MSE and have filled the values.yaml accordingly (see values.yaml below). I have enabled the multi-target and have specified that the password is not the same for both targets.
My targets are both MariaDB databases : one is external to the cluster (accessible by IPv4) and the other one is an helm-deployed MariaDB inside the cluster.
Some attention points :
- exporter user is present on both the primary and the replica, with permission access from everywhere (%) for testing purposes.
my.cnfis deployed as a secret in the cluster and recovered successfully by the mysql-exporter pod in/etc/cnf/my.cnf(see file below)
What I get when I do helm upgrade is the following, from my pod's logs :
caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
Questions :
Is it a stable feature ?
Is the documentation accurate when it says to not use [client] but instead [client.name] ?
Can it work if one of both databases connection is not successful ?
How to make it work ?
my.cnf
[client]
user=NOT_USED
password=NOT_USED
[client.mariadb-staging]
user=exporter
password=XXXXXXXXX
[client.replica-1]
user=exporter
password=XXXXXXXXXX
values.yaml
## Default values for prometheus-mysql-exporter.
## This is a YAML-formatted file.
## Declare variables to be passed into your templates.
## namespaceOverride overrides the namespace which the resources will be deployed in
namespaceOverride: ""
## override release name
fullnameOverride: ""
replicaCount: 1
image:
registry: quay.io
repository: prometheus/mysqld-exporter
## if not set charts appVersion var is used
tag: ""
pullPolicy: "IfNotPresent"
# imagePullSecrets:
# - name: secret-name
imagePullSecrets: []
service:
labels: {}
annotations: {}
name: mysql-exporter
type: ClusterIP
externalPort: 9104
internalPort: 9104
serviceMonitor:
# enabled should be set to true to enable prometheus-operator discovery of this service
enabled: true
# interval is the interval at which metrics should be scraped
interval: 30s
# scrapeTimeout is the timeout after which the scrape is ended
scrapeTimeout: 10s
namespace: monitoring
# additionalLabels is the set of additional labels to add to the ServiceMonitor
additionalLabels: {}
jobLabel: ""
targetLabels: []
podTargetLabels: []
metricRelabelings: []
# Set relabel_configs as per https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelings: []
# Enable multi target scraping.
# target connection information with name (required), endpoint (required) and port (optionnal)
# if sharedSecret is not enabled the name must match an entry client.{{ name }} existing in the secret
multipleTarget:
enabled: true
targets:
- name: mariadb-staging
endpoint: xx.xxx.xxx.32
port: 3306
- name: replica-1
endpoint: metabase-mariadb.database.svc.cluster.local
port: 3306
# Enable shared credentials for all targets
sharedSecret:
enabled: false
name: ""
serviceAccount:
# Specifies whether a ServiceAccount should be created
create: false
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
annotations: {}
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
#limits:
# cpu: 100m
# memory: 128Mi
#requests:
# cpu: 100m
# memory: 128Mi
nodeSelector: {}
tolerations: []
affinity: {}
podLabels: {}
# Extra Volume Mounts for the mysql exporter container
extraVolumeMounts: []
# - name: example
# mountPath: /example
# Extra Volumes for the pod
extraVolumes: []
# - name: example
# configMap:
# name: example
podSecurityContext: {}
# fsGroup: 65534
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 65534
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9104"
config: {}
# Allow to set specifc options on the exporter
# logLevel: info
# logFormat: "logger:stderr"
collectors:
# auto_increment.columns: false
# binlog_size: false
engine_innodb_status: true
# engine_tokudb_status: false
# global_status: true
# global_variables: true
# info_schema.clientstats: false
# info_schema.innodb_metrics: false
# info_schema.innodb_tablespaces: false
# info_schema.innodb_cmp: false
# info_schema.innodb_cmpmem: false
# info_schema.processlist: false
# info_schema.processlist.min_time: 0
# info_schema.query_response_time: false
# info_schema.tables: true
# info_schema.tables.databases: '*'
# info_schema.tablestats: false
# info_schema.schemastats: false
# info_schema.userstats: false
# perf_schema.eventsstatements: false
# perf_schema.eventsstatements.digest_text_limit: 120
# perf_schema.eventsstatements.limit: false
# perf_schema.eventsstatements.timelimit: 86400
# perf_schema.eventswaits: false
# perf_schema.file_events: false
# perf_schema.file_instances: false
# perf_schema.indexiowaits: false
# perf_schema.tableiowaits: false
# perf_schema.tablelocks: false
# perf_schema.replication_group_member_stats: false
slave_status: true
# slave_hosts: false
# heartbeat: false
# heartbeat.database: heartbeat
# heartbeat.table: heartbeat
# mysql connection params which build the my.cnf config
mysql:
db: ""
host: ""
## config my.cnf https://dev.mysql.com/doc/c-api/8.0/en/mysql-options.html
additionalConfig:
- connect-timeout=5
- debug
pass: ""
port: ""
protocol: ""
user: ""
# secret with full config my.cnf
existingConfigSecret:
name: "config-mysql-exporter"
key: "my.cnf"
# secret only containing the password
#existingSecret: "connection-string-mariadb"
#existingPasswordSecret:
# name: ""
# key: ""
