Since I'm using Jenkins official helm chart which contains JCasC plugin (Jenkins Configuration as Code), I was able to configure the static slave attachment using it including the creation of relevant credentials.
Here's the relevant section of the values.yaml file I use to install/upgrade the Jenkins helm chart:
description: "Username/Password Credentials for CentOS_7.1_k4.14.11"
name: "CentOS 7.1 k4.14.11"
jvmOptions: "-Xms256m -Xmx1024m -XX:MaxPermSize=1024m"
Now whenever I re-deploy or upgrade Jenkins helm chart, it automatically configures these credentials and add the static slave.