The context of the question is Secrets management.
As the title says, how do you change a secret that half of your production applications use right now?
DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.Sign up to join this community
This question ties in heavily with your other question with where the secrets are set and where they retrieved from.
I assume your question is asking about 'canary deployments' where you change the config for only a small portion of your apps, to test things out before deploying everywhere.
If you follow the best practice of using the 12 Factor app's recommendation of 'environment variables', then changing secrets becomes very easy. At the very worst, you could log into the production server, update the ENV VAR, and then restart your app manually. In Kubernetes, this is even easier, as you can edit the ENV variables from the UI, and then restart the pod.
However, you don't want to be doing that manual process if you can avoid it. If your Configuration Management tool sets the ENV VAR, then you can tell it to update a specific group of servers only. If your build server injects the ENV vars, and it supports canary deployments, then you can easily just do a canary deployment like normal.
i wouldn't 'change' the secret at least in the short term but roll out a second secret with the same permission set as the original and roll that out through your configuration management or secrets management. ideally following @boomshadow 's answer. once the new credential is fully rolled out invalidate the old one/s repeat as necessary.