Prodding Helm to refresh a deployment?

Question

I am using Flux to automatically deploy a Helm Release from a git repository to a Kubernetes cluster. If I directly edit a resulting deployment object's spec (using k9s or kubectl edit, e.g. to temporarily try a different version of container image) then Kubernetes will roll out the new pods with no downtime. How can I revert such changes, that is, how can I trigger updating of the deployment (and other objects) to match the hr, without interrupting service?

If I manually delete the hr then the service will briefly go down before I can re-run the Flux pipeline to recreate it. If I materially changed the git repo, then the modifications would auto-deploy without downtime, but re-running the pipeline with inconsequential commits appears not to refresh the deployment.

Answer 1

Say you have a CI pipeline that, upon any git push/merge, logs into a kubernetes cluster and runs the command fluxctl sync (and then uses kubectl rollout status on each deployment object, to wait for completion before following up with tests). The flux daemon/agent itself may also check and perform syncs periodically (as well as checking container registries for new image tags to commit to the manifests in the repo).

Presuming that the git repo consists of helm releases, then flux should ensure that the hr's in the cluster are kept consistent with those in the repo. But flux shouldn't affect any cluster objects that are not directly represented in the repo. Instead, Helm is responsible for ensuring the cluster has deployments matching the hr's. (And then, the deployment controller is responsible for ensuring pods exist matching the deploy spec.)

The flux sync apparently does nothing if the git head commit hasn't changed (or even if the changes in the repo do not appear significant?). If you are unwilling to engineer a new commit, an option with only brief downtime would be: kubectl delete hr foo then fluxctl sync.

Presuming that only the deployment (rather than the hr) has been mutated, another possibility may be to issue helm rollback foo <latest-revision> (although this isn't guaranteed to succeed cleanly).

If the deployment has only been edited once, then kubectl rollout undo deploy foo is appropriate.

Answer 2

Flux is pretty smart and has some built it mechanism to avoid unnecessary updates.

Helm has a version in the Chart.yaml, I'm assuming an inconsequential commit wouldn't update this version so flux might be assuming that it doesn't need to do anything. Changing the version would be something to try.

To avoid the downtime problems, you need to double check your configs. Are you using the replace strategy to the install? Are you using force updates? Both of those strategies would delete the deployments before updating and that might be the cause of your downtime.