Consul Helm chart: Connect Injector service not working

Question

Below is my config that I am applying over the latest version of the chart (0.4.0).

The consul client daemonset pods come up and successfully register with the cluster. The connect injector webhook deployment comes up and shows as running successfully. This the is pod startup log:

Listening on ":8080"... Updated certificate bundle received. Updating certs...

10.42.0.1 - - [18/Dec/2018:20:36:26 +0000] "GET /health/ready HTTP/2.0" 204 0

From then on, I tried some sample deployment using the annotation: "consul.hashicorp.com/connect-inject": "true" This did not spin up any sidecar containers. Checking the injector pod, there were no messages that the deployment was even contacted.

Initially I had issues with this in a cluster without RBAC. I removed the RBAC references from the chart and deployed; but had the issue above. I then setup/enabled RBAC on another cluster and the chart installed correctly. This did not work either even with the correct RBAC settings in place.

I am using Vault as a certificate provider for Consul; checking both logs, I don't see any errors, and Consul is setup correctly as the provider.

Does anybody have any ideas on what to check?

This is my config:

global:
  enabled: false
  domain: consul
  image: "consul:1.4.0"
  imageK8S: "hashicorp/consul-k8s:0.3.0"
  datacenter: dc1


client:
  enabled: true
  join: ["provider=aws region=region tag_key=tag tag_value=tag"]
  grpc: true
  extraConfig: |
    {
      "enable_local_script_checks": true,
      "encrypt": "key"
    }


dns:
  enabled: true


syncCatalog:
  enabled: false


connectInject:
  enabled: true
  default: false
  imageEnvoy: "envoyproxy/envoy:v1.8.0"

Thanks.

Answer 1

I was reading the documentation to learn more about the admission controllers, when I read the prerequisites; the pertinent one being

Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers are enabled

Per the docs there is a recommended set of controllers to use:

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota

I manage K8s clusters with Rancher in non-test environments. By default, Rancher only enables a few of these controllers:

NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

When I upgraded the cluster, this did not get enabled as I suspected with just the version upgrade. After updating the Rancher K8s configuration and applying the changes, I successfully spun up the test client/server deployments which successfully injected the envoy containers and registered the services in Consul.

TL;DR: RTFM