Deploying Anchore to Kubernetes Cluster using Helm

Question

Kubernetes version 1.14.2 (Client) 1.14.1 (Server) Anchore Chart anchore-engine.1.0.5 App Version 0.4.0

I'm trying to deploy Anchore via Helm to a Kubernetes cluster, but I'm getting failures in every pod.

Helm is properly configured and I've confirmed this by installing several other releases. I install Anchore with

helm install --name anchore-release stable/anchore-engine

Checking pods confirms all of the pods install

anchore-release-anchore-engine-analyzer-659854799f-qkvdn     0/1     Running   0          108s
anchore-release-anchore-engine-api-5d75bbd9d9-9hnms          0/1     Running   0          108s
anchore-release-anchore-engine-catalog-58d7f8bfcd-7d2jz      0/1     Running   0          108s
anchore-release-anchore-engine-policy-6d7579d494-zcg4g       0/1     Running   0          108s
anchore-release-anchore-engine-simplequeue-d8f4f4dbc-62hst   0/1     Running   0          108s

However, none ever enter Ready. Taking a peek at any one of them reveals the issue.

     Type     Reason     Age                   From                Message
  ----     ------     ----                  ----                -------
  Normal   Pulled     17m                   kubelet, XXXXXX  Container image "docker.io/anchore/anchore-engine:v0.4.0" already present on machine
  Normal   Created    17m                   kubelet, XXXXXX  Created container anchore-engine-analyzer
  Normal   Started    17m                   kubelet, XXXXXX  Started container anchore-engine-analyzer
  Normal   Scheduled  17m                   default-scheduler   Successfully assigned default/anchore-release-anchore-engine-analyzer-659854799f-9qdpq to XXXXXX
  Warning  Unhealthy  15m (x5 over 15m)     kubelet, XXXXXX  Liveness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
  Warning  Unhealthy  2m50s (x91 over 17m)  kubelet, XXXXXX  Readiness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused

All of the pod experience liveness/readiness probe failures. In order to examine this closer I moved onto the master node and attempted to query these ports manually. I found that all of the Anchore pods were pingable but had exposed no ports. There is no firewall running anywhere within the cluster, and I installed an Nginx pod to confirm that. The other pod was able to expose and access ports in the 8000 range.

Flannel is serving as the networking manager within the cluster, and due to the Nginx test-deployment properly exposing its ports I'm inclined to believe it's set up properly.

Is anyone familiar with what could be cause the Anchore pods to not expose any of their ports, or another issue which could give the appearance of that? As additional information, here's an example description of one of the pods.

Name:               anchore-release-anchore-engine-analyzer-659854799f-9qdpq
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               XXXXXX/XXX.XXX.XXX.XXX
Start Time:         Wed, 05 Jun 2019 17:41:11 -0400
Labels:             app=anchore-release-anchore-engine
                    component=analyzer
                    pod-template-hash=659854799f
Annotations:        <none>
Status:             Running
IP:                 XXX.XXX.XXX.XXX
Controlled By:      ReplicaSet/anchore-release-anchore-engine-analyzer-659854799f
Containers:
  anchore-engine-analyzer:
    Container ID:  docker://977890fe98a69f6399db55a06e26911befa2e94f98a1dc2d102041132961364e
    Image:         docker.io/anchore/anchore-engine:v0.4.0
    Image ID:      docker-pullable://docker.io/anchore/anchore-engine@sha256:1db43d98a0895cb266658b1cee9db275227b585f19590824ece58ad497bb772f
    Port:          8084/TCP
    Host Port:     0/TCP
    Command:
      anchore-manager
    Args:
      service
      start
      analyzer
    State:          Running
      Started:      Wed, 05 Jun 2019 17:41:13 -0400
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:analyzer-api/health delay=120s timeout=10s period=10s #success=1 #failure=6
    Readiness:      http-get http://:analyzer-api/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      anchore-release-anchore-engine  Secret     Optional: false
      anchore-release-anchore-engine  ConfigMap  Optional: false
    Environment:
      ANCHORE_POD_NAME:  anchore-release-anchore-engine-analyzer-659854799f-9qdpq (v1:metadata.name)
    Mounts:
      /analysis_scratch from analyzer-scratch (rw)
      /config/config.yaml from config-volume (rw,path="config.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nzsfd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      anchore-release-anchore-engine
    Optional:  false
  analyzer-scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-nzsfd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nzsfd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

I appreciate any input. Thanks.

Answer 1

A co-worker ended up taking over deploying this a few days ago and managed to get it working. I'm not certain what he did differently, but I know discovered very similar issues I was facing with other Helm charts boiled down to the database they depended on not deploying properly which caused the other components to fail during initialization.

Database issues were different between the other services I was deploying, but one was rejecting the mounting of a root NFS share. The other appeared to have a mismatch between the base-chart and its packaged-in database chart where the database name and password configuration names were different. Anchore's database was likely failing due to the root-NFS share issue, which then cascaded to other components.