2

We have an issue in our GKE cluster that started suddenly yesterday where intermittently we get errors interacting with the control plane. This affects our Jenkins builds which use the Kubernetes plugin, but also even simply running and attaching to pods. For example, if we do a kubectl run -i --tty --rm --image=busybox some-name, about 20% of the time we'll get the following error:

Error attaching, falling back to logs: error dialing backend: dial timeout

Hoping someone can help suggest how we can diagnose further.

Verbose logs from the kubectl run:

I0421 09:38:47.720435    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods
I0421 09:38:47.720448    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.720452    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.720456    2483 round_trippers.go:431]     Content-Type: application/json
I0421 09:38:47.720460    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.852510    2483 round_trippers.go:446] Response Status: 201 Created in 132 milliseconds
I0421 09:38:47.860462    2483 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:47.860489    2483 reflector.go:211] Listing and watching *v1.Pod from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:47.860610    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsome-name&limit=500&resourceVersion=0
I0421 09:38:47.860619    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.860628    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.860633    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.912301    2483 round_trippers.go:446] Response Status: 200 OK in 51 milliseconds
I0421 09:38:47.919196    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name&resourceVersion=298429547&timeout=6m8s&timeoutSeconds=368&watch=true
I0421 09:38:47.919212    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.919219    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.919224    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.964479    2483 shared_informer.go:253] caches populated
I0421 09:38:48.002537    2483 round_trippers.go:446] Response Status: 200 OK in 83 milliseconds
I0421 09:38:49.416988    2483 reflector.go:181] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:49.417115    2483 attach.go:338] defaulting container name to some-name
If you don't see a command prompt, try pressing enter.
I0421 09:38:49.417419    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods/some-name/attach?container=some-name&stdin=true&stdout=true&tty=true
I0421 09:38:49.417431    2483 round_trippers.go:427] Request Headers:
I0421 09:38:49.417436    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0421 09:38:49.417440    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
I0421 09:38:49.417444    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
I0421 09:38:49.417447    2483 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
I0421 09:38:49.417451    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:19.524483    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30106 milliseconds
Error attaching, falling back to logs: error dialing backend: dial timeout
I0421 09:39:19.525463    2483 attach.go:338] defaulting container name to some-name
I0421 09:39:19.525813    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name/log?container=some-name
I0421 09:39:19.525829    2483 round_trippers.go:427] Request Headers:
I0421 09:39:19.525837    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:19.525843    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.585194    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30059 milliseconds
I0421 09:39:49.591557    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name
I0421 09:39:49.591578    2483 round_trippers.go:427] Request Headers:
I0421 09:39:49.591585    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:49.591611    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.636553    2483 round_trippers.go:446] Response Status: 200 OK in 44 milliseconds
I0421 09:39:49.639346    2483 round_trippers.go:420] DELETE https://35.189.86.41/api/v1/namespaces/default/pods/some-name
I0421 09:39:49.639366    2483 round_trippers.go:427] Request Headers:
I0421 09:39:49.639373    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:49.639379    2483 round_trippers.go:431]     Content-Type: application/json
I0421 09:39:49.639386    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.692164    2483 round_trippers.go:446] Response Status: 200 OK in 52 milliseconds
pod "some-name" deleted
I0421 09:39:49.692460    2483 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://10.31.60.49:10250/containerLogs/default/some-name/some-name\": dial timeout",
  "code": 500
}]
F0421 09:39:49.692479    2483 helpers.go:115] Error from server: Get "https://10.31.60.49:10250/containerLogs/default/some-name/some-name": dial timeout
Improve this question

1 Answer 1

Reset to default
1

Google Support tracked down the issue down for us:

this issue appears to have been caused by a known Issue affecting Konnectivity, under some circumstances a memory leak could occur, in the case of your cluster, this appears to end up causing connection timeouts after some time, this issue has already been addressed on the Konnectivity network proxy [1], and the fix is already available within GKE on version 1.19.9 or higher.

[1] https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/168

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.