1

I have some doubts about the HTTP readinessProbe:

  1. Suppose that the probe fails several times (failureThreshold) and a pod is marked as NOT READY. Will k8s keep trying the readinessProbe forever and eventually mark the pod as Ready if it begins to respond correctly? Or a pod that is marked as Not Ready is left there forever without retrying?
  2. In case the Not Ready pod is left there forever, will the K8s Deployment try to start new pods? Or I will be left with less pods available?
  3. If a pod passes the readinessProbe, will k8s keep checking if it is ready and eventually (if it begins to fail) mark it as not ready in the future? Is that possible? Or a pod that is marked once as ready is ready forever?

1 Answer 1

1

1 - yes, containers are left there forever. Think of readiness probe as a breaker, if the app gets overwhelmed it cuts the traffic to the pod until it's ready to get traffic again.

2 - no, by itself, readiness probes only stop/start traffic to a pod. To signal kubernetes that the pod needs to be restarted use liveness probes.

3 - yes, as per answer 1. Readiness probes are a way of letting kubernetes know that the pod can handle traffic or if they need a break.

probe: startup probes - waits for the app to start

use case: apps that need some time before start serving traffic, for example, building assets on start, pre-fetching assets, building caches, etc.

liveness probes - checks if the app is healthy

use case: check if the app is running, if the app has exited due to an exception and needs to be restarted, or if the app is running but can't serve requests

readiness probes - checks if the app can handle traffic

use case: apps that can arbitrarily stop receiving traffic, e.g. maintenance mode, data intensive apps that can degrade in performance and need a way to

You should use liveness and readiness probes always, use startup probes when your app takes more time than usual to start.

Example:

  • python app with postgres database:
  • liveness probe: get /health status 200
  • readiness probe: get /health status 200, timeout 1s

the /health endpoint queries the database using a simple query (select id, email from user limit 1) and returns a json

if the pod loses connectivity to the database, the liveness probe will fail and the pod will be restarted. if the database is being overwhelmed by requests, the query will take more than 1s, the probe will mark the pod as not ready, preventing that additional requests reach the database and preventing it from progressing from overwhelmed to crashed. once the database is able to catch up and starts returning queries in less than 1s, readiness probe will mark the pod as healthy again and kubernetes will resume serving requests to the pod.

2
  • Thanks for the explanation! However I read that liveness probe shouldn't test a different component (like the database), because a failure in one component can damage the availability of entire application (with continuous restarts)
    – collimarco
    Nov 15, 2021 at 12:32
  • yup! modern applications rely on several components like files, databases, caches, and even external services. whatever you do will cause a chain reaction in the entire ecosystem if the other services aren't prepared, so there's no general answer to this. The general frame of mind would be: can your app do anything without the database? like validating JWT tokens, serving pages directly from redis, etc. If the database goes down, is your app is worthless? I used the database example, but you should think on checking a route that will give you a reliable response time and not a false positive.
    – Magus
    Nov 15, 2021 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.