6

For a project I'm working on, we need to create a Jenkins node in an AWS environment. For this task, we will work with the docker image, which is available on https://github.com/jenkinsci/docker-ssh-slave . The Jenkins node will be responsible for executing AWS commands which can only be done from inside the AWS environment (not outside).

This Jenkins Master is running in our local environment. We will add the node as a permanent node in the Jenkins master. The Jenkins master is not responsible for setting up the Jenkins Node, since it has no permissions to do this ( this is managed in AWS ).

However, we would like to monitor the Jenkins Slave inside AWS to see if it it still running successfully. With other docker containers, we also expose a health check. If the health check fails, the docker container is treated as unhealthy, and is restarted accordingly. Just like we do with the other docker containers, we would like to expose such a health port as well on the Jenkins Slave. If it is not responding anymore, AWS will restart the docker container, and we have a stable system.

We already explored the Nagios monitoring, but it looks like this is exposing the health in the master, and not directly in the slave.

Is it available, or how can we add such functionality? Could you lead us in the right direction, or recommend plugins which need to be installed, ...

  • Can we open the default HTTP API of Jenkins in such a Jenkins slave (https://wiki.jenkins.io/display/JENKINS/Remote+access+API)
  • Another approach could be to add and start NGINX in docker and point the healthcheck to NGINX. But with this approach, we are not really checking the health of the the Jenkins slave.
Improve this question
2
  • 1
    I would really recommend pursuing actual monitoring further (e.g. Nagios, monit, etc.), since Jenkins is not intended to be a monitoring tool.
    – jayhendren
    Jun 19, 2018 at 17:51
  • 1
    @jayhendren: I think the question is not clear, we are not trying to monitor our slaves with Jenkins. We are using AWS Cloudwatch as a monitoring tool., But in order to do some things we need an HTTP-endpoint in our (docker) slaves where we can verify if this slave is still up and behaving correctly. At this moment, i have no idea how to do this. So any help is welcome.
    – Rob Van Pamel
    Jun 20, 2018 at 10:30

2 Answers 2

Reset to default
1

To me it looks like you are trying to reinvent the wheel. As I understood you would like to restart a container if a health check fails. As you are using docker you should use a orchestration platform in my opinion like Fargate (ECS) or Kubernetes (k8s). I know from the latter that it is possible to define healthchecks, like readiness and health probes. If the health check fails, kubernetes will restart the POD (auto healing).

It could be possible that you would argue that k8s is not an option. If that is the case, you could also have a look to AWS OpsWorks. Although I did not try it myself yet, it is able to perform auto healing as well.

Improve this answer
2
  • we are using ECS for the healthchecks, only problem was that Jenkins slaves do not expose a Health check on their container images. So i was wondering what the best solution would be to add such a healthcheck on a Jenkins Slave: 1) Creating a custom image, and add NGINX which exposes a healthcheck or 2) try to use that Jenkins API in a slave which doesnt seem to work. We used option 1 for now and this is working fine.
    – Rob Van Pamel
    Aug 23, 2019 at 14:21
  • Could you not do something very basic such as check if a process is running or check for exceptions in a log? Historically this is how you would check for arbitrary services on *nix systems.
    – Grizzle
    Jan 13, 2021 at 10:34
-1

Take a look at this Jenkins Plugin: https://plugins.jenkins.io/slave-status/

It'll expose a healthcheck endpoint + other information like memory on port 3141 (you can configure a custom port). You can then ping that healthcheck endpoint and send the data to CloudWatch.

Improve this answer
2
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review
    – Bruce Becker
    Mar 5, 2021 at 13:22
  • @BruceBecker updated
    – Akram Fares
    Mar 5, 2021 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.