Monitor program progress on multiple servers

Question

We have three servers that are running python programs that are running data analysis tasks inside a tmux session. The method we are using at the moment is ssh'ing into each of them connecting the tmux session and watching the output on the command line.

This method is tedious, so what we are looking for is a solution that automates monitoring of program progress(output on CLI) for multiple servers at the same time. We would ideally like a web UI solution but a CLI would also be perfectly suitable.

Thank you for reading.

Answer 1

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling.

Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the simplest thing is probably to start sending the output through syslog to a centralized logging system (syslog itself, or ELK, or whatever). That way you can monitor everything from a central location.

Thinking forward, if this isn't a one-off task, monitoring should be automated. That is, you should never have to just watch logs to see if things are progressing as they're supposed to. Instead, you should assume they are (and continue with other work) until your alerting fires off. This is an investment of time into getting reliable and wide-coverage alerting, but as your systems grow in complexity, it will pay off as you don't have to monitor everything any time you change anything.

Answer 2

Graylog

Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: Graylog.

I implemented and used a couple of ELK stacks for both log aggregation and business intelligence and also run/maintain graylog for about two years now at my current employer. I recommend graylog since it has the following features built-in and is - in my opinion - a bit easier to setup and maintain:

• A web-interface
• Multiuser capabilities
• Alerting

As far as I understand your scenario, it looks as if you need to act or be alerted on certain events that appear in your stream of log messages. If we look at the Graylog features:

Trigger actions or get notified when something needs attention, such as failed login attempts, exceptions or performance degradation.

Ideas: Send an email or Slack message to your team. Spawn a new machine to balance the processing load. Block IP ranges in your firewalls automatically when an attack is detected.

To give graylog a try I'd recommend the following two steps:

• Setup a dedicated host which is reachable by all you application hosts to run graylog (and its dependencies MongoDB and ElasticSearch)
• Send logs from your application to graylog (possibly as GELF messages)

Note: These two steps have the ability to fill pages and pages of best practices and should receive at least a couple of thoughts. Not to mention that graylog isn't a monitoring solution and graylog itself should be monitored by a proper monitoring tool (like e.g. Icinga, Prometheus, Nagios to name just a few).

Answer 3

I agree with @Xiong Chiamiov and I want to give more clarify option. If you want every line in the CLI to be monitored, I would suggest to redirect all the output to specific file and the error to another file, then use logstash or filebeat to send both of this files to Elasticsearch, then you can configure Logtril with Kibana to give you view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface

Answer 4

centralized tmux

While the other answers are smarter and wiser for the long term I think the quick hacky CLI solution is worth mentioning. Run tmux on one server that can reach all of the others. A good place for this would be a jump box or some other place that folks are commonly logged in anyway. Within this "central" tmux ssh to each box in a different pane and tail whatever log files are necessary. You can use ctrl-b " to get more panes in one tab within tmux. Now all someone has to do to check things is attach to the "central" tmux session and they can see the whole cluster at a glance.

I spent a lot of time building the web UI solutions that you are working toward, but if you need it today hacking together something with tmux can save the day.