4

For less trafficked parts of your web service, synthetic transactions are a good option. There are some trade-offs to be made when choosing either using an API to or using a full browser style check. API checks require changing the application being tested. I'd try and go this route first, but it is dependant on the functionality you need to test. These can ...


2

PD has a lot of native integrations that can do auto-resolve but for those that don't you can use the API and/or rulesets to do that. Here is a link to the ruleset documentation: https://support.pagerduty.com/docs/rulesets And here is documentation on the incident resolution API: https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/...


2

This is conventionally called a watchdog, deadman, or health check. Bruce's comment is good, including the part about a raspberry pi. You could also spin up something simple like monit in a container someplace like GCE or AWS. Other than setting up an external service or server, there are also a bunch of phone apps that will do the pinging and/or http ...


2

You can copy the actual logs from /var/lib/docker/containers/<container id>/<container id>-json.log to archive them then clear the logs manually (since you have only 2 services, influxdb and grafana). The containers won't stop with this procedure. There is a full and clear answer on this StackOverflow question about clearing container logs. It ...


1

In situation like this I tend to visit the project and dig in some code. For this particular case this is the PR that the metric was introduced in - https://github.com/argoproj/argo-cd/pull/1184. You can go from there and do some more research. argocd_app_reconcile_count is metric related to Argo CD Application Controller, basically each time a reconcile ...


1

I propose the following solution although it will require 1 extra metricbeat running per monitored entity, it's simple, not optimized. The approach will also work for other non-kubernetes contexts like docker-compose with a little bit of tweaking. Deploy sidecar (extra container per pod) to monitor something (in this example, it's logstash but doing same ...


1

There could be multiple ways to solve this. Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like failed-job in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use watcher here) Don't need to read logs. ...


1

You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box https://docs.docker.com/config/containers/logging/syslog/ { "log-driver": "syslog", "log-opts": { "syslog-address": "udp://...


1

This is more of an opinion based question but let me try to answer it. installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research? Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor? ...


1

To piggy back off of @eduardo-baitello answer, A third option is to increase the CPUThrottlingPercent config here


1

Here's my two pieces of advice: Work out schedules so that you're not on-call 24 hours a day. There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week. Get in the habit of taking notes ...


Only top voted, non community-wiki answers of a minimum length are eligible