You can copy the actual logs from /var/lib/docker/containers/<container id>/<container id>-json.log to archive them then clear the logs manually (since you have only 2 services, influxdb and grafana). The containers won't stop with this procedure.
There is a full and clear answer on this StackOverflow question about clearing container logs. It ...
You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box
This is more of an opinion based question but let me try to answer it.
installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?
Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?
In terms of embedding in an external page, you have some options out of the box with both Kibana (docs here) and Grafana (docs here).
If I understand correctly, you are trying to generate realistic clickable thumbnails - in that case, you should try Grafana first since it better supports server-side image rendering
I think to have found a suitable grafana plugin that can activate/deacttivate an actuator by posting a value with a call api rest. The name of this plugin that can be easily donwloaded and istalled in the grafana is cloudspout button panel plugin. and can be found at grafana.com/grafana/plugins/cloudspout-button-panel