Log/Data rotation in Docker Container

Question

My question is more on the concept or best practices to follow.

I have two docker containers running InfluxDB and Grafana (monitoring applications). I have mounted external volume to each.

now, the problem is, these containers are running for quite some time (1 year approx), i am little bit worried about Host VM resources on which these containers are running. specifically storage resources as these are monitoring applications, which may be storing some data for historical purpose as well.

So, my ask is, what is the best practice to manage logs/data stored in docker volume. I don't want my Host VM to go out of storage and crash my containers.

Any possible solution, if we can rotate stored data so that my container won't stop and i have the historical data as well. or if possible from application point of view (InfluxDB and Grafana) to archive old data.

Thanks in advance... Please let me know if you need more details on this.

below is my docker-compose.yml file

version: "2"
services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    user: root
    privileged: true
    restart: always
    ports:
      - 3000:3000
    networks:
      - monitoring
    volumes:
      - grafana-volume:/var/lib/grafana
  influxdb:
    image: influxdb
    container_name: influxdb
    restart: always
    ports:
      - 8086:8086
    networks:
      - monitoring
    volumes:
      - influxdb-volume:/var/lib/influxdb
networks:
  monitoring:
volumes:
  grafana-volume:
    external: true
  influxdb-volume:
    external: true

Answer 1

You can copy the actual logs from /var/lib/docker/containers/<container id>/<container id>-json.log to archive them then clear the logs manually (since you have only 2 services, influxdb and grafana). The containers won't stop with this procedure.

There is a full and clear answer on this StackOverflow question about clearing container logs. It also explains how to configure the Docker daemon to rotate logs.

From this question there's a one-liner that you can run:

echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)

[...]

You can also set this as part of your daemon.json file instead of modifying your startup scripts:

{
  "log-driver": "json-file",
  "log-opts": {"max-size": "10m", "max-file": "3"}
}

@BMitch - https://stackoverflow.com/a/42510314/3756843

As adviced, you should do this only until you configure the Docker daemon as your needs.

Answer 2

You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box

https://docs.docker.com/config/containers/logging/syslog/

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://localhost:514"
  }
}