0

I have podman-based containerized systemd services running on Ubuntu 22.04. Every line of output is sent to journald twice: once by conmon and once by podman.

As an example, here's my /etc/systemd/system/mongodb.service file:

[Unit]
Description=MongoDB container

[Service]
Restart=always
ExecStart=/usr/bin/podman start -a mongodb
ExecStop=/usr/bin/podman stop -t 10 mongodb

[Install]
WantedBy=multi-user.target

I have a similar file for a container running my own Node.js application, and exactly the same thing happens, so it's nothing to do with the applications inside the containers.

I don't see non-podman-based systemd services doing this at all.

I know the duplication comes from podman and conmon due to lines like this:

Dec 11 08:59:24 52.1.177.253.worker.holon-mirror.apos.dev conmon[4959]: {"t":...}
Dec 11 08:59:24 52.1.177.253.worker.holon-mirror.apos.dev podman[4768]: {"t":...}

Details of the output omitted, but yes 100% identical each time.

I could use grep to filter out either conmon or podman, however I also need to be able to tell the systemd units (services) apart. So in practice I prefer to use:

journalctl --output with-unit

Which gives me:

Mon 2023-12-11 08:59:24 UTC 52.1.177.253.worker.holon-mirror.apos.dev mongodb.service[4959]: {"t":...}
Mon 2023-12-11 08:59:24 UTC 52.1.177.253.worker.holon-mirror.apos.dev mongodb.service[4768]: {"t":...}

Now I can tell which lines come from the mongodb service, and not the other one... but, I can't tell podman from conmon.

I've determined that I can use --output json to get access to everything at once. From there, I could manually look for "SYSLOG_IDENTIFIER":"podman" and "_SYSTEMD_UNIT":"mongodb.service", which is enough information for me to solve this problem the hard way.

But before I commit to that, I'd like to know if there's a way to just get podman to only log in one of these two ways in the first place.

3
  • What about journalctl -u mongodb.service?
    – AlexD
    Feb 20 at 11:33
  • journalctl -u mongodb.service restricts to that service, but still shows everything twice, under the two different syslog identifiers. Feb 27 at 15:18
  • I wound up using the JSON format and narrowing that down to a less verbose JSON format for long term storage and analysis. I do the filtering-out of podman (had to pick one) at that stage. I also wound up writing tools that can output that directly or reconstruct classic syslog format. Feb 27 at 15:20

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.