AWS CloudWatch Logs Publisher plugin hangs on job completion

Question

We use the AWS CloudWatch Logs Publisher plugin on our Jenkins and most of the time the logs are sent within 2-5 seconds of the request being sent out. However, sometimes the step will hang for up to 5-10 minutes, with no errors in the console or in Jenkins' log. The job hangs with the following status message;

AWS CloudWatch Logs Publisher is waiting for a checkpoint on [...]

It always completes successfully, even with the delay.

Has anyone seen this before? Is there anything I can do to prevent the step hanging?

Answer 1

Googling returns only this Q&A:

It looks like that it is not a reported issue. Also combined with your remark, i.e.:

It always completes successfully, even with the delay.

I wonder whether it is really an issue as I have seen a lot of logs and also weird messages while everything worked. What I do in such situation is talk with the developer of the code. In this case I would create an issue in the Jenkins bug tracker and discuss whether this is an issue and whether the developers should change the code that triggers this message.

Answer 2

This is conjecture, but after watching this for almost six months here's my theory;

I think the jobs have to post their logs in sequential order. If you have multiple jobs, or even multiple runs of the same job that all push logs it will wait for them to complete in the order they were started. This can be troublesome if you have one hour-long build that started before a 10-minute job. The shorter one started second even if it finishes first, so will wait for the hour long one to finish before finishing itself.

There doesn't seem to be any fix for it, but we've learned to just ignore it and count the jobs as done if they're hanging at the logs phase.