We are using Confluent Kafka cluster on production K8s. It looks like our consumers are stuck in a loop as offset is static and lag keeps going up. When I restart zookeper/brokers (persistent volume) the offset moves by a small amount and then it is stuck again. I have checked my consumers and only 1 of them has assigned something, that is processed and all assignments finish after. What happens on consumer side is that sometimes we have an error, we tried ignoring the message but Kafka is still affected. What can be done outside of the default config. We are not sure why message are stopped being consumed or it looks like there is some loop state leading to attempting to process same message over and over agian.
Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.