DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.
Sign up to join this community
Projects like Thanos, which are based on prometheus storage formats, seem to say that it is impossible to back-fill data in a prometheus based system.
Can anyone explain why and if there are any possibilities for changing this? Lack of ability to backfill would make it difficult to switch metrics systems without impact (when previous systems can already retain multi years' data).
Answering my own question (after a few days of research).
I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose:
PromQueen made possible to record prometheus metrics offline. PromQueen can, > therefore, backfill the recorded data inside a native prometheus database.
PromQueen is composed of two primary tools:
- promrec tapes the metrics in a specified output file.
- promplay backfills the prometheus database from scratch.
It doesn't have a lot of contributors but it comes from another POC repo and it was updated in the last 6 months, so I suspect it'll work fine.
In any case, it proves that you can plausibly back-fill prometheus, so it's not impossible.
You can also find this in the prometheus road-map, so it's clearly slated to be built-in eventually.
https://prometheus.io/docs/introduction/roadmap/
Backfill time series Backfilling will permit bulk loads of data in the past. This will allow for retroactive rule evaluations, and transferring old data from other monitoring systems.
While the backfill is still in review for Prometheus, you can use https://github.com/sepich/thanos-kit/ import command to import custom data in prometheus text format to Thanos.
