Snapshot or Saving Prometheus Data for Performance testing and comparison

Question

I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, Memory, File I/O, etc, for a test run, and save them for comparison against another test run. I'll be running multiple tests at different times with different configurations but the resource metrics will be standardized.

For Example, for two 5 minute tests, Comparing CPU:

• Test 1 - 0:00-5:00
  • CPU Graph
• Test 2 - 0:00-5:00
  • CPU Graph

And then Comparing Memory:

• Test 1 - 0:00-5:00
  • Mem Usage Graph
• Test 2 - 0:00-5:00
  • Mem Usage Graph

Is there a way to save or record these metrics for A/B comparison using Prometheus and Grafana? Or would I be forcing these tools too hard to do something that they really aren't designed to do? If so, is there a better way/tool to capture and compare the data I'm looking for this type of performance testing?

Answer 1

Grafana has option for directly pointing certain timeframe. https://grafana.com/docs/grafana/latest/dashboards/time-range-controls/

Save start and end time of your test OR test use fixed test duration added to startime to get endtime(for similar size of images) and compare exactly same grafana view from different time.

E.g.

Test1 is executed 1.12.2022 12:45:02 - 1.12.2022 12:47:02

Test2 is executed 5.12.2022 00:25:02 - 5.12.2022 00:27:02

Then you get two url ending for exactly same grafana view. It makes possible comparison, but you need to have two browser instances open in parallel.