What is a good storage for storing performance results and visualizing in grafana

Question

Intro

I am a tester and need to analyze the results of automated tests (response times and etc.)

Current situation

1) I run performance tests using locust.io
2) During test execution, response times of successful results are pushed to Graphite database
3) I visualize the results in Grafana

Problem

I tried to calculate the average using Grafana to see if it matches the results generated by Locust. It doesn't.

After a while, i tried to figure out what could be a problem. Graphite has data retention. As a result, the smallest time period for a data point is 1 second.

My situation is that I could have several data points with a time difference of several milliseconds.

I thought I could use something like Prometheus as a substitute for graphite, but it doesn't seem to have an import of data. A solution would be installing Graphite Exporter, such that i can imitate "pushing" data to db. But I think installing this extension and setting everything up is an overkill. There should be a much simpler solution.

Question

Could someone suggest a solution from personal experience? I need a db/storage to which I can write in a format query_name responce_time timestamp, and then use this storage as a data source for visualizing a report in Grafana.

Thanks in advance!

Answer 1

Sadly nobody answered, so I will provide feedback about how i did it so far.

Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last minute, the first 99 would be dropped, the 100th one would be stored. This fact was found by me somewhere on some forum. I do not claim I have setup/configured/used graphite correctly, yet the result was several days of debugging a bug which appeared an actual feature, lots of stress and questioning if my head and hands grow out of my shoulders.

Next option was to use InfluxDB. Sadly didn't work for the following reason:
- If running InfluxDB in a Linux docker container, I was not able to bind mount it to a specific folder on host's OS outside of docker installation folder. A solution would be to try and use docker named volume. Maybe that would have worked, but I was short on time and patience.

The final solution was to use simply a Postgres DB. Created a timetable with all the metrics and a timestamp field. As a result, Locust writes the final session statistics into a CSV file. CSV file is parsed by a python script and parsed data is inserted into the table. Grafana then connects to the DB and quite nicely visualizes the results using a simple query.

Next possible question would be:

how to store the results of every single request to gain better precision for the results of my performance tests?

My personal suggestion would be to use influxDB with a consistent storage saved in a named volume if you are using Windows host. There the smallest time period can go down to milliseconds and data retrieval was very easily setup by manually writing a simple select query. Yet, please do not take my word for it and try this out, because i haven't done this, so my thoughts are nothing but a theory.

Happy performance testing.