10

I have a few VMs on Windows Azure that run our ecommerce website, and lately we started using Telegraf, InfluxDb and Grafana to keep an eye on these machines. After a couple weeks of gathering data, I have noticed a weird pattern related to the Memory Available metric:

Everyday almost always at the same period of the day, I have noticed that there is an abrupt amount of memory being freed up which, due to my very very very limited DevOp skills, I cannot figure out what is causing this.

Here's a chart that shows this pattern:

Odd pattern

My question is: What could lead to something like this? I feel tempted to suspect that a Memory Leak is to blame but...Free memory never drops below 70% and only happens in two of the VMs with the most traffic!

Should I be concerned when I see something like this?

P.S.: I have stared gathering metrics for Private e Virtual bytes for each of the windows services we have running and for the w3wp process...although I have read that this metrics are not very reliable to find out if you have a memory leak, but at least I'll try to get some sort of trend and see if it correlates with the pattern shown above.

Improve this question
3
  • 2
    You're seeing an usual garbage collector or cache cleaning IMHO. In wich language is your website done ? (this could be your app, your websever, and even the system doing some cleanup)
    – Tensibai
    Commented May 9, 2017 at 12:33
  • That was something I was suspecting too...It's done in ASP.NET MVC 4, so the garbage collecting theory makes some sense. Also, on a side note, the metrics I've been gathering on the w3wp process and windows service look absolutely normal.
    – António Sérgio Simões
    Commented May 9, 2017 at 13:18
  • I'm know nearly nothing in ASP, but I assume there's way to graph the memory consumption and garbage collection as in java, this should help ensuring this is the root cause.
    – Tensibai
    Commented May 9, 2017 at 13:30

2 Answers 2

Reset to default
7

I've seen this same "sawtooth" pattern in other systems, in particular a Java-based data tool. Based on your description, I think you're looking at .NET garbage collection (assuming this is a .NET app.) Java and .NET are both memory-managed languages and frameworks that use garbage collection.

A memory leak is typically found in frameworks that lack memory management, or in a program on a memory-managed framework that is overriding or confusing the garbage collector.

The fact that these are your highest-traffic servers makes sense. You're seeing the .NET framework allocate memory as needed, then the garbage collector kicks in on a regular cycle and reclaims unused memory using the garbage collection algorithms. Unless you're tracking specific performance issues I don't think this memory usage pattern is a problem.

Improve this answer
1
  • It is indeed a .NET app and from my research in the past couple days it makes a lot of sense what you and @Tensibai wrote.
    – António Sérgio Simões
    Commented May 10, 2017 at 11:06
7

I think have found out why this graph looks like this.

I am also gathering metrics for the ASP.NET Applications / Errors Total performance counter, and I have noticed that exactly at the same time that a memory available surge happens, the Total Errors metric resets to 0.

According to msdn this counter resets to 0 everytime an application restart/shutdown happens.

This leads me to believe that the cause for this Memory Available sawtooth pattern is due to Application restarts happening.

Here's how my charts look like:

Errors Total ASP.NET Application Memory Surge

UPDATE

This happens because the Private Bytes of the W3WP process hit the limit for a recycle (We have a private bytes limit configured in the app pool). And looking closely to the Private Bytes chart, we can see something abnormal is happening because Memory usage leaps from 650MB to 3.2GB and a few hours later leaps from 3.6GB to 16.6GB! This is when the recycle happens.

Improve this answer
1
  • 2
    This is much more plausible explanation. Sudden memory freeing happens almost exclusively with process restarts. Mechanisms to free memory of running processes are never this sharp and rarely actually free the memory rather than freeing some space on preallocated heap.
    – Jiri Klouda
    Commented May 15, 2017 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.