Memory leak in large ASP.NET application
August 21st, 2011
On our main application at work (a car rental booking system) we would every so often see an OutOfMemoryException thrown. At the time it was on a single .NET 2 server and each exception we looked at provided us with different stack trace! Initally we set the set the application pool recycle time to 4 hours as a temporary fix. However as the application is used for many clients over time more and more clients were consuming more resources and once again we started seeing more OutOfMemoryExceptions.
To help us manage the situation we increased the application pool recycle time to 2 hours, not ideal, but it would least prevent the application from crashing. We were lucky that we keep our session data out of the in process memory and instead in the database so users were not really affected directly.
We were already aware that the memory leak was a gradual leak as it happened at random times with random stack traces. Trying to work out how often the OutOfMemoryException was thrown was the first step to the problem, every so often I logged on to the each server and took note of the memory usage of the worker process, not the most sophisticated way but gave me a good indication of how fast the leak was growing. See below for usage.
Trying to recreate on my local machine was not successful, including trying many of the available tools on web such as .NET memory profiler and ANTS memory profiler. These did pick up a few leaks, these were fixed but the problem remained. Anyway, after a bit of googling I came across the holy grail of memory leak information, Tess Ferrandez’s blog!, now at first glance of some of the tools she uses were a bit daunting but I taking the time to play about with them, in particular Windbg and it was well worth it.
Finally on to something?
Watching a video from Tess Ferrandez basically showed I needed to get a memory dump of the server, load into Windbg and analyse using my application PDB files.
Taking the largest memory dump I could and using Windbg and SOS.dll … I got nothing, it wouldn’t let me open it, this was due to my 32bit development machine not being able to analyse a x64 memory dump. Your machine your debugging your memory dump must match your production architecture.
With the video from Tess Ferrandez and playing around Windbg commands I was able to analyse what objects should not be hanging about in memory. Making small changes to the code and checking things were being disposed correctly etc I was able to see over time the memory was going down, but still was way too high for a normal ASP.NET application.
XslTransformComplied … why you eating so much memory!
Continuing to look through the memory dumps I noticed some large strings containing the emails we send for each bookings. Now, I was always under the impression the strings were one of the largest .NET objects in memory and were used by so many parts of the .NET framework I didn’t initially focus my attention here. However, I noticed some large strings and spotted that these were in fact booking emails which contained a date time and were produced many hours ago, I thought these would have been cleaned up in memory a long time ago.
Looking deeper into how these email were created we had a booking object that was serialized and an XSLT applied to it, with the power of google I found that one of our XslTransformComplied objects was creating a temporary assembly each time but was not being cleared up. This meant the temporary assemblies remained within the application pool until the application pool recycles or in our case an OutOfMemoryExcepion was thrown.
Simply passing ‘false’ into the XslCompliedTransform constructor indicated not to use debug mode and thus stopped temporary assemblies being created. Amazing how a single boolean could cause so much pain!
Just because .NET has a garbage collector does not mean your free of memory leaks. Locating and solving a memory leak can be time consuming process but with the right tools and approach it can be a worth while process. I think the most frustrating part of looking for a memory leak is there is not a single right way of doing it… it always depends on your setup and application. As mentioned above I highly recommend Tess Ferrandez blog as a starting point and the tools she uses, even if they look scary, they work and are well worth investing your time.