1 Introduction
During our last project, we had big problems with the performance of our website on both the integration and to production systems. The website was rather slow and consumed a lot of memory. So we digged into this to find out what’s happening. As a result, this post gives an introduction to dealing with performance and memory issues, and it provides links to sites with further information.
Performance issues can be classified like this. Hereby, the magnitude of the problem increases from one to three.
1. Slow performance – application not responsive
2. Hangs
3. Crashes
We can find ourselves in two basic situations when experiencing slow performance or hangs
· Low CPU àwaiting for external resources and or locks
· High CPU àBusy Server, Infinite Loops, long time in GC
A Crash is recycling of the applications app domain. A app domain recycle is a normal process of IIS and ASP.NET which usually happens in the following situations (amongst others):
· IIS recycles app pools by default after a period of time
· When changes to web.config or directory names are made.
· When the number of compilations (ASPX, ASCX, or ASAX) exeeds the configured number
Beside those regular recycling processes, the app domain can be recycled unwanted e.g. by the follwing things:
· StackOverflow Exception
· OutOfMemory Exception
· Fatal Excecution Engine Exception (Assert in CLR)
o Managed Heap Corruption e.g.
· When unhandled Exceptions in non-request threads, COM-Component Calls, Destructors /Finalizers
A StackOverflow exception is generally caused by bad recursions. It is quite hard to write non-recursive code that causes the stack to overflow. The reason for an OutOfMemory exception is not so obvious. It can be caused by insufficient memory, or by memory fragmentation. Regardless of the real cause of the OutOfMemory exception, the occurrence of this exception means that your worker process is unstable, and it should not continue it’s work.
To dive deeper into understanding memory issues, here is a little recap of the .NET memory organisation. There a two memory areas. The stack and the heap. The stack is the place where .NET stores value types (char, int, long, byte, enum) and structures. It is organized via Last In – First Out principle (LIFO), and it uses scoping to identify blocks of local variables. So the stack is self-cleaning and doesn’t use garbage collection.
The heap on the other hand is the place where reference types are stored. For example strings as a very simple object, where each char takes up to 2 bytes on the heap. The pointers – or in .NET terms references – to the objects on the heap are stored on the stack as they are simple types. (They only contain the memory address.) Every object on heap is immutable according to memory allocation. This is also the reason for using StringBuilder for string manipulation, rather then concatenating strings with +=. Every concatenation creates a new object on the heap. The heap has an extra area for objects that are bigger than 85000 bytes. It is called the large object heap (LOH).
The heap is frequently cleaned and compressed by the .NET garbage collector (GC). It takes care about freeing heap memory that an application has allocated, but no longer uses it. The clean-up takes place by categorizing objects into three generation Gen 0, Gen 1 and Gen 2. At the beginning every object is in generation Gen 0. With every GC run that an object survives, it moves to the next generation but at max. to Gen 2. So the generation of an object tells us something about it’s lifetime. GC runs are are increasingly resource critical from 0 to 2, because a run on e.g. Gen 1 also includes all objects on Gen 0. GC runs for Gen 2 are runs on the complete .NET heap, and they even suspend all other threads in the worker process and therefore slow down the application.
The compression step of GC is not applied to the LOH, because it would take too much CPU. So the LOH gets fragmented over time. When you consider that the .NET runtime allocates space on the heap in chunks of several mega bytes, you can surely image that you can get an OutOfMemory Exception alltough there is still enough memory left. – Just because there is no big enough contiguous block.
2 Meassuring Performance and Memory
Having in mind those basic facts about the .NET memory organization, lets take a look at available performance counters that help us in finding out what goes on on the server. The following table provides a list of the most important counters and their significance.
To get a picture about the performance of you application, you need to be able to set the application under load, and you have to monitor to following criteria.
- Throughput. This includes the number of requests executed per second and throughput related bottlenecks, such as the number of requests waiting to be executed and the number of requests being rejected.
- Cost of throughput. This includes the cost of processor, memory, disk I/O, and network utilization.
- Queues. This includes the queue levels for the worker process and for each virtual directory hosting a .NET Web application.
- Response time and latency. The response time is measured at the client as the amount of time between the initial request and the response to the client (first byte or last byte). Latency generally includes server execution time and the time taken for the request and response to be sent across the network.
- Cache utilization. This includes the ratio of cache hits to cache misses. It needs to be seen in larger context because the virtual memory utilization may affect the cache performance.
- Errors and exceptions. This includes numbers of errors and exceptions generated.
- Sessions. You need to be able to determine the optimum value for session timeout and the cost of storing session data locally versus remotely. You also need to determine the session size for a single user.
- Loading. This includes the number of assemblies and application domains loaded, and the amount of committed virtual memory consumed by the application.
- View state size. This includes the amount of view state per page.
- Page size. This includes the size of individual pages.
- Page cost. This includes the processing effort required to serve pages.
- Worker process restarts. This includes the number of times the ASP.NET worker process recycles.
For monitoring those criteria, you can use performance counters on the server.
2.1 Interessting Performance Counters
Performance counters are a good way to estimate your current implementation. Often, it is not easy to judge the impact of a code change just from browsing the pages. You should set up some of the following performance counter to get hard numbers.
|
Counter Name
|
Description
|
Significance/Treshold
|
|
Process\Private Bytes
|
This counter indicates the current number of bytes allocated to this process that cannot be shared with other processes.
|
Depends on your application and on settings in the Machine config.
ASP.NET default is min(60 percent available physical RAM, 800 MB)
|
|
.NET CLR Memory\# Gen X Collections
|
This counter indicates the number of times the generation X objects are garbage-collected from the start of the application. (X ={0,1,2})
|
X : X+1 = 10 : 1
|
|
.NET CLR Memory\% Time in GC
|
the percentage of elapsed time spent performing a garbage collection since the last garbage collection cycle.
|
Should average about 5 percent for most applications when the CPU is 70 percent busy, with occasional peaks
|
|
.NET CLR Memory\# Bytes in all Heaps
|
Sum of Gen 0 Heap Size, Gen 1 Heap Size, Gen 2 Heap Size and Large Object Heap Size. Is always smaller then Process\Private Bytes
|
Server-Dependant. Should reach a max value after time.
|
|
.NET CLR Memory\Large Object Heap Size
|
Place for objects that are > 85 Kb. LO-Heap can’t be compacted after GC. So it gets fragmented.
|
|
|
ASP.NET\Requests Rejected
|
The total number of requests not executed because of insufficient server resources to process them. This counter represents the number of requests that return a 503 HTTP status code, indicating that the server is too busy.
|
|
|
ASP.NET\Requests WaitTime
|
The number of milliseconds that the most recent request waited in the queue for processing.
|
|
|
WebService\ISAPI Extension Requests/Sec
|
The rates at which the server is processing ISAPI application requests. If this value decreases due to increasing load, you might need to redesign the application.
|
|
|
ASP.NET\Requests Queued
|
The number of requests waiting for service from the queue. When this number starts to increment linearly with increased client load, the Web server computer has reached the limit of concurrent requests that it can process. The default maximum for this counter is 5,000. You can change this setting in the Machine.config file.
|
|
|
ASP.NET\Requests in Application Queue
|
The number of requests in the application request queue.
|
Configured in appRequestQueueLimit.
|
|
ASP.NET\Applications\ Requests Executing
|
This counter is incremented when the HttpRuntime begins to process the request and is decremented after the HttpRuntime finishes the request.
|
|
|
ASP.NET\Worker Process Restart
|
The number of aspnet_wp process restarts.
|
1
|
Table 1ASP.NET and IIS 6.0 Performance Counters

- ASP.NET Performance Counter (Figure from Microsoft)
2.2 Improving application performance
Now that we know how to judge the performance of our application with performance counters, we should investigate the real issue. Herefore, you can use automatic tools, memory dump analysis or just a simple code review.
2.2.1 By Code Review
A code review is probably the most effective way to identify performance issues. Beside some commonly known issues, that are listed here, also logical mistakes can be found that way.
- Use Output buffering where possible
- Use StringBuilder for string concatenation
- Don’t use .ToLower() for case-insensitive string comparison. Using String.Compare (string strA, string strB, bool ignoreCase); avoids temp memory allocations
- Don’t use exception handling to control the flow of your normal application logic
- Don’t use a try-catch block inside a performance critical loop
- Use strongly typed collections over generics (Hashtable, Dictionary, Arraylist) for storing simple types. This avoids unnecessary boxing.
- Initialize collections to an approximate final size
- Prefer StringDictionary instead of Hashtable for storing strings
- Call Dispose on IDisposable objects in finally blocks to free ressources
- Avoid calling Page.DataBind and bind each control individually to optimize your data binding. Calling Page.DataBind recursively calls DataBind on each control on the page.
- Use Repeater Control rather than DataList, DataGrid and DataView controls. It has a smaller HTML output
- DataBinder.Eval uses reflection, which affects performance. In most cases DataBinder.Eval is called many times from within a page, so implementing alternative methods provides a good opportunity to improve performance.
eg: Bad <%# DataBinder.Eval(Container.DataItem,”field1″) %>
Good <%# (string)DataBinder.Eval(Container.DataItem,”field1″) %>
- Mind DataSet Serialization: Avoid multiple versions of the data. Only return relevant data in the DataSet and call AcceptChanges before serializing a DataSet.
- If you need to search a DataSet using a primary key, create the primary key on the DataTable. This creates an index that the Rows.Find method can use to quickly find the required records. Avoid using DataTable.Select, which does not use indices
- Do not call GC.Collect() yourself.
- If your code uses recursion, consider using a loop instead. A loop is preferable in some scenarios because each recursive call builds a new stack frame for the call. This results in consumption of memory, which can be expensive depending upon the number of recursions.
- Avoid foreach loops when performance is critical. Using foreach can result in extra overhead because of the way enumeration is implemented in .NET Framework collections.
- Do not use exceptions as a tool to exit one or more loops.
- Use Output Caching of pages or user controls whenever possible
- Disable SessionState and Viewstate for pages that don’t need it. If a page only needs to read from session, consider using <%@ Page EnableSessionState=”ReadOnly” . . .%>
A more complete checklist as well as a lot of other interesting information about .NET code performance can be found at (J.D. Meier, 2007)
2.2.2 With Tools
Sometimes resolving the issue just a code review is not possible. In our case, we could identify some issues in our code, but the biggest issue was found at another place. It turned out to be a memory leak in the used ISAPI Rewrite engine. So how do you find such issues?
Here is a list of tools which can help you with analysing performance issues of web applications.
· Debug Diagnostic Tool (DebugDiag)
· Debugging Tools for Windows (X-copy deployment possible!!)
o WinDbg (a Microsoft debugger)
o ADPlus (Recording of Memory Dumps)
o TinyGet (Command Line Load Tester)
o Son-Of-Strike (SOS.dll) (.net extension for native debugger windbg)
· SysInternals Process Explorer / Task Manager
Debug Diag is probably the easiest tool, to record and analyse performance dumps. We could eventually find our problem with DebugDiag on integration. The program has to be installed, what might not be an easy process on a production landscape. But for those cases, we can use the Debugging Tools for Windows, which can be just copied to the server.
Here is a list of important commands for doing the analysis with the debugging tools for windows.
|
Program
|
Commands
|
Description
|
|
Command Line
|
adplus -hang -p [PID] –quiet
|
Starts recording a memory dump. Tracking all current activity. Not only hangs are recorded
|
|
Command Line
|
adplus -crash -p [PID] –quiet
|
Sets up a recording session that waits in the background for crashes. When a crash occurs the dump is recorded.
|
|
WinDbg
|
.loadby sos mscorwks
|
Load the Managed Extensions Module in WinDbg
|
|
WinDbg
|
!runaway
|
Lists all threads that used the most CPU time in during the recording session of the memory dump
|
|
WinDbg
|
~* e!clrstack
|
Shows the callstack of all threads in the memory dump
|
|
WinDbg
|
!synchblk
|
Which treads holds a lock
|
Table 2Commands for analysing performance problems
3 Bibliography
Douglass, David. 2006. .NET on my mind. .NET on my mind. [Online] 03 14, 2006. [Cited: 12 06, 2008.] http://geekswithblogs.net/.netonmymind/archive/2006/03/14/72262.aspx.
Ferrandez, Tess. 2008. If broken it is, fix it you should. If broken it is, fix it you should. [Online] 2008. [Cited: 12 06, 2008.] http://blogs.msdn.com/tess/default.aspx.
J.D. Meier, Srinath Vasireddy, Ashish Babbar, John Allen and Alex Mackman. 2004. Improving .NET Application Performance and Scalability . Improving .NET Application Performance and Scalability . [Online] 2004. [Cited: 12 06, 2008.] http://msdn.microsoft.com/en-us/library/ms998549.aspx.
J.D. Meier, Srinath Vasireddy, Ashish Babbar, Rico Mariani, and Alex Mackman. 2007. Web Application Performance Design Inspection Questions. Web Application Performance Design Inspection Questions. [Online] 12 18, 2007. [Cited: 12 14, 2008.] http://www.guidanceshare.com/wiki/Web_Application_Performance_Design_Inspection_Questions.
2005. Monitoring Applications That Use the IIS 6.0 WWW Service. Monitoring Applications That Use the IIS 6.0 WWW Service. [Online] Microsoft, 08 22, 2005. [Cited: 12 12, 2008.] http://technet.microsoft.com/en-us/library/cc775979.aspx.
Seguin, Karl. 2008. Foundations of Programming – Building better software. www.codebetter.com, Canada/Ontario : s.n., 07 17, 2008.