Health Monitoring and Instrumentation - Prescriptive Guidance
Categories
As ASP.NET developers we are all familiar with the concept of writing trace statements to the Trace log so that we can use that information later at runtime to assist us with diagnosing problems in our applications. For example, writing exception messages to the Trace log can help understand why things are not working as we expect - especially when our code is layered and finding the root cause of small inconsistencies may not be easy to detect by simply viewing the output of a web page.
As for prescriptive guidance about where to place tracing statements within an application’s code, the obvious strategic targets that are most likely to cause problems at runtime should be targeted so that we can work out how to diagnose those errors when they occur. At a minimum I would suggest instrumenting the following code:
- Exception handling blocks
- Calls to external systems which may be expensive so that we can monitor how long those operations are taking
- Calls to complex business logic operations to help detect erroneous logic
Writing trace statements is useful for diagnosing problems that already exist in our applications but wouldn't it be nice to be alerted to potential problems before they actually occur so that we can do something to remedy them before they start causing our application to fail. In this way we can be proactive to potential problems rather than always being reactive. The activity of and detecting certain types of issues before they become problems is known as “Health Monitoring” and involves keeping track of performance counters in our code and viewing these vital statistics periodically to keep a look-out for things that might be going wrong.
ASP.NET 2.0 contains an event based health monitoring system known simply as “Health Monitoring” that we can tap into to keep track of the health of our applications. We can use the Health Monitoring system to monitor the health of our applications and send notifications as thresholds for certain types of events are reached. For example, periodically the ASP.NET process will be forced to recycle which will cause an application restart to occur. Most of the time these restarts are to be expected, such as when an administrator makes a change to the web configuration file an application restart will occur. However there are other times when application restarts indicate that an application is experiencing extreme difficulty and we would very much be interested in receiving notifications about these events to let us know that everything is not fine. As an example, large numbers of application restarts are a typical symptom that a denial-of-service attack is taking place. Using the Health Monitoring service we could configure our application to listen for application restarts and configure a rule that would cause a notification to be sent after a certain threshold is breached within a given time.
The Health Monitoring system comes with a large number of standard events that are raised during the lifecycle of the application and also some standard providers (Sinks) that are used to log the event notifications. The standard provders include an Email provider for sending email, a SqlWebEventProvider for logging notification information into a SQL Server database, and a WMIEventProvider that maps ASP.NET health-monitoring events to Windows Management Instrumentation (WMI) events.
The Health Monitoring system is highly configurable and so we can easily create our own events and providers to record information about certain types of events and have them forwarded to a store of our choosing. For example, we might want to send an SMS to an application administrator whenever the time to make web service calls to external sites exceeds a certain time threshold - such as 3 seconds.
Here are some excellent articles that will help you get started with implementing Application Health Monitoring in your ASP.NET applications:
PAG : How To instrument Code
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/paght000016.asp
PAG: How to use Health Monitioring
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/PAGHT000011.asp
Link to an excellent article which demonstrates how to create custom Events and Providers
http://msmvps.com/blogs/gbvb/archive/2004/09/15/13578.aspx