Recycling an Application Pool with C# (Part 2) by Matt Wrock

The other day I shared how to use the DirectoryServices namespace to restart an app pool via C# code. The code I used had two key flaws:

  1. It used a Thread.Sleep(2000) to wait for the old app pool to be destroyed.
  2. The use of DirectoryServices required the enabling of the windows feature: IIS Metabase and IIS 6 configuration compatibility.

Also just to recap why you would even want to do this: My use of this code is for writing integration tests of web app functionality. It allows me to test scenarios where I want to ensure a certain outcome after the application restarts. It also helps to isolate test state from one test to another.

Anyhoo, a coworker of mine, @mmanela (Matt Manela) mentioned hosting a powershell script instead of the DirectoryServices implementation. As we discussed it further, we assumed that the PowerShell WebAdministrationiModule was probably using some other API and that it would be interesting to discover what that was and see if you could use that. Well after spending some quality time with reflector and the WebAdministrationModule DLLs, I was not able to tell what that API was. However, I did discover another API that appeared to be a better alternative to DirectoryServices.

The API can be found in %WinDir%\system32\Inetsrv\Microsoft.Web.Administration.dll. See this post for a good overview. Here is my new helper method:

public static void RecyclePool(){    using (var manager = new ServerManager())    {        var pool = manager.ApplicationPools["RequestReduce"];        Process process = null;        if(pool.WorkerProcesses.Count > 0)            process = Process.GetProcessById(pool.WorkerProcesses[0].ProcessId);        pool.Recycle();        if(process != null)        {            while (!process.HasExited)                Thread.Sleep(0);            process.Dispose();        }    }}

So in addition to using a different API, I’m also no longer using the hacky Thread.Sleep(2000) to wait for the app pool to die. Instead, I use this API to get the Process ID of the about to be recycled app pool. I then wait for the pool to exit. I have tested this and it works perfectly. So now my tests move on as soon as the app pool is completely destroyed. I don’t have to wait any extra time in case this happens more quickly than two seconds and I don’t risk a failed test if two seconds is not long enough. In case you are wondering why it is so important to wait for the old app pool’s worker process to terminate before proceeding, it is because I may have cleanup code that deletes files and that code will likely fail if the old worker process had a lock on the file.

Recycling an Application Pool with C# by Matt Wrock

I have been developing a CSS background image spriting, merging and minification application where I often force an app pool recycle in my integration tests. This is handy because it essentially allows me to reset the state of a web application in my test and it is a bit more light weight than performing a full IIS reset. I can specifically test certain scenarios where I want to make sure that some data or state can be persisted if the application is shut down. Its also a good way to isolate a test case from the effects of other integration tests. Things like output caching or any staticly held application memory resources are flushed and the next test has a clean slate.

To perform an app pool refresh, I use the following helper method in a class called IntegrationFactHelper:

public static void RecyclePool(){  using (var pool =     new DirectoryEntry("IIS://localhost/W3SVC/AppPools/RequestReduce"))  {    pool.Invoke("Recycle", null);  }  Thread.Sleep(2000);}

Make sure you have a using statement pulling in the System.DirectoryServices namespace. The path above (IIS://localhost/W3SVC/AppPools/RequestReduce) would be the path to your IIS application pool. Note this is the IIS application and not the IIS site.

I'm not too proud of the Thread.Sleep(2000) here. I just have not invested time in a better way to actually wait for the pool to restart. The call to Invoke does not block and wait for the restart to complete. I briefly played with polling the application state but still found that after the application claimed to be on (or whatever the state name is) that the app was unresponsive. I tend to think that I have not investigated that far enough and would be delighted if someone commented with a way to more elegantly accomplish this. Having said that, I have found that on my system, 2 seconds is the sweet spot.

UPDATE: See this post for an improved implementation that avoids this Thread.Sleep kludge and also gets around the dependency discussed below.

One cautionary and rather annoying note on using this DirectoryServices call on IIS applications. You may encounter this not so delightful error:

System.Runtime.InteropServices.COMException : Unknown error (0x80005000)
 at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
 at System.DirectoryServices.DirectoryEntry.Bind()
 at System.DirectoryServices.DirectoryEntry.get_NativeObject()
 at System.DirectoryServices.DirectoryEntry.Invoke(String methodName, Object[] args)

Isn't that nice? I love unknown errors...almost as much as unspecified ones.

There may be other causes, but I have found that one reason this error may occur is if you have not enabled the Windows feature: IIS Metabase and IIS 6 configuration compatibility (see image below). I am using IIS 7, but this feature is required to use the above code.

Three pitfalls to avoid when writing a response filter by Matt Wrock

I was looking at a response filter that someone else had written yesterday and I noticed a few things it was doing that ideally you want to avoid in a response filter. This is a particularly fresh topic for me since I am nearing the end of V1 development on a response filter that will automatically find a response's css, merge them, find their background images, sprite the ones it can then create a new and minified css with these sprites. I'll be blogging much more on that next month.

Now, in order to write a good filter that will work with any site and be performant is not particularly easy. If your filter is limited to a small or smaller sites, this advise may be considered to lie in the category of preoptimization. But real quick...before I elaborate on these pitfalls...

What is a response filter?

A response filter is simply a class that derives from System.IO.Stream. This class is attached to an HttpResponse's Filter property like so:

Response.Filter = new MyFilter(HttpContext.Current.Response.Filter, 
    HttpContext.Current.Response.ContentEncoding);

As the underlying response outputs to its OutputStream, this output is sent to the filter which has the opportunity to examine and manipulate the response before it gets to the browser. The filter does this by overriding Stream's Write method:

void Wite(byte[] buffer, int offset, int count);

When the filter is ready to send its transformed response to the browser or just forward the buffer on unchanged, it then calls the underlying stream's write method. So your filter might have code like this:

        public ResponseFilter(Stream baseStream, Encoding encoding)
        {
            this.encoding = encoding;
            BaseStream = baseStream;
        }

	protected Stream BaseStream { get; private set; }

        public override void Write(byte[] buffer, int offset, int count)
        {
		var header = encoding.GetBytes("I am wrapping");
		var footer = encoding.GetBytes("your response");
		BaseStream.Write(header, 0, header.Length);
		BaseStream.Write(buffer, offset, count);
		BaseStream.Write(footer, 0, footer.Length);
         }

This is a common implementation used for adding compression to a site or ensuring that a site's content is always wrapped in a common header and footer.

So with that background, here are some things to try and avoid in a solid filter: 

Assuming UTF-8

This is easy to overlook and honestly it will work most of the time, but if you think that your filter will ever be dropped on a Japanese website, or a website that is intended to be localized to a double byte unicode locale you might be disapointed. Very disapointed. Avoid doing something like this:

BaseStream.Write(encoding.GetBytes("I am wrapping"), 0, 
    "I am wrapping".Length);

In a Japanese locale, the underlying encoding will be unicode and the length of the byte array will be twice the size of  "I am wrapping".Length which is likely UTF-8. So the users see just half the stream. But thats ok, the first half was way better.

Copying the buffer to a string

 You might be tempted to do something like this:

public override void Write(byte[] buffer, int offset, int count)
{
    var output = encoding.GetString(buffer, offset, count)
    var newOut = encoding.GetBytes("header" + output + "footer");
    BaseStream.Write(newOut, 0, newOut.Length);
}

You have now managed to double the memory footprint of the original response by copying it to a new variable. This can be a sensitive issue with filters since they often process almost ALL output in a site. Unfortunately, if you need to do alot of text searching and replacing on the original byte array and you want to be efficient, this can be difficult and tedious code to write, read and test. I intend to devote a future post to this topic exclisively.

Ignoring the offset and count parameters

You might think that using the offset and count parameters in your Write override is not necessary. After all, you are confident that your transformations can go to the browser as is because you don't have any code that would need to do further processing on the buffer. Well maybe you don't but someone else might. You may have no control over the fact that someday another HttpModule will be added to the site that registers another filters. Response filtering fully supports the ability to chain several filters together. Someone elses module might have have the code mentioned above in their own class:

Response.Filter = new MyFilter(HttpContext.Current.Response.Filter, 
    HttpContext.Current.Response.ContentEncoding);

So if this is called after your own filter was added to the response, then YOU are HttpContext.Current.Response.Filter. That new filter might do something like:

public override void Write(byte[] buffer, int offset, int count)
{
    int[1] headBoundaryIndexes = FindOpeningHead(buffer, offset, count);
    BaseStream.Write(buffer, 0, headBoundaryIndexes[0]);
    BaseStream.Write(anEvenBetterHead, 0, anEvenBetterHead.Length);
    BaseStream.Write(buffer, headBoundaryIndexes[1], (offset + count) - headBoundaryIndexes[1]);
}

So if your filter is this filter's BaseStream and your Write looks like this:

public override void Write(byte[] buffer, int offset, int count)
{
    var output = Encoding.UTF8.GetString(buffer);
    var newOut = output.Replace("super", "super duper");
    BaseStream.Write(Encoding.UTF8.GetBytes(newOut), 0, newOut.Length);
}

Ouch. Your users are probably looking at something other than what you intended. The upstream filter was trying to replace the head but now they three. After several years in the industry and meticulous experimentation, I have found that 1 is the perfect number of heads in a web page.

Oh and look, this code managed to violate all three admonitions in one blow.

Nuget and Powershell WebAdministration Module Issues by Matt Wrock

I’ve been working on getting an authentication library  packaged using nuget. The library uses an X509 certificate to encrypt and decrypt auth ticket data stored in a cookie. So to make it easy to get the library up and running, I wanted to create an install.ps1 script that would install the x509 cert in the correct location of the cert store. In order for the library to access the certificate, the account under which the hosting web application runs needs to have read permissions on the cert. I need to figure out which account that is and then use winhttpcertcfg.exe to grant that account the appropriate permissions.

I had assumed that this would simply involve loading the WebAdministration powershell module and then accessing its cmdlets to query which website has a physical path matching the one of the project that is referencing  my package. Then find its application pool’s process model to determine which account the application runs under.

When I began testing this script from within the nuget package manager console, I started getting an unexpected error on the first call into any cmdlet of the WebAdministration module:

Get-Website : Retrieving the COM class factory for component with CLSID {688EEEE5-6A7E-422F-B2E1-6AF00DC944A6} failed d
ue to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).
At line:1 char:12
+ GET-WEBSITE <<<<
    + CategoryInfo          : NotSpecified: (:) [Get-Website], COMException
    + FullyQualifiedErrorId : System.Runtime.InteropServices.COMException,Microsoft.IIs.PowerShell.Provider.GetWebsite
   Command

After spending some time researching this, I discovered that nuget uses the 32 bit powershell.exe shell and that the module is looking for com classes not registered in the wow64 registry on a 64 bit system. This proved to be very frustrating and I was wondering if discovering the IIS user account would not be possible in my nuget install.

After more research, I discovered that I could get to what I needed using the .net Microsoft.Web.Administration assembly. While probing this assembly is not quite as friendly and terse as the WebAdministration module, it met my needs perfectly. Here is the powershell script that determines the application pool identity:

 

param($installPath, $toolsPath, $package, $project)

function GetAppPoolAccount([string] $webDirectory){    [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Web.Administration")    $iis = new-object Microsoft.Web.Administration.ServerManager    $account = "NetworkService"    $site = $iis.sites | where-object {$_.state -eq "Started"}|         foreach-object {$_.applications} | where-object {            $_.virtualdirectories -contains (                foreach-object {$_.virtualdirectories} | where-object {                    $_.physicalpath -eq $webDirectory                }            )        }    if($site)    {        $poolName = $site.applicationPoolName        $pool = $iis.applicationpools | where-object{$_.name -eq $site.ApplicationPoolName}        if($pool.processModel.identityType -eq "SpecificUser"){            $account = $pool.processModel.userName        }        elseif($pool.processModel.identityType -eq "ApplicationPoolIdentity"){            $account = "IIS APPPOOL\$poolName"        }        else{            $account = $pool.processModel.identityType        }    }    return $account}

. GetAppPoolAccount -webDirectory (Get-Item $project.FullName).Directory.$toolsPath\InstallAuthCerts $toolsPath "INT" $account

Essentially the Microsoft.Web.Administration assembly exposes the IIS applicationHost.config data in the same format that it exists in xml. My script loads the assembly, instantiates the ServerManager class and iterates its members to get the information I need. I then call into my InstallAuthCerts script with the path of the cert, the environment, and the account name which will install the cert and grant the appropriate permissions to $account.

Now if someone can point out how to use the WebAdministration module from a nuget console, I’d be grateful for that information. I dug around quite a bit and from what I can tell, it can’t be done. At least not on  a 64 bit machine.

Getting TransactionScope to play nice with NHibernate by Matt Wrock

My team is beginning to integrate NHibernate into a fairly large code base that makes frequent use of TransactionScope to ensure that the saving of related entities succeed or fail together. My own NHibernate experience has been primarily on green field projects and on existing projects where the NHibernate work was well isolated from existing plain ADO calls. Furthermore, my use of transactions has typically been from within stored procedures and I have never used any of the transactional functionality exposed by the ADO APIs. So this strange thing called TransactionScope was an entirely new concept for me to understand.

Now before I elaborate on the drama and adventure that accompanied this learning experience and some of the failures that ensued, I’ll quickly jump to the moral of this story for those of you that simply want to know what is the right way to get NHibernate transactions to participate in a transaction defined by TransactionScope. Simply stated: its all in the nesting. Instantiate a TransactionScope on the outside and and begin and commit/rollback an NHibernate transaction on the inside.

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();    using (var transaction = session.BeginTransaction())    {        session.SaveOrUpdate(entity);        transaction.Commit();    }    scope.Complete();}

What is happening here? Instantiating a new TransactionScope creates an “ambient” transaction.  What the heck is an ambient Transaction? An ambient transaction is part of the implicit programming model provided by the Systems.Transactions namespace. The msdn documentation advises developers to use this model instead of creating their own explicit transactions. The TransactionScope is ThreadStatic and therefore all ADO operations on that thread will participate in this ambient transaction.

There are some optional constructor arguments that can be used when creating a new TransactionScope that controls whether to create a brand new ambient transaction or to participate in one that already exists. For example, in the above block, it is possible that SaveEtitiesNotManagedByNH calls into methods that also have using blocks around a new TransactionScope. If new TransactionScopes are created with the default constructor, those scopes will participate in the same transaction as the one created by the outermost  TransactionScope. See the MSDN documentation for details on creating new TransactionScopes.

Since NHibernate 2.1 onward, the NHibernate transactions will participate in an ambient transaction if one exists. So in the code block above, all NHibernate  persistence logic is a part of the same transaction as any non NHibernate persistence code since it all occurs within the same TransactionScope.

When scope.Complete() is called, it simply sets a bit on the TransactionScope that claims that everything within the scope succeeded. When the TransactionScope is disposed at the end of the using block (or the outermost block if there are nested scopes), the transaction is commited as long as Completed was called. If completed was never called, the transaction is rolled back.

TransactionScope/NHibernate Pitfalls

So now that we have looked at the right way to manage the relationship between the  TransactionScope and NHibernate Transaction, lets look at some other techniques that might seem like a good idea but I assure you are bad.

Following a Single Transaction per Request Unit of Work pattern

Our original NHibernate implementation was following a Unit of Work pattern. This is a common and very nice pattern that typically involves opening an NHibernate session upon the first call into NHibernate and then closing the session in the EndRequest method of global.asax or a custom HttpModule. Our infrastructure took this one step further and opened a transaction after creating the session and committed or rolled it back in EndRequest. For now lets kindly suspend our disbelief in bad transaction management and pretend that this is a good idea because it supports a simple model where the developer can assume that all activity in a single request will succeed or fail together.

Following this technique, it is likely that an NHibernate explicit transaction will be created before an Ambient transaction exists.

session.GetEntityById(id);

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();     session.SaveOrUpdate(entity);     scope.Complete();}

Here, NHibernate is used to retrieve an entity which, in our framework, causes a Transaction to begin. Therefore the non NHibernate activity is in a completely separate transaction. When the ambient transaction is created, it will look for the presence of an existing ambient transaction. There will be none and a completely new and separate transaction will be created.  As a result, operations in these two contexts will have no visibility of uncommited data in the other.

To remedy this, we created a TransactionScope in BeginRequest and stored it in HttpContext. On EndRequest, we would retrieve that TransactionScope and call Completed and dispose of it. This way we are guaranteed that the ambient transaction  exists before BeginTransaction is called on the NHibernate transaction and we can be assured that the NHibernate operations will enlist in that transactions.

This would fail randomly because in ASP.NET, there is no guarantee that BeginRequest and EndRequest will occur on the same thread. A Transaction scope will throw an exception if you try to dispose of it on a different thread than the one where it was created. So in the occasional event that EndRequest executed on a different thread then BeginRequest, this exception was thrown.

I then tried using PreRequestHandlerExecute and PostRequestHandlerExecute instead of BeginRequest and EndRequest. Those will always occur on the same thread. It appeared that this was a working solution and worked in our dev environments. However, when we moved this to an integration environment, we began seeing database timeout errors. Reviewing the active connections in the database, spids were blocking and the spid at the top of the blocking chain was holding an open transaction. What was particularly odd was that this spid was locking a resource that we thought was being inserted outside of a TransactionScope or any explicit transaction.

It ended up that the reason this was happening is that there are certain circumstances where PostRequestHandlerExecute is never called. For example on a Response.Redirect where endRequest is false. Under such circumstances, the TransactionScope is not disposed. Since the TransactionScope is in Thread Local Storage, it remains tied to the request’s thread even after the request ends. The thread returns to the worker pool and later gets pulled by another thread. In our case, this was a background thread that typically never used explicit transactions and simply inserted a row into a commonly queried table. After the background thread finished its work, since it never calls Commit, the table remains locked and simple queries by other requests on that table time out. To summarize this unfortunate series of events:

  1. Web request begins and creates a new TransactionScope.
  2. The Request returns a 302 redirect and PostRequestHandlerExecute does not execute and therefore the TransactionScope is not disposed and remains in thread al storage.
  3. Thread returns to the worker pool.
  4. Background thread is launched from the worker pool and thread with undisposed TransactionScope is used.
  5. Thread inserts row into Table A.
  6. Thread exits and still does not call dispose and the transaction remains open.
  7. A new web request queries Table A and waits for the open transaction to release its exclusive lock.
  8. The new request times out since the lock is never released as the thread that owns the locking transaction is sitting idle in the thread pool.

This is a nasty bug that thankfully never reached production. These kinds of bugs are always difficult to to troubleshoot, occur in nondeterministic patterns and affect not only the user who came in on that thread but can bring down the entire application.

Reflecting on this scenario, it became apparent that:

  1. TransactionScope should always be limited to the confines of a using or Try/Finally block. Risking the fact that Dispose may never be called is too great.
  2. Keeping a transaction open during the entirety of a web request is extremely dangerous. It is unwise to assume that something, perhaps completely non database related, will never trigger a long running request and therefore lock up database resources that  again could bring down an entire site. This is one of the cardinal rules of ACID transactions to keep them as short as possible.

So I decided to eliminate the one transaction per request technique. We continue to use Unit of Work for Sessions, but developers will be responsible for defining their own transactions according to the needs of the request.

Failing to use explicit NHibernate Transactions within the TransactionScope

Having resolved to abandon a single master TransactionScope and continue to use the TransactionScopes currently sprinkled throughout the code, it seemed rational that we could simply invoke our NHibernate persistence calls within a TransactionScope and all would be well. Something like the following seemed innocent enough:

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();    session.SaveOrUpdate(entity);     scope.Complete();}

Well this quickly started throwing an error I had never seen before when disposing the TransactionScope. The message was something like “The state of this transaction is in doubt” with some methods in the stack trace calling something like PhaseOneCommit and an inner timeout exception. I’ll admit that  I have not dug into the NHibernate source to see exactly what is happening here, but it sounds like a distributed transaction gone bad. My guess is that the NHibernate activity is treated like a distributed transaction even though it is operating on the same database as the on NHibernate code. When NHibernate saves the data, that operation enlists in the TransactionScope but NHibernate has no awareness that it is involved and therefore it will never Commits its end of the distributed transaction and causes the TransactionScope to timeout.

The Final Solution

So the final solution involved a nesting pattern like the one shown at the beginning of this post. However, to make it easier to implemented I created a wrapper to wrap both the TransactionScopoe and the NHibernate transaction:

public interface IOrmOuterTransactionScope : IDisposable{    void Complete();}

public class NhOuterTransactionScope : IOrmOuterTransactionScope{    private readonly ISession session;    private readonly TransactionScope scope;    private readonly ITransaction transaction;

    public NhOuterTransactionScope(ISession session)        : this(session, TransactionScopeOption.Required, new TransactionOptions {            IsolationLevel = IsolationLevel.RepeatableRead})    {    }

    public NhOuterTransactionScope( ISession session,                                     TransactionScopeOption transactionScopeOption,                                     TransactionOptions transactionOptions)    {        scope = new TransactionScope(transactionScopeOption, transactionOptions);        this.session = session;        transaction = session.BeginTransaction();    }

    public void Complete()    {        session.Flush();        transaction.Commit();        scope.Complete();    }

    public void Dispose()    {        try        {            transaction.Dispose();        }        finally        {            scope.Dispose();        }    }}

Using an NHOuterTransactionScope, avoids the need for developers to create and commit the separate NHibernate transaction and more importantly it enforces the appropriate nesting order.