What you should know about running ILMerge on .Net 4.5 assemblies targeting .Net 4.0

I might have also entitled this:

“How to avoid TypeLoadException: Could not load type 'System.Runtime.CompilerServices.ExtensionAttribute' “

But I didn’t.

First, the moral of this story

I am about to take you on a debugging journey that will make some laugh and others cry and a fortunate few will travel through the entire spectrum of human emotion that will take them down into the seven bowels of hades only to be resurrected into the seven celestial states of ultimate being that will consummate in a catharsis that unifies soul, body and mind with your favorite My Little Pony character. If this does not appeal to you then know this:

If you use ILMerge to merge several assemblies into one on a machine with .Net 4.5 Beta installed and intend to have this merged assembly run on a machine running .Net 4.0, DO NOT use the following TargetPlatform switch value:

/targetplatform:"v4,c:\windows\Microsoft.NET\Framework\v4.0.30319"

 

Instead use this:

/targetplatform:"v4,C:\Program Files\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0"

If you are interested in learning some details about targeting different frameworks, some nice IL debugging tips or what that means in an upgrade like 4.5 that is “in place” or does not officially change the runtime version along with some techniques for debugging such interesting scenarios, then read on.

Twas the night before Beta

So it seems serendipitous that I write this on the eve of the Visual Studio 11 launch. This last weekend I installed the beta bits on my day to day development environments. As a member of the team that owns the Visual Studio Gallery and the MSDN Code Samples Gallery and their integration with the Visual Studio IDE, I’ve been viewing bits hot out of the oven for some time now. The product seems stable and everyone seem to feel comfortable installing it side by side with VS 10. You can target .Net 4.0 so why not just dev on it full time to enjoy the full, rich dogfooding experience. At home where I do development on my OSS project RequestReduce, I work on a 5 year old Lenovo T60P laptop. Its name is dale. So the perf and memory footprint improvements of VS11 have a special appeal to me. Oh and to Dale too. Right Dale? Thought so.

Most solutions can be loaded in both VS10 and VS11 without migration

So day 1, Saturday after some initial pain of getting a XUnit test runner up and running it looks like I’m ready to go. I load up RequestReduce and all projects load up fine. I’m also happy that after loading them in VS11, they still load and compile in VS10. Next I build in VS11 and all unit and integration tests pass. Sweet! Lets get to work and make it happen.

Cut on the bleeding edge

So I had been exchanging emails all week with a developer having issues with getting RequestReduce to play nice with the Azure CDN. Turns out Azure handles CDN URLs differently from the other CDNs I have worked with in the past by requiring all CDN content to be placed in a CDN directory on the origin server. However the CDN URL should not include the CDN directory in the path. There were some minor changes I needed to make to get the RequestReduce API to work nice with this setup. Just to be sure my changes were good, I spun up a new Azure instance and created a CDN endpoint. Then I deployed my test app with RequestReduce plugged in to do bundling and minification and WHAT?!

[TypeLoadException: Could not load type 'System.Runtime.CompilerServices.ExtensionAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.]

This doesn’t look good and my first reaction is “Stupid Azure.” But I have users who report that RequestReduce runs fine on azure. To double check I run the same test app on my dev box, all is good. So despite the promises that .net 4.5 and VS11 will run fine on .net 4.0, I create a clean VM with .Net 4.0 and run my test app there. Yep. I get the same error. I promptly shout to my secretary, “Marge, get me building 41 and cancel my tennis match with Soma!” Perhaps a cunning power play like this will motivate him to light a proverbial fire under the sorry devs that are causing these exceptions to be thrown. Then it all hits me: its Saturday, I have no secretary and lets be honest, I wont be playing tennis with Soma (head honcho at MS DevDiv that pumps out Visual Studio) any time soon. I weep.

Debugging Begins

I soon wake up recalling that I don’t even like tennis and I’m ready to fight. I’m gonna win this. From the stack trace I can see that the error is coming from my StructureMap container setup. StructureMap is a very cool IOC container used for managing Dependency Injection. It allows you to write code that references types by their interfaces so that you can easily swap out concrete implementation classes via configuration or API. There are many reasons for doing this that is beyond the scope of this post, but the most widely used application I use it for is testing. If I have all my services and dependencies handed to me from StructureMap as Interfaces, then I can create tests that also pass in interfaces with mock implementations. Now I have tests that don’t have to worry if all that code in the dependent services actually works. I have other tests that test those. I can control the data that these services will pass into the method I am testing and that allows my test to hyper focus on just the code in my method I’m testing. If this sounds unfamiliar or confusing, I urge you to research the topic. It transformed the way I write code for the better.

So one thing StructureMap has to do to make all of this possible at app startup time is scan my assemblies for interfaces and the concrete classes that I tell it to use. It is here where I am running into this TypeLoadException. So who are you 'System.Runtime.CompilerServices.ExtensionAttribute' and what is your game? Why do you play with me?

After some googling, I get a little dirt on this Type. Apparently it is an Attribute that decorates any class that contains an extension method In C#, you will never have to include this attribute directly. The compiler will do it for you. You will see it in the IL. Well this type has moved assemblies in .Net 4.5. Apparently it was too good for System.Core and has moved to an executive suite in mscorelib. This is all complicated by the fact that  the upgrade from 4.0 to 4.5 is what they call in “the industry” an In Place Upgrade. That means that you will not see a v4.5.078362849 folder in your c:\windows\Microsoft.Net\Framework directory. No. The 4.5 upgrade gets paved over the 4.0 bits and simply updates the DLLs in the C:\Windows\Microsoft.NET\Framework\v4.0.30319 folder. Not a real fan of this and I don’t know what the reasoning is but that’s how its done.

So now I’m thinking that this must be some edge case caused by StructureMap using Reflection in such a way where it demands to load types from what assemblies are there at compile time. I should also mention that I have a class in my assembly that has an extension method. So I find a way to tell StructureMap to go ahead and scan the assemblies but just don’t worry about any attributes. Since you can decorate classes with special StructureMap attributes that tell StructureMap that a class is a type that can be plugged into a specific interface, it will try to load all attributes it finds to see if it is one of these special attributes. Well I don’t use those so I tell StructureMap IgnoreStructureMapAttributes().

Ahhh. I am convinced that I “have it.” while my code builds and deploys to my VM all on my 5 year old laptop (remember Dale?), I have time to file a Connect bug nice and snug in the self righteous knowledge that I am doing the right thing. I have been wronged by these bleeding edge bits but I’m not angry. I am simply informing the authorities of the incident in the hopes that others will not need to suffer the same fate.

Ok. My code has now been built and deployed and is ready to run. Its gonna be great. I’ll see my asp.net web page with bundled and minified javascript and I can move on with my weekend chores. I launch the test app in a browser and…Criminey! Same error but different stack trace. Now its coming from code that Instantiates a ResourceManager. This is not instilling confidence. This isn’t even technically my code. It is code auto generated by VS when you add Resources to the Resources tabs in the VS Project properties. Really? VS’s own code isn’t even backwards compatible? The rubbish I have to work with. So it turns out the ResourceManager does something similar as StructureMap’s initialization, it scans an assembly for resources. It iterates over every type to see if it matches what you have told the ResourceManager what to look for. Ok Ok. I guess I’m just gonna have to refactor this too. And what next? When does it stop? When is enough enough?!

So I do refactor my Resource.

using (var myStream = Assembly.GetExecutingAssembly()    .GetManifestResourceStream("RequestReduce.Resources.Dashboard.html")){    using (var myReader = new StreamReader(myStream))    {        dashboard = myReader.ReadToEnd();    }}

If you are familiar with RequestReduce, this is the HTML page that is the RequestReduce Dashboard. I embed this as a resource in the RequestReduce DLL.  Now I load it through a ManifestResourceStream into a static string and to be honest this does seem much more efficient than scanning every class for resources when I know exactly which file contains my resource  and only need to load the contents of that file.

So I build and deploy again. And now, sweet victory, I see the beautiful blue background that is the default ASP.NET project home page and I see my minified CSS. But wait…oh no…something’s not right. The navigation tabs are stacked on top of each other and don’t look like tabs at all. Where is my JavascriptjQuery152004988980293273926_1358662903428 It is completely gone? If RequestReduce has a zero byte string after minification (maybe it was just a comment that gets minified out) then it will remove the script all together. So diving into the code deeper and running several tests to narrow the possibilities, I discover that the call into the MS Ajax minifier returns an empty string. Now that’s Minification!!

Its not .net 4.5’s fault

So all of the sudden I begin to wonder. Oh no! Is this not the Framework itself causing this mayhem but perhaps related to my use of ILMerge.exe which takes RequestReduce.dll, AjaxMin.dll, Structuremap.Dll and nQuant.dll and merges them all into one RequestReduce.Dll? I do this because the principle behind RequestReduce is to make Website optimizations as easy and automatic as dropping a single DLL into your bin. After I replace my merged DLL with the original unmerged ones in my test app on my .Net 4.0 VM, everything magically works. So I know now that it is either a problem in ILMerge or perhaps still a problem in the framework that is surfaced when interacting with ILMerge. Either way I want to get back up and running so I need to figure out what is going on with AjaxMin specifically. Is there something I can fix with the way I use AjaxMin or can I figure the more root problem with the Framework or ILMerge? Wouldn’t it be great if I were using an old version of ILMerge and simply updating it would fix everything?

I am using a version of ILMerge that was last updated in May and there have been two updates since the last being in November. I’m hopeful that since November was after the Preview Release of Visual Studio 11, this latest update will address .Net 4.5 issues. I update my ILMerge bits and alas, the problem still exists. So now I’m hoping that some thumbing through the ILMerge documentation or searching online will turn up some clues. Nothing. This is always the problem with working with bleeding edge bits. You don’t often get to learn from other peoples problems. You are the other people. I write this today as that “other person” who will hopefully shine light on your problem.

In a last ditch effort I send an email to Mike Barnett, the creator of ILMerge in hopes he may have come across this before and can provide guidance to get me around the issue and on my way to running code. He responded as I expected. He hadn’t played with the 4.5 bits yet and was not surprised that there would be issues especially given the fact of the in place upgrade. He was gracious enough to offer to look at the problem if I could provide him with the breaking code and access to a machine with.Net 4.5 installed.

I’m not one to quickly hand off problems on to someone else. First because it is rude and second I enjoy being able to solve problems on my own especially these kinds. In fact I had walked into the exact kind of problem that (while frustrated that my code was not running) I enjoy the most and often have a nack for getting to the bottom of figuring out what is going on. This is not because I am smart but because I am stubborn and off balance and will stick with problems long after smarter people have moved on with their lives.

The first thing I do is pull down the source code of the Ajax Minifier. I have a suspicion that the minifier is stumbling on the same exception I have been fighting with and it simply catches it and gracefully returns nothing. I discover that if there are any internal errors in the minification of content given to the minifier, it will store these errors away in a collection accessible from the minifier’s ErrorList property. When I inspect this property, there is one error reporting a type initialization error in Microsoft.Ajax.Utilities.StringMgr. So I look up that class and bang:

// resource manager for retrieving stringsprivate static readonly ResourceManager s_resourcesJScript =     GetResourceManager(".JavaScript.JScript");private static readonly ResourceManager s_resourcesApplication =     GetResourceManager(".AjaxMin");

// get the resource manager for our stringsprivate static ResourceManager GetResourceManager(string resourceName){  string ourNamespace = MethodInfo.GetCurrentMethod().DeclaringType.Namespace;  // create our resource manager  return new ResourceManager(    ourNamespace + resourceName,    Assembly.GetExecutingAssembly()    );}

My friend the ResourceManager again. Unfortunately because this is not my code, I cant refactor it as easily. Sure it is an open source project that I think takes pull requests and whose owner, Ron Logan, is very responsive to bug fixes, but refactoring these run ins with ExtensionAttribute is beginning to feel like an unwinnable game of Whack-A-Mole and since the error does not occur without ILMerge, I need to figure out what is going on there instead of cleaning up after the mess. As far as I’m concerned at this point, the only viable options are to find a way to work with ILMerge that will prevent these errors or gather enough data that I can give to either Mike Barnett or the .net team to pursue further. I’m hoping for the former but thinking the later scenario is more likely.

Isolate and minify the problem space

I often find with these sorts of problems, the best thing to do at this point is to widdle down the problem space to as small of a surface area as possible. I create a new VS11 solution with two projects:

Project 1

static class Program{    static void Main()    {        System.Console.Out.WriteLine(Resources.String1);    }

    public static string AnotherTrim2(this string someString)    {        return someString.Trim();    }}

 

This project also contains a simple resource string and the auto generated file produced by Visual Studio that contains the following line of code that reproduces the error:

global::System.Resources.ResourceManager temp =     new global::System.Resources.ResourceManager(        "ReourceManUnMerged.Properties.Resources",         typeof(Resources).Assembly    );

Project 2

An empty class file. I just need a second assembly produced that I can merge.

I ILMerge them together on my .Net 4.5 machine and then I copy ILMerge.exe and my unmerged bits to my .Net 4.0 VM and merge the same bits on the 4.0 platform. I then run both merged versions on .net 4.0 and sure enough the one that was merged on the .Net 4.5 machine breaks and the one merged on .Net 4.0 runs just fine. I now know I can work with these assemblies to troubleshoot. With the minimal code, there is a lot les to look at and get confused by. I did mention that I get easily confused right? Hell I’m confused as I type right now. Did I also mention that I am one of the individuals responsible for deploying MSDN Win 8/Vs11 Beta documentation in the next hour? Don’t let my boss know about the whole confusion thing. Some things are better kept a secret.

Pop open the hood and look at IL

The first thing I want to do is look at both assemblies using a new tool put out by JetBrains, the makers of such popular tools like Resharper, called DotPeek. This is the equivilent of the highly popular tool Reflector except it is free. It lets you view the C# source code of the disassembled IL of any .net assembly. A very handy tool when you cannot access the source of an assembly and want to peek inside. I’m curious if ILMerge reassembled these assemblies in such a way that they have different source that would clue me in to something useful.

They do not. The only difference between the two sets of source code is that the assembly that works includes a reference to System.Core – the .Net 4.0 home of the offending ExtensionAttribute. While this is interesting at one level, its not very actionable data since my source code explicitly references System.Core. So I cant just add the reference and expect things to fix themselves.

Next thing I do is I use ILDasm.exe, a tool that ships with the .Net SDK that can decompile a .Net DLL down to its IL code. Ive mentioned IL a couple of times now. This is the Intermediate Language that all .net languages get compiled to. I use this tool to view the actual IL emitted by ILMerge’s compile. You can access this tool from any VS command prompt or from C:\Program Files\Microsoft SDKs\Windows. Now I’m seeing something more interesting. The two sets of IL are identical except the one merged on 4.5 contains three instances of this line:

.custom instance void [mscorlib]    System.Runtime.CompilerServices.ExtensionAttribute::.ctor()     = ( 01 00 00 00 )

and the one merged on 4.0 has the same line but slightly different:

.custom instance void [System.Core]    System.Runtime.CompilerServices.ExtensionAttribute::.ctor()     = ( 01 00 00 00 )

See the difference? Now I’m thinking that one option I have is to modify my build script to use ILDasm to extract the IL, do a simple string replacement to get ExtensionAttribute to load from System.Core and then use ILAsm to reassemble the transformed IL back to a usable assembly. This is certainly a workable option but it does not feel ideal. What would be ideal is to find some way to tell ILMerge to use System.Core instead of mscorelib. Oh ILMerge, can’t we work together on this. Why must we fight. I love you, you love me, we’re a happy fam… Whoa. Where am I…oh yeah. When do I need to deploy those Beta bits?

What’s the compiler doing and how to get it to target 4.0

So I ask myself what is VS doing that is different when you tell it to target .Net 4.0 as opposed to 4.5? Since it seems that the outdated 4.0 bits are simply blown away when you install 4.5, how does the compiler know to use mscorelib when targeting 4.0? If I could answer this question, maybe that would reveal something I could work with. To discover this I go to Tools/Options… in VS11 and then I select Projects and Solutions –> Build and Run on the left. This gives me options to adjust the verbosity of the MSBuild output displayed in the output window at compile time. I switch this from Detailed to Diagnostic. I want to see the actual call to CSC.exe, the C# compiler and what switches Visual Studio is passing into it. Thankfully I get just that. When targeting 4.0, the command line call looks like this:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\Csc.exe     /noconfig     /nowarn:1701,1702,2008     /nostdlib+     /platform:AnyCPU     /errorreport:prompt     /warn:4     /define:DEBUG;TRACE     /errorendlocation     /highentropyva-     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\Microsoft.CSharp.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\mscorlib.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.Core.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.Data.DataSetExtensions.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.Data.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.Xml.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.0\System.Xml.Linq.dll"     /debug+     /debug:full     /filealign:512     /optimize-     /out:obj\Debug\ReourceManUnMerged.exe     /resource:obj\Debug\ReourceManUnMerged.Properties.Resources.resources     /target:exe     /utf8output     Program.cs     Properties\AssemblyInfo.cs     Properties\Resources.Designer.cs     "C:\Users\mwrock\AppData\Local\Temp\.NETFramework, Version=v4.0.AssemblyAttributes.cs" (TaskId:26)

When targeting 4.5 I get this:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\Csc.exe     /noconfig     /nowarn:1701,1702,2008     /nostdlib+     /platform:AnyCPU     /errorreport:prompt     /warn:4     /define:DEBUG;TRACE     /errorendlocation     /highentropyva+     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\Microsoft.CSharp.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\mscorlib.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.Core.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.Data.DataSetExtensions.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.Data.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.Xml.dll"     /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework \.NETFramework\v4.5\System.Xml.Linq.dll"     /debug+     /debug:full     /filealign:512     /optimize-     /out:obj\Debug\ReourceManUnMerged.exe     /resource:obj\Debug\ReourceManUnMerged.Properties.Resources.resources     /target:exe     /utf8output     Program.cs     Properties\AssemblyInfo.cs     Properties\Resources.Designer.cs     "C:\Users\mwrock\AppData\Local\Temp\.NETFramework,Version=v4.5.AssemblyAttributes.cs"     obj\Debug\\TemporaryGeneratedFile_E7A71F73-0F8D-4B9B-B56E-8E70B10BC5D3.cs     obj\Debug\\TemporaryGeneratedFile_036C0B5B-1481-4323-8D20-8F5ADCB23D92.cs (TaskId:21)

Telling ILMerge where to find mscorelib – the devil in the details

There are a few differences here. The one that is most notable is where the framework references are being pulled from. They are not pulled from the Framework directory in c:\windows\Microsoft.Net which is where I would expect and where one usually looks for framework bits. Instead they are coming from c:\Program Files\Reference Assemblies. The MSBuild team talks about this folder here.

When you call ILMerge to merge your assemblies, you pass it a /targetplatform swich which tells it which platform to build for. Currently this switch can take v1, v1.1, v2 or v4 followed by the Framework directory. When I build for 4.0 I use this command line call via powershell:

.\Tools\ilmerge.exe     /t:library     /internalize     /targetplatform:"v4,$env:windir\Microsoft.NET\Framework$bitness\v4.0.30319"     /wildcards     /out:$baseDir\RequestReduce\Nuget\Lib\net40\RequestReduce.dll     "$baseDir\RequestReduce\bin\v4.0\$configuration\RequestReduce.dll"     "$baseDir\RequestReduce\bin\v4.0\$configuration\AjaxMin.dll"     "$baseDir\RequestReduce\bin\v4.0\$configuration\StructureMap.dll"     "$baseDir\RequestReduce\bin\v4.0\$configuration\nquant.core.dll"

Most point the directory like I am to the actual platform directory that can always be located off of %windir%\Microsoft.net It’s the obvious location to use. Oh wait…hold on…

Ok I’m back. The win8 and VS11 beta docs are now deployed. Now where was I. Oh yeah…the framework directory passed to ILMerge. According to the ILMerge documentation it just needs the directory containing the correct version of mscorelib.dll. So I’m thinking, lets use this Reference Assemblies directory. I do that and re merge on .Net 4.5. I run on my 40 VM and…Hallelujah!! There’s my javascript in all of its minified glory.

I hope this helps someone out because I didn’t find any help on this error.

Also, if you have made it this far, as I promised in the beginning, you should now have a sense of unity with your favorite My Little Pony character. Mine is Bon Bon what's yours?

VerificationException: “Operation could destabilize the runtime” An exception with a bark much worse than its bite

As a software engineer my greatest ambition is to produce code that will have a lasting impact on my fellow humans. I want to make good things happen to both good and bad people. The last thing I want to do is destabilize the runtime. Whether it be my runtime or your runtime, if you cant run and there is no time…hmmm…sounds kinda like death which of course has its own baggage and negative set of connotations.

So, you can now understand the horror I must have experienced yesterday morning when I got this:

Couple things worth noting here: “[No relevant source lines]”. Oh that’s helpful. Is the source so potentially harmful that the runtime has deemed the offending code “irrelevant.” Yeah? Well I think YOU are irrelevent .NET! I’m gonna put your worker threads in a warehouse and shut down your I/O completion ports. Minutes pass. Absolute silence. .Net remains unmoved by my threats and I realize that I must use what little intelligence I have to slog through this and figure it out for myself. The rest of this post is a narrative account of just that process. So sit back, reach for that glass of pinot and enjoy this tale of Verification Exceptions in Medium Trust on the .Net 2.0 runtime. Mmmm. You can tell already its gonna be good.

What is a Verification Exception

According to the official MSDN Library documentation, a VerificationException is “The exception that is thrown when the security policy requires code to be type safe and the verification process is unable to verify that the code is type safe.” Good enough for me. But for the obtuse, we will explore further.

Well unfortunately it doesn’t take much exploring to discover that there is not a lot of detailed explanation on this error and that it can also be raised by a large variety of very different scenarios. From what I can gather from all the fragments of blogs and forums I found that touched on various permutations of my scenario.

So what is my Scenario?

Occurs in .net 2.0 runtime under Medium Trust on x64

I have an assembly that I want to be able to run in .net 3.5 and up and in hosting environments that are restricted to Medium Trust. Here is the line that triggers this exception:

AuthorizedUserList = config == null || string.IsNullOrEmpty(config.AuthorizedUserList)                                     ? Anonymous                                     : config.AuthorizedUserList.Split(',').Length == 0                                           ? Anonymous                                           : config.AuthorizedUserList.Split(',');

Its note worthy that variables AuthorizedUserList and Anonymous are both typed IEnumerable<string>.

This exception is only thrown when running this line on .net 3.5 in Medium trust. What I find particularly odd and don’t have an explanation for is that it worked fine in .net 3.5 Medium trust on my 32 bit machine but throws this exception on my 64 bit work laptop. I’m not convinced that it is the bitness level and not some other environment issue that makes the difference here. Sometimes things are just more fun when they remain a mystery. Especially in software don’t you think?

How to find the root of the problem

So looking at the above line of code my first reaction was that there was something wrong with the debug symbol mapping to the assembly I was using. I mean how does this line look harmful. Wrap it in fur and stuff it with cotton and you’d want to do no less than squeeze it close to your bosom and sing soft lullabies to it. So I proceeded to play with compiling configurations and changing references which proved entirely futile.

The golden nugget that I was missing was a tool that ships with the .net SDK called PEVerify. I made this connection reading this StackOverflow answer. One key thing to be aware of is that each version of the .NET runtime has its own version of PEVerify so make sure to use the one that ships with the version of the runtime you are getting this exception with. In my case I needed the .Net 2.0 SDK which you can find here.

PEVerify is a Command line utility that verifies that the IL in an assembly is type safe in a particular runtime environment. Why the .net 2 compiler cant report these as warnings, I do not know. So entering:

C:\Program Files (x86)\Microsoft Visual Studio 8\SDK\v2.0\Bin>peverify "C:\RequestReduce\RequestReduce\bin\v3.5\debug\RequestReduce.dll" /verbose

I got this output:

Microsoft (R) .NET Framework PE Verifier.  Version  2.0.50727.42Copyright (c) Microsoft Corporation.  All rights reserved.

[IL]: Error: [C:\RequestReduce\RequestReduce\bin\v3.5\debug\RequestReduce.dll :RequestReduce.Configuration.RRConfiguration::.ctor][mdToken=0x60000ce][offset 0x000000BB][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IEnumerable`1[System.String]'] Unexpected type on the stack.1 Error Verifying C:\RequestReduce\RequestReduce\bin\v3.5\debug\RequestReduce.dll

Ahhh. Its all making perfect sense now. Well if you think about it (a practice that I’m quite rusty with but sometimes still capable of), it does help. As I mentioned above and is illustrated here, AuthorizedUserList expects an IEnumerable<string>. The PEVerify output complains that it is getting a plain old IEnumerable (not its generic cousin). This does make sense since the config.AuthorizedUsers.Split(‘,’) returns a string[].

Obviously under most circumstances, there will be no problem implicitly casting the string[] to an IEnumerable<string>. The code does run in .Net 2 and 4 in Full Trust and has never caused a problem. So .Net 2.0 must think that this conversion could potentially be un type safe and if the user is running in partial trust, the fact that its runtime type checking verification fails, causes this exception to be thrown.

Forcing a cast using ToList() fixes the problem

So adding nine characters to the end of the line fixes everything:

AuthorizedUserList = config == null || string.IsNullOrEmpty(config.AuthorizedUserList)    ? Anonymous    : config.AuthorizedUserList.Split(',').Length == 0        ? Anonymous        : config.AuthorizedUserList.Split(',').ToList();

And now the output of PEVerify is:

C:\Program Files (x86)\Microsoft Visual Studio 8\SDK\v2.0\Bin>peverify "C:\RequestReduce\RequestReduce\bin\v3.5\debug\RequestReduce.dll" /verbose

Microsoft (R) .NET Framework PE Verifier.  Version  2.0.50727.42Copyright (c) Microsoft Corporation.  All rights reserved.

All Classes and Methods in C:\RequestReduce\RequestReduce\bin\v3.5\debug\RequestReduce.dll Verified.

So that’s it. Hopefully someone will find this useful but from what I can tell from others who have experienced this same error, the causes can vary widely.

Looking back on this title, I’m wondering if the bark really is worse than the bite. I suppose the good news is that in the end, no runtimes were destabilized in the making of this blog post or the events that led up to it.

Being Nice

One of qualities that I think make a great Software Development Engineer not to mention a good human being is the ability to be nice. In fact if I had to list the top 5 things that have helped me in my career, being nice would be one of them. I’m not saying that I have been particularly successful in the art of being nice (yes, it is an art) and I can think of far too many times when I was not nice, but I have made being nice a guiding principle from which I draw to color my decisions and actions and it is an incredibly powerful tool. This may be obvious and if you are thinking the same, that’s a good sign for you and your employer. One may think that it’s a shame that such a concept even deserves a blog post but alas…

The world has far too many meanies

I came head to head with this reality many many moons ago when I worked as a waiter. Up until this point I was certainly aware that there were a good share of not nice and downright mean and inconsiderate people in the world. However, I thought they were a small minority. Waiting on tables for a “Just a notch above Denny’s scale” restaurant quickly brought home the reality that there are a lot more unpleasant people than I had ever thought. I had never suspected that there could be so many rude, inpatient, and down right awfully behaved adults among me.

Well fortunately I run into fewer not nice people per capita in the Software industry, but they are there. However, this post is not really intended to focus on mean programmers, but about developers who go the extra mile to be nice. Sure a lot of us can be nice or at least  tolerable, but some of us are lacking when it comes to being nice backed with intention.

Being nice adds value

When I think of the top dozen people that have been a true joy to work with and taught me the most in terms of leadership and software best practices, they were all nice. Being nice provides a wonderful delivery mechanism for all sorts of value beyond just being nice. While I believe there are rare exceptions, being nice is the best means by which you can color your personal interactions and effectively drive home quality. Of course quality can certainly be delivered without being nice – certainly without going out of one’s way to be nice, but I guarantee you that if you add niceness to your interactions, it will positively influence your ability to drive home your philosophies, to promote adoption of your software and to unlock career opportunities.

Going back to thinking about the people in the industry I admire who are either “celebrity” types I don’t personally know or people I have worked extensively with, their ability to be nice really is more than a nuance, it a concrete thing about their character that is obvious and ends up being something that I specifically call out as being something I like about that person and motivates me to want to interact with them again.

It has to be genuine

The last thing I want is for someone to interpret this as a Machiavellian attempt to gain favor from others or get others to do your bidding. We have all been exposed to those types. You’ll see this a lot in recruiters or sales individuals. You can often smell the insincerity. They like to say your name a lot in a way that makes you want to shower with a wire brush after each utterance.

No. When I think about these nice value packed individuals, they are nice because it’s a core part of who they are. They didn’t just read a pamphlet of nice pointers. They are nice because they want to be nice and they enjoy being nice.

Nice. A definition.

So what is this nice I speak of? Perhaps nice is not the best word for the point I am trying to get across. I’m talking about more than just good manners but I’m not necessarily talking about Jesus Christ either. Nor am I talking about Santa Clause. But he does seem super super nice.

I’m taking about an intentional, proactive means of communicating and relating with others that seeks out their benefit (and perhaps among several other benefits) and is ignorant of status or other artificial cast systems in your organization. Interacting with others as an equal and trying to be helpful. Here’s a small but perhaps tangible example. I’m talking about the difference between someone that answers all their IMs with “Hey”  or “Yeah” compared with Hi Matt. This can be incredibly subtle. Here is why this example resonates with me. I used to work with a developer in Chile. Remotely of course. He was incredibly competent and he was very nice. He never sent me flowers or showed huge concern if I was sick but there was a tone to every IM conversation and email he sent. A tone of sincere helpfulness. Every initial answer to an IM was “Hi Matt!” His emails were often genius. But there was no ego in them, they were plain, to the point, clear and thorough and had an air of “I hope that this is helpful” to them. Everyone thought he was nice but then no one would say, “That Fernando, he is one of the nicest guys I know.” He was pretty ordinary but an example of how simply being nice can go a long long way.

In an an environment where the dev team hated to work with offshore vendors, everyone loved working with this person. He was one of the most highly paid engineers on staff. At the time he reported to me, he made more money than I made and he was worth every penny. In large part because he was nice. Honestly, when I think back to working with this individual, that was one of his most valuable characteristics. Sure his technical competencies were superb, but colored with his niceness, they were effective as well.

Can brilliance be an excuse for not being nice?

I know there are some who will argue that although being nice is indeed a desirable quality, There are some who are certainly not nice but they are “brilliant.” I agree. There are some truly brilliant minds who are not nice. Yes, these people have often managed to be successful and influential. But generally speaking, I’m not interested in working with them. I may value a lot of what they have to say, but I’m not particularly keen to really engage with them.

When it comes to recruiting, I simply will not hire someone (or suggest hiring someone) who comes across as being a jerk. While we should strive to produce quality software and attract  candidates who are great problem solvers and solution architects, above that I want quality of life. I want to work with others who are pleasant to be around. Its good for me, its good for the product and its good for the team. Software development competence is easily taught, being nice is built on a lifestyle developed over a lifetime  Having interviewed hundreds of candidates, if I could put candidates into buckets of not nice, some what nice (basically not not nice) and candidates who were truly nice. The ones in the not nice bucket just don't get hired, the somewhat nice group have a fair chance of being hired proportional to their standard competencies. The nice ones often get hired and people are excited to hire them and look for ways to justify hiring them. Don’t get me wrong. They have to show promise as developers, but being nice is like the Malt Vinegar on fish and chips. It’s the more than subtle nuance that makes me say Mmmm.

Things that make me go Mmm (Things that make me go Mmm) Things that make me go Mmm, yeah-eah-eah.Things that make me go Mmm, Mmm, Mmm (Mmm). Things that make me go Mmm.

I apologize. That was embarrassing. I will now commence with the blog post.

Compare two developers next to each other, one is difficult to be around but extremely competent and the other is very pleasant and competent but not as much as the former. I would argue that the potential of the second individual’s success is greater than the first.

The consequences of impolite behavior is often more destructive than valuable

As software engineers, many of us were not popular when we were younger. We escaped to a world of code and video games to avoid others and feed our spirit with challenges and opportunities for private success. Many of us were not stars on the soccer field. We were asked to kick the ball out (I was) instead of risking what we might actually do with that ball in the extremely unlikely event that we manage to actually gain control of the ball at all (I never did).

So it should come as no surprise that many of us share the urge to put others down in order to enhance our own sense of lacking self importance. Just spend a day reading your technically saturated twitter feed. There is a lot more complaining and snark than truly helpful tweets. But I’d ask anyone to think back on their own snarky remarks. The ones they thought were true gems. They really put a person or followers of a particular technology in their place. A place much inferior to your own views. Now consider how valuable the comment was. How many were likely to change their attitudes or stumble upon better practices after hearing such remarks? Did the remark have an impact? Yeah it felt good to say at the time, but looking back did it add or subtract from your net value as a contributor to your craft? I’m gonna guess that it deducted more value than it added. At least that is my own experience.

When I think back on some of the scenarios where I publicly put someone down or spoke condescendingly to someone as I complained about their technology or practices, in the end I really only made myself feel good and look bad at the same time. And none of those moments were moments that “made a difference” in moving my team, my values or my career forward.

I hope you have found this to be a nice post.

Bug fixes and enhancements included in RequestReduce 1.7.26

I usually don’t blog on bug fix releases. However the bug fix release I deployed today addresses a couple serious bugs (albeit edge cases) and their fixes forced a few significant enhancements I want to call out.

1. Css that reference the same image twice may produce sprite sheets that cut out some of the images at the end of the sprite sheet.
2. The logic that determines which css selectors can be included within another css selector sometimes breaks with child selectors (eg. .parent-class > .child).
3. Upgraded to v4.44 of the MS Ajax Minifier which addresses a bug that causes a JS error in IE8 with Google’s Prettify.js. While IE8 throws an error, all other browsers suffer from the fact that this bug essentially breaks the intentions of prettify.js.
4. RequestReduce was not handling @font-face urls or some data URIs. These are now supported.

There were other bugs but these are the really important ones worth mentioning. At work (Microsoft MSDN and Technet web properties) we are preparing for our February release and there have been a couple minor issues raised in addition to the bugs above that have forced the following enhancements.

1. RequestReduce now handles background-position properties that use pixels, percentages or the directional attributes of top, left, right, center, bottom. Previously, RequestReduce did not consider percentage values or positive pixel units.
2. Some tweaks to the nQuant quantizer parameters increase the quality of the images without sacrificing file size. While RequestReduce generally produces high quality png8 sprites, there are rare cases where images may appear grainy or overly opaque. Today’s fix addresses most of these instances. There may still be times where image quality is sub par. This is most likely to happen in very smooth gradients that has a lot of variance in transparency levels. The best way to deal with these is to either disable the image quantization or to decrease the spriteColorCount setting. The fewer colors that the nQuant quantizer has to reduce to 255, the higher the quality of the final image. There is really no hard fast rule here. RequestReduce defaults to 5000, but depending on your images, 10,000 may be fine or you may need to reduce to 2000.
3. Thanks to PureKrome, the RequestReduce dashboard is a little easier on the eyes when displaying lists of urls and it includes exception info for failed requests. I really like this and it improved my quality of life doing some debugging last night.
4. Added a API hook to allow one to transform the absolute url generated by RequestReduce. A popular application of this is to add prefixed subdomains to CDN hosts to allow browsers to load more content at the same time. See the API Wiki for details.
5. Upgraded the SassAndCoffee dependency to 2.0.2. This eliminates some of the x64 instability issues of the former version and uses the new IE Chakra JS engine if available. It no longer uses V8.

Released RequestReduce 1.7.0: Giving the RequestReduce onboarding story a happy beginning

About six weeks ago I blogged about an issue with RequestReduce and its limitations with resolving image properties of each CSS class. To recap, until today, RequestReduce treated each CSS class as an atomic unit and ignored any other classes that it may be able to inherit from. The worst side effect of this is a page that already has sprites but uses one class to specify the image, width, height, and repeatability. Then uses several separate classes each containing the background-position property of each image in the sprite sheet. Something like this:

.nav-home a span{    display:block;    width:110px;    padding:120px 0 0 0;    margin:5px;float:left;    background:url(../images/ui/sprite-home-nav.png?cdn_id=h37) no-repeat 0 1px;    cursor:pointer}

.nav-home a.get-started span{background-position:0 1px}

.nav-home a.download span{background-position:-110px 1px}

.nav-home a.forums span{background-position:-220px 1px}

.nav-home a.host span{background-position:-330px 1px}

What RequestReduce would do in a case like this is resprite .nav-home a span because it has all of the properties needed in order to construct the viewport and parse out the sprite correctly. However, once this was done, the lower four classes containing the positions of the actual images rendered a distorted image. This is because RequestReduce recreated a new sprite image with the original images placed in different positions than they were on the original sprite sheet. So the background positions of the other nav-home images point to invalid positions.

If you are creating a site that pays tribute to abstract art, you may be pleasantly surprised by these transformations. You may be saying, “If only RequestReduce would change my font to wing dings, it would be the perfect tool.” Well, unfortunately you are not the RequestReduce target audience.

RequestRecuce should never change the visual rendering of a site

One of the primary underlying principles I try to adhere to throughout the development of RequestReduce is to leave no visible trace of its interaction. The default behavior is always to optimize as much as possible without risk of breaking the page. For example, I could move scripts to the bottom or dynamically create script tags in the DOM to load them asynchronously and in many cases improve rendering performance but very often this would break functionality. Any behavior that could potentially break a page must be “requested” or opted in to via config or API calls.

This spriting behavior violated this rule all to often. I honestly did not know how wide spread this pattern was. My vision is to have people drop RequestReduce onto their site and have it  “just work” without any tweaking. What I had been finding was that many many sites and most if not all “sophisticated” sites already using some spriting render with unpleasant side-effects when they deploy RequestReduce without adjustments. While I have done my best to warn users of this in my docs and provide guidance on how to prepare a site for RequestReduce, I had always thought that the need for this documentation and guidance would be more the exception than the rule.

I have now participated in onboarding some fairly large web properties at Microsoft onto RequestReduce. The process of making these adjustments really proved to be a major burden. Its not hard like rocket science, its just very tedious and time consuming. I think we’d all rather be building rockets over twiddling with css classes.

Locating other css properties that may be used in another css class

It just seemed to me that given a little work, one could discover other properties from one css class that could be included in another. So my first stab at this was a very thorough reverse engineering of css inheritance and specificity scoring. For every class, I determined all the other classes that could potentially “contribute” to that class. So given a selector:

h1.large .icon a

Dom elements that can inherit from this class could also inherit from:

a.icon ah1 .icon a.large .icon a.large aetc...

For every class that had a “transformable” property (background-image or background-position), I would iterate all other classes containing properties I was interested in (width, height, padding, background properties) and order them by specificity. The rules of css specificity can be found here. Essentially each ID in a selector is given a score of 100, each class and pseudo class a score of 10 and each element and pseudo element a score of 1. Inline styles get a score of 1000, but I can’t see the dom and the “Universal” element or * is given a score of 0. Any two selector with a matching score determines its winner by the one that appears last in the css.

Once I had this sorted list, I would iterate down the list stealing missing properties until all my properties were occupied or I hit the end of the list.

At first this worked great and I thought I was really on to something but I quickly realizing that this was breaking experience all too often. Given the endless possibilities of dom structures, there is just no way to calculate without knowledge of the dom, which class is truly representative. Eventually I settled on only matching up a background image less selector with a background-property with the most eligible and specific selector containing a background-image. While even this strategy could break down,  so far every page I throw this at renders perfectly.

Although this feature does not add any optimization to other sites and only assists larger sites to RequestReduce, I’m excited to provide a smoother adoption plan. As a project owner who wants his library to be used, I want this adoption plan to be as smooth and frictionless as possible.

What else is in v1.7.0?

Here is a list in the other features that made it into this release:

1. Improved performance of processing pages with lots of sprites. This is done by loading each master sprite sheet into memory only once and not each time an individual sprite image is found.
2. Prevent RequestReduce from creating an empty file when it processes a single script or css file containing only a comment. After minification, the file is empty.
3. Support Windows Authentication when pulling script/css/sprite resources.

What’s Next?

Good question. Probably being able to process scripts due to expire in less than a week. Soon after that I want to start tackling foreground image spriting.