Microsoft’s MSDN and Technet Forums and Search adopt RequestReduce for Web Performance Optimization /  by Matt Wrock

This week RequestReduce was launched on Microsoft’s MSDN and Technet web sites specifically with the Forums and Search applications. So if you ever land on a Forums page within these sites or conduct a search within the MSDN or Technet environment, RequestReduce is crunching and merging the CSS and javascript as well as spriting their background images resulting in about a 25-30% Reduction in the number of HTTP requests made.. RequestReduce already services the MSDN and Technet gallery sites which include Code Samples and the Visual Studio Gallery. All of these sites combined represent several million page views a day and prove that RequestReduce can perform under considerable load.

This migration process did surface some “edge case” bugs in the Sql Server cache flushing logic that would likely only surface in high traffic environments. The Sql Server cache is an add on assembly that is aimed to synchronize the RequestReduce generated Scripts, CSS and images across multiple servers and ensure that users do not view different or broken content when moving from one server to another but rather guarantees a consistent user experience throughout a session.

The discovery of these bugs provided the bulk of the code that went into this week’s 1.4 Release of RequestReduce. The release was largely a fix and polish release but here are some highlights:

Dashboard Page

Troubeshooting issues in the Forums/Search migration inspired me to create a dashboard page to assist in catching and isolating some tough to catch bugs. The page is currently extremely bare bones and rather cryptic. It displays what css and javascript has been processed, what is queued for processing and what is currently in the middle of being processed. It also provides links for flushing individual merged css/js files from the cache. The page can be accessed from:

http://<your site>/<RequestReduce virtual directory>/dashboard

Its greatest benefit is to assist in determining what RequestReduce is currently up to if anything and where RequestReduce is in terms of processing all queued content. Expect this page to get much more informative and maybe prettier in coming releases.

Performance Optimization

The javascript processing introduced in release version 1.3, added a lot of work to the Response Filter which has to iterate over every character of the response stream to the browser. This is the single piece of code that needs the most perf optimization since it is the only request blocking activity within RequestReduce. Its speed has a direct impact on the Time to First Byte download of the hosting RequestReduce enabled web page.

So I changed some List<T> implementations to arrays and removed some unnecessary lines of code. Granted these are micro optimizations but even 50 – 100ms of time filtering a page is far too long and this process has to be extremely optimized. Its also a bear to read, write and change which I plan to address in a future release. Sure I could eliminate several lines of code and overly complicated logic by using regular expressions, but that would actually degrade performance significantly.

Speaking of regular expressions. I do make extensive use of them in the background processing of content. Especially in CSS parsing for sprites and imports. So I eliminated a lot of my static Regex instances and replaced them with instance member Regex’s in a singleton class. .net will cache the most recently used 15 static regular expressions in an application and I do not want to infringe upon that cache especially when I have about 8.Using a singleton allows me to manage my own cache.

Support for CSS @Media

CSS media types is something that can be applied to an entire CSS stylesheet, instructing the browser to use its styles only for specific media. For example: print and screen are the most popular and obvious. RequestReduce had been oblivious to the Media attribute of the Link tag and @import directive. This caused some CSS to break on certain sites. Especially if the last CSS on a page was targeted for Print media and strips out a lot of styling. This was resulting in the web experience having the styles striped out as well. No longer. RequestReduce now wraps the contents of any <Link/> or @import inside @media braces to preserve the media constraints they specify.

Support for ScriptResource.axd scripts and all other Compressed content

RequestReduce already supported the minification and merging of WebResource.axd files. However, RequestReduce always assumed that the content it was requesting was uncompressed text since it did not specify compression in its Accept-Encoding request headers. Well it ends up that ScriptResource.axd files violate HTTP by sending GZIP’d content regardless of the value in this header. This is a show stopper for an upcoming large RequestReduce adoption in the near future that includes pages with over 30 to 40 CSS and script files including tons of ScriptResource.axd files. RequestReduce now looks to see if a Response is compressed with gzip or deflate and uncompresses accordingly. RequestReduce leaves the recompression up to IIS.

This really makes RequestReduce an attractive solution for sites mired in asp.net 2.0/asp.net ajax constructs that had good intentions of encapsulating scripts but horrible perf implications.

Released: RequestReduce 1.3 Now Runs on .net 3.5 and a Preview of Support for Third Party Javascripts in your Minification and Merge Process /  by Matt Wrock

Welcome .net 3.5 Users!

Earlier (MUCH earlier) this morning I released version 1.3 of RequestReduce. While not as exciting as the previous javascript crunching and merging release, I am very happy that a broader base of users will now have access to this resource. It can now be installed on any server running .net 3.5. Huge thanks to my co worker Matt Hawley (@matthawley) for this pull request.

As part of this effort, Matt ripped out the Sql synchronization code and moved it into a new assembly RequestReduce.SqlServer which is now available as a separate nuget package. This code uses EntityFramework Code First which is limited to .net4.0. So if you need this functionality (most do not), you must be running .net4 (I’m also open to pull requests using a different data access strategy – not at all married to EF). If you are interested in learning more about synchronizing RequestReduce across multiple servers using SqlServer, see this wiki.

What else is in this release?

  1. Fixed javascript inside IE conditional comments. RequestReduce will ignore these just as it does with CSS links inside conditional comments
  2. Added a configuration setting to turn off image spriting: imageSpritingDisabled
  3. Added some optimizations to the Response filter

Troubleshooting Guide

In addition to these features, I have added a troubleshooting wiki to the github site. I have gotten a couple issues reported from people who were unable to get RequestReduce running at first due to issues in their environment. This guide should allow most users to at least begin troubleshooting their situation. However, if for any reason you find yourself spending a lot of time investigating this, please do not hesitate to file an issue and I will get on it usually within 24 hours. If most cannot successfully use RequestReduce from a typical nuget install, I have failed.

What’s Next?

The two key features up next for RequestReduce are:

  1. Support for the CSS Media attribute. RequestReduce has no awareness of this attribute and ignores it. While most do not use this, it can have breaking consequences right now with RequestReduce.
  2. Automatic Content refreshing. This is a feature I’m particularly excited about because it will not only make changing content refresh more quickly, it will allow users to have many third party scripts included in the crunch and merge process. Right now RequestReduce ignores many third party scripts that have expiring headers. With this feature, RequestReduce will occasionally send a head request to the original script in the background to see if content has changed, if it has, RequestReduce will refetch and reduce the content.

This Release (1.3) is now available at http://www.RequestReduce.com and on Nuget. Don’t forget: if there are features in RequestReduce you would like to see added, please suggest them on the github issues page. I received two requests last week and both were added to this release.

Released: RequestReduce 1.2 with Javascript merge and minify /  by Matt Wrock

With this week’s release of RequestReduce 1.2, RequestReduce is even more effective in reducing the HTTP requests made by a browser when visiting your website. Most significantly, this release adds the minification and merging of external javascript resources. A big thanks goes to Matt Manela for providing the initial Javascript Reduction code! Other mentionable features in this release include:

  • Automatic expansion of CSS @Import. All imports will be downloaded and injected into the calling CSS and then minified and merged with the rest of the page’s head CSS.
  • Wider support for PNG lossless compression. RequestReduce now issues the –fix parameter to optipng which will attempt to fix PNGs with extraneous data.
  • Ignore 404 and 500 failures of individual resources. In other words, if one script or stylesheet fails to crunch, the rest should still succeed.
  • Several bug fixes to the overall response filtering.

RequestReduce 1.2 is available via Nuget installations and from downloading it at http://www.RequestReduce.com. I will discuss the new features in some detail but first…

What makes RequestReduce the best static resource cruncher available?

OK. Glad I got that off my chest. Now for the details of this release’s features:

Javascript Crunching: What exactly is crunched

RequestReduce will minify all external javascripts that have a valid javascript mime type, do not have a Cache-Control header set to No-Cache or No-Store and does not have an Expires or max-age of less than a week. Furthermore you can explicitly tell RequestReduce what scripts you would like it to skip by supplying a comma separated list of url substrings. For example, on the Microsoft properties I work on, I have this set to /tiny_mce since the way that /tiny_mce loads its plugins does not play nice when the scripts are not in the same directory where tiny_mce expects them to be.

By default, if no urls are provided in this list, RequestReduce will ignore the Google and Microsoft JQuery CDNs. It does this since it is likely that browsers will not have to download this content because most will have already cached those resources while visiting a Google or Microsoft property in the past.

You can also use more complicated or customized rules via code. RequestReduce exposes a Func for approving the processing of urls. You can hook into this by registering a lambda or delegate to this Func in your Global.asax startup. Here is an example:

var javascriptResource = RRContainer.Current.GetInstance<JavaScriptResource>();javascript.TagValidator = ((tag, url) =>             {                if(url.EdsWith(".myjss") || tag.Contains("data-caching=dont"))                    return false;            });

The first input param is the full script tag element and the url is the javascript SRC.

Important considerations in the placement of your javascripts

In a perfect world, I would crunch all your scripts together and dynamically inject them into the DOM to load asynchronously. In fact, because the world is perfect, I could even defer the loading of the scripts until after document complete. But alas, the world is not perfect. RequestReduce knows absolutely nothing about your code and takes the safest approach to ensure that your page does not break.Therefore, RequestReduce will make sure that any dependencies that your markup has on your scripts or that your scripts have on other scripts are not broken.

RequestReduce will only merge blocks of scripts that can be reduced contiguously. For example assume you have the following scripts on a page:

<html>    <head>        <link href="http://localhost:8877/Styles/style1.css" rel="Stylesheet" type="text/css" />        <link href="/Styles/style2.css" rel="Stylesheet" type="text/css" />        <script src="http://localhost:8877/Scripts/script1.js" type="text/javascript"></script>        <script src="/Scripts/script2.js" type="text/javascript"></script>    </head>    <body>        <div class="RatingStar">&nbsp;</div>        <div class="Up">&nbsp;</div>        <div class="Rss">&nbsp;</div>        <div class="Search">&nbsp;</div>        <script src="http://localhost:8877/Scripts/script3.js" type="text/javascript"></script>        <script src="http://www.google-analytics.com/ga.js" type="text/javascript"></script>        <script src="/Scripts/script4.js" type="text/javascript"></script>    </body></html>

You will end up with five scripts:

  1. The scripts in the <head/> will be merged together and remain in the head.
  2. Script 3 will be minified but will not be merged since it lies just below common markup and above a near-future expiring script (google analytics).
  3. The google analytics script remains entirely untouched since it is a near future expiring script.
  4. Script 4 will be minified but not merged since it is not adjacent to any other non near futures script.

Although script three and four do not benefit from merging, they are at least minified and served with far future expiration and a valid ETag.

With all this in mind, you want to make sure that you group scripts together that are not near future expiring and can remain together without causing issues with load order dependencies. Also make sure that you do not have any setting that causes all of your javascript to be served with a near futures expiration unless you have a particularly good reason to do so.

I intend to roll out more functionality here to allow site owners to provide hints via data attributes in the script tags where scripts can be loaded asynchronously or deferred. I also intend to use some different approaches in the ways I discover changing content on the origin server and may be able to merge and minify near future expiring content. The fact is that most near future expiring content does not change often but wants to enforce that client’s check with the server frequently just in case there is a change.

Plugging in alternative minifiers and settings

RequestReduce attempts to follow a decoupled architecture which allows developers to swap out certain parts with their own behavior. To override RequestReduce's use of the Microsoft Ajax minifier library, you simply create a class that derives from IMinifier. There is not much to IMinifier:

public interface IMinifier{    string Minify<T>(string unMinifiedContent) where T : IResourceType;}

Here Is RequestReduce's implementation:

public class Minifier : IMinifier{    private readonly Microsoft.Ajax.Utilities.Minifier minifier = new Microsoft.Ajax.Utilities.Minifier();    private readonly CodeSettings settings = new CodeSettings {EvalTreatment = EvalTreatment.MakeAllSafe};

    public string Minify<T>(string unMinifiedContent) where T : IResourceType    {        if (typeof(T) == typeof(CssResource))            return minifier.MinifyStyleSheet(unMinifiedContent);        if (typeof(T) == typeof(JavaScriptResource))            return minifier.MinifyJavaScript(unMinifiedContent, settings);

        throw new ArgumentException("Cannot Minify Resources of unknown type", "unMinifiedContent");    }}

It's not difficult to imagine how you would change this implementation to use something like the YUI Compressor for.Net. Lets say you had a YuiMinifier class that you want RequestReduce to use instead of its own minifier. You would just need to add the following code to your startup code:

 
RRContainer.Current.Configure(x => x.For<IMinifier>().Use<YuiMinifier>());

That's it. Now your minification code will be used.

Suddenly CSS @import is not quite so evil

Just about anyone worth their salt in web development will tell you never to use CSS @Import. Except maybe the internet latency monster. He loves CSS @import but will quickly bite off your toes when he see you using it. Suffice it to say that the internet latency monster is a meanie.

RequestReduce can now identify @import usages and auto expand them so that no extra latency is incurred. I still do not think it is a good idea to use @import but now it lacks its sting.

So if you have not started using RequestReduce already, I strongly encourage you to do so. I would also greatly appreciate any feedback you have of what features you would like to see added or bugs you are running into. I’d also like to hear your success stories. You can submit all feature requests and bug reports to the github page at https://github.com/mwrock/RequestReduce/issues?sort=comments&direction=desc&state=open.

nQuant Reduces The Visual Studio Gallery and MSDN Code Samples page size down by 10% /  by Matt Wrock

Today the Microsoft Galleries team where I work and which supports the Visual Studio extensions gallery and the MSDN Code Samples gallery, among many others, began quantizing its sprited images with nQuant and has realized a 10% reduction in page size.

image

 

 

 

 

 

 

 

A few months ago, the visual studio gallery and the MSDN code samples gallery adopted my OSS project RequestReduce, which merges and minifies CSS as well as automatically sprites and optimizes background images. As I reported then, we experienced a 20% improvement in global page load times. At that time RequestReduce performed lossless PNG compression which can dramatically reduce the size of the sprited images which RequestReduce generates. I had also played with some “lossy” open source command line utilities that further reduced the size of PNG images – sometimes dramatically and often without perceptible quality loss. However, when I integrated these utilities into RequestReduce and applied it to some of the galleries that the Microsoft Galeries team develops (most notably the Silverlight and Azure galleries), the lossy optimization quality was simply unacceptable.

I did quite a bit of research on the topic of image quantization which is the process of removing colors from an image to produce a much smaller image while utilizing sophisticated algorithms to make this color loss imperceptible (or nearly imperceptible) to the human eye. It quite possibly may even be just as effective on alien eyes but to date, we lack the empirical evidence. You can count on me to update this post as more data accumulates in that exciting area of study.

While investigating this, I came across an algorithm developed by Xiaolin Wu that appeared to optimize RGB images (without transparency) with a quality unmatched by any other algorithm I had experimented with. Unfortunately, the algorithm was not immediately compatible with the transparent PNGs generated by RequestReduce. After several weeks of tinkering during very odd hours, I managed to adapt the algorithm to convert 32 bit transparent PNGs to 8 bit 256 color PNGs with far superior quality than those produced by many popular C command line tools. Furthermore, this is a C# library that can be easily integrated into any other .net assembly, nQuant also provides a command line wrapper which can be used for build tasks or ad hoc optimizations.

If you would like to see how nQuant can optimize images, head on over to nquant.codeplex.com where you can download either the compiled assembly and command line utility or the full source code. The site also provides complete instructions on the propper use of the nQuant API. It is dead simple. Here is an example of how to quantize a single image from within C#:

var quantizer = new WuQuantizer();using(var bitmap = new Bitmap(sourcePath)){    using(var quantized = quantizer.QuantizeImage(bitmap, alphaTransparency, alphaFader))    {        quantized.Save(targetPath, ImageFormat.Png);    }}

Using the command line, you would issue a command like this:

nQuant myimage.png /o mynewimage.png

If you would like to not only optimize your images but also minify and merge your CSS as well as sprite your CSS background images into a single sprite, then check out RequestReduce. Unlike other similar optimization tools, you do not need to change your code or rearrange your folder structure and you do not need to supply a wealth of config options to get started. It works with multiple server environments and supports content on a CDN. RequestReduce also includes nQuant which it uses to reduce the size of the images it produces.

For more details on the algorithm included in the nQuant library and my efforts to adapt it to ARGB transparent images, read my post on this subject.

www.orangelightning.co.uk makes 40% improvement in page load performance using RequestReduce /  by Matt Wrock

This week I worked with Phil Jones (@philjones88) of to get RequestReduce up and running on his web site hosted on AppHarbor. There was a couple issues specific to AppHarbor’s configuration that prevented RequestReduce’s default configuration from working. Its actually a fairly typical situation where their load balancers forward requests to their web servers on different ports. RequestReduce then assumes that the site is publicly accessible on this non standard port which it is not and things quickly begin to not work too well. In fact they did not work well at all. It was easy to work around this and by doing so, I was able to make my app all the more accessible.

So now that Phil has got orangelightning up and running on RequestReduce, their Google Page speed score went from 81 to 96 and their Yslow grade went from a low B at 83 to a solid A at 95.

image

image

 

Total number of HTTP requests were cut in half from 13 to 6 requests. And a page size of 93K to 54K.

And of course the bottom line is page load times. Using http://www.webpagetest.org, I tested from the Eastern United States (orangelightning  is in the UK) over three runs here are the median results:

With RequestReduce
image

 

Without RequestReduce
image

RequestReduce is free, Requires very little effort to install and supports both small blogs and large multi server, CDN based enterprises. You can download it from http://requestreduce.com/ or even easier, simply enter:

Install-Package RequestReduce

From the Nuget Packet Console right inside Visual Studio. Source code, wiki with thorough documentation and bug reporting is available from my github page at https://github.com/mwrock/RequestReduce.