RequestReduce

Released: RequestReduce 1.2 with Javascript merge and minify by Matt Wrock

With this week’s release of RequestReduce 1.2, RequestReduce is even more effective in reducing the HTTP requests made by a browser when visiting your website. Most significantly, this release adds the minification and merging of external javascript resources. A big thanks goes to Matt Manela for providing the initial Javascript Reduction code! Other mentionable features in this release include:

  • Automatic expansion of CSS @Import. All imports will be downloaded and injected into the calling CSS and then minified and merged with the rest of the page’s head CSS.
  • Wider support for PNG lossless compression. RequestReduce now issues the –fix parameter to optipng which will attempt to fix PNGs with extraneous data.
  • Ignore 404 and 500 failures of individual resources. In other words, if one script or stylesheet fails to crunch, the rest should still succeed.
  • Several bug fixes to the overall response filtering.

RequestReduce 1.2 is available via Nuget installations and from downloading it at http://www.RequestReduce.com. I will discuss the new features in some detail but first…

What makes RequestReduce the best static resource cruncher available?

OK. Glad I got that off my chest. Now for the details of this release’s features:

Javascript Crunching: What exactly is crunched

RequestReduce will minify all external javascripts that have a valid javascript mime type, do not have a Cache-Control header set to No-Cache or No-Store and does not have an Expires or max-age of less than a week. Furthermore you can explicitly tell RequestReduce what scripts you would like it to skip by supplying a comma separated list of url substrings. For example, on the Microsoft properties I work on, I have this set to /tiny_mce since the way that /tiny_mce loads its plugins does not play nice when the scripts are not in the same directory where tiny_mce expects them to be.

By default, if no urls are provided in this list, RequestReduce will ignore the Google and Microsoft JQuery CDNs. It does this since it is likely that browsers will not have to download this content because most will have already cached those resources while visiting a Google or Microsoft property in the past.

You can also use more complicated or customized rules via code. RequestReduce exposes a Func for approving the processing of urls. You can hook into this by registering a lambda or delegate to this Func in your Global.asax startup. Here is an example:

var javascriptResource = RRContainer.Current.GetInstance<JavaScriptResource>();javascript.TagValidator = ((tag, url) =>             {                if(url.EdsWith(".myjss") || tag.Contains("data-caching=dont"))                    return false;            });

The first input param is the full script tag element and the url is the javascript SRC.

Important considerations in the placement of your javascripts

In a perfect world, I would crunch all your scripts together and dynamically inject them into the DOM to load asynchronously. In fact, because the world is perfect, I could even defer the loading of the scripts until after document complete. But alas, the world is not perfect. RequestReduce knows absolutely nothing about your code and takes the safest approach to ensure that your page does not break.Therefore, RequestReduce will make sure that any dependencies that your markup has on your scripts or that your scripts have on other scripts are not broken.

RequestReduce will only merge blocks of scripts that can be reduced contiguously. For example assume you have the following scripts on a page:

<html>    <head>        <link href="http://localhost:8877/Styles/style1.css" rel="Stylesheet" type="text/css" />        <link href="/Styles/style2.css" rel="Stylesheet" type="text/css" />        <script src="http://localhost:8877/Scripts/script1.js" type="text/javascript"></script>        <script src="/Scripts/script2.js" type="text/javascript"></script>    </head>    <body>        <div class="RatingStar">&nbsp;</div>        <div class="Up">&nbsp;</div>        <div class="Rss">&nbsp;</div>        <div class="Search">&nbsp;</div>        <script src="http://localhost:8877/Scripts/script3.js" type="text/javascript"></script>        <script src="http://www.google-analytics.com/ga.js" type="text/javascript"></script>        <script src="/Scripts/script4.js" type="text/javascript"></script>    </body></html>

You will end up with five scripts:

  1. The scripts in the <head/> will be merged together and remain in the head.
  2. Script 3 will be minified but will not be merged since it lies just below common markup and above a near-future expiring script (google analytics).
  3. The google analytics script remains entirely untouched since it is a near future expiring script.
  4. Script 4 will be minified but not merged since it is not adjacent to any other non near futures script.

Although script three and four do not benefit from merging, they are at least minified and served with far future expiration and a valid ETag.

With all this in mind, you want to make sure that you group scripts together that are not near future expiring and can remain together without causing issues with load order dependencies. Also make sure that you do not have any setting that causes all of your javascript to be served with a near futures expiration unless you have a particularly good reason to do so.

I intend to roll out more functionality here to allow site owners to provide hints via data attributes in the script tags where scripts can be loaded asynchronously or deferred. I also intend to use some different approaches in the ways I discover changing content on the origin server and may be able to merge and minify near future expiring content. The fact is that most near future expiring content does not change often but wants to enforce that client’s check with the server frequently just in case there is a change.

Plugging in alternative minifiers and settings

RequestReduce attempts to follow a decoupled architecture which allows developers to swap out certain parts with their own behavior. To override RequestReduce's use of the Microsoft Ajax minifier library, you simply create a class that derives from IMinifier. There is not much to IMinifier:

public interface IMinifier{    string Minify<T>(string unMinifiedContent) where T : IResourceType;}

Here Is RequestReduce's implementation:

public class Minifier : IMinifier{    private readonly Microsoft.Ajax.Utilities.Minifier minifier = new Microsoft.Ajax.Utilities.Minifier();    private readonly CodeSettings settings = new CodeSettings {EvalTreatment = EvalTreatment.MakeAllSafe};

    public string Minify<T>(string unMinifiedContent) where T : IResourceType    {        if (typeof(T) == typeof(CssResource))            return minifier.MinifyStyleSheet(unMinifiedContent);        if (typeof(T) == typeof(JavaScriptResource))            return minifier.MinifyJavaScript(unMinifiedContent, settings);

        throw new ArgumentException("Cannot Minify Resources of unknown type", "unMinifiedContent");    }}

It's not difficult to imagine how you would change this implementation to use something like the YUI Compressor for.Net. Lets say you had a YuiMinifier class that you want RequestReduce to use instead of its own minifier. You would just need to add the following code to your startup code:

 
RRContainer.Current.Configure(x => x.For<IMinifier>().Use<YuiMinifier>());

That's it. Now your minification code will be used.

Suddenly CSS @import is not quite so evil

Just about anyone worth their salt in web development will tell you never to use CSS @Import. Except maybe the internet latency monster. He loves CSS @import but will quickly bite off your toes when he see you using it. Suffice it to say that the internet latency monster is a meanie.

RequestReduce can now identify @import usages and auto expand them so that no extra latency is incurred. I still do not think it is a good idea to use @import but now it lacks its sting.

So if you have not started using RequestReduce already, I strongly encourage you to do so. I would also greatly appreciate any feedback you have of what features you would like to see added or bugs you are running into. I’d also like to hear your success stories. You can submit all feature requests and bug reports to the github page at https://github.com/mwrock/RequestReduce/issues?sort=comments&direction=desc&state=open.

www.orangelightning.co.uk makes 40% improvement in page load performance using RequestReduce by Matt Wrock

This week I worked with Phil Jones (@philjones88) of to get RequestReduce up and running on his web site hosted on AppHarbor. There was a couple issues specific to AppHarbor’s configuration that prevented RequestReduce’s default configuration from working. Its actually a fairly typical situation where their load balancers forward requests to their web servers on different ports. RequestReduce then assumes that the site is publicly accessible on this non standard port which it is not and things quickly begin to not work too well. In fact they did not work well at all. It was easy to work around this and by doing so, I was able to make my app all the more accessible.

So now that Phil has got orangelightning up and running on RequestReduce, their Google Page speed score went from 81 to 96 and their Yslow grade went from a low B at 83 to a solid A at 95.

image

image

 

Total number of HTTP requests were cut in half from 13 to 6 requests. And a page size of 93K to 54K.

And of course the bottom line is page load times. Using http://www.webpagetest.org, I tested from the Eastern United States (orangelightning  is in the UK) over three runs here are the median results:

With RequestReduce

image

 

Without RequestReduce

image

RequestReduce is free, Requires very little effort to install and supports both small blogs and large multi server, CDN based enterprises. You can download it from http://requestreduce.com/ or even easier, simply enter:

Install-Package RequestReduce

From the Nuget Packet Console right inside Visual Studio. Source code, wiki with thorough documentation and bug reporting is available from my github page at https://github.com/mwrock/RequestReduce.

RequestReduce now fully compatible with AppHarbor and Medium Trust hosting environments. by Matt Wrock

Now even more sites can take advantage of automatic CSS merging and minification as well as image spriting and color optimization with no code changes or directory structure conventions.

This week I rolled out two key features which add compatibility to RequestReduce’s core functionality and some popular hosting environments. In a nutshell, here is what has been added:

  • Support for web server environments behind proxies. No extra configuration is needed. It just works.
  • Full support for AppHarbor applications. If you have not heard of AppHarbor, I strongly encourage you to check it out. It ties into your GIT repository and automatically builds and deploys your Visual Studio solution upon git push.
  • RequestReduce now runs in Medium Trust environments such as GoDaddy. There are some features that will not work here such as image color and compression optimizations and other multi server synchronization scenarios, but the core out of the box functionality of CSS merging, minification and on the fly background image spriting will work in these environments.

And as from the beginning, RequestReduce will run on ANY IIS hosted environment including ASP.NET Web Forms, all versions and view engines of MVC, Webmatrix “Web Pages” and even static html files.

So download the latest bits from www.RequestReduce.com or simply enter:

Install-Package RequestReduce

from the nuget power shell to get these features added to your site with no change to your code, almost no configuration and no rearranging of files and stylesheets into arbitrary folder conventions. As long as your background images are marked no-repeat and have explicit widths in their class properties, RequestReduce does all of the tedious work for you on the fly and makes sure that these resources have far future Expires headers and multi server friendly ETags allowing browsers to propperly cache your content.

 
Do you need multi server synchronization and CDN support? RequestReduce has got you covered.

Adopt RequestReduce and see immediate Yslow and Google Page Speed score improvements not to mention a faster site! by Matt Wrock

Since March I have been working in my “free” time on a framework to reduce the number and size of HTTP requests incurred from loading a web page. In short it merges and minifies css and javascript on your page and automatically sprites and optimizes css background images. All this is done on the fly (with caching) with no code changes or configuration required. All processed and reduced resources are served with far future caching headers and custom ETags.

In August I had something solid enough to push to production on Microsoft’s Gallery platform (what brings home my bacon) which hosts the Visual Studio Gallery, the MSDN Code Samples Gallery, the Technet Gallery and Script Center and many more galleries.

UPDATE: In November, RequestReduce was adopted by MSDN and Technet Forums and Search which serve millions of page views a day.

Results on Microsoft Gallery Platform

I’m very pleased with the results. We saw an 18% improvement in global page load times. We have a large international audience and the further you are from Washington state the more you will benefit from this improvement. VisualStudio Gallery raised its YSlow score from a B to an A and went from 41 HTTP requests to 30. Additionally, our workflow for spriting background images is completely automated.

Results from China Without RequestReduce: Results from China With RequestReduce:

ChinaNoRR

chinarr

 

Key RequestReduce WebPage Optimizations

RequestReduce will do the following on any page where the RequestReduce HttpModule is loaded:

  1. Look for background images that it can sprite. This is the process of combining multiple images into a single image and using some CSS syntax to pull in specific images from that single file into a CSS class’s background.
  2. Merge these images into a single PNG that is quantized down to 256 colors and then run through optipng for lossless compression. Unsatisfied with the quality I was getting from the popular opensource quantizers, I created a quantizer based on the Wu quantization algorithm and have released that separately as nQuant on codeplex. This often reduces the image size up to 3x smaller than the original.
  3. Merges all CSS in the head and minifies it. This includes any text/css resource so it includes files like WebResource.axd.
  4. Automatically expands CSS @imports.
  5. Minifies and merges all adjacent javascript on the page that do not have a nocache or no-store header and an expired or max-age less than a week. This includes any valid javascript mime type file so ScriptResource.axd and WebResource.axd are included.
  6. Manages the downloads of these CSS and image requests using ETag and expires headers to ensure optimal caching on the browser.

Other Great RequestReduce Features

Since I wanted to deploy RequestReduce on Microsoft websites, it obviously needed to scale to Millions of page views and be maintainable in an enterprise environment. To do this RequestReduce supports:

  1. CDN hosting of the CDN and Sprited images.
  2. Synchronizing generated CSS and image files across multiple servers via a Sql Server db or a distributed file replication system.
  3. Custom API allowing the addition of your own minifier or filtering out specific pages or resources.

Of coarse RequestReduce works perfectly on a small site or blog as well. This blog went from a YSlow C to an A after using RequestReduce.

Why I Created RequestReduce

First, I’ve been looking for an idea for quite some time to use for an Open Source project. This one struck me while on a run along the Sammamish River in February. Over the past 10 years I have worked on many large, high traffic websites that used somewhat complicated frameworks for organizing CSS. These often make including simple minification an impossible task in a build script especially if CSS can be changed out of band. Also, image spriting has always been difficult to keep up with. New images get rolled in to CSS and we are too busy getting features out the door; so spriting these images falls through the cracks. To have a process that did all of this automatically and at run time (Note: RequestReduce does not block requests while it does this. That would be a perf catastrophe. See here for details.) seemed ideal. I wanted a plug and play solution. Drop a dll in the bin directory and it just happens.

RequestReduce makes this vision come very close to reality. In this version, there are some things that RequestReduce expects of the CSS class containing the background image in order to successfully sprite it. In a future release I will be taking advantage of CSS3 which will mean RequestReduce will be able to sprite more images on modern browsers. The Microsoft Gallery sites have to support IE 7 and 8 so the first release had to be CSS2 compliant.

RequestReduce is now available for community use and contributions

To get started using RequestReduce:

  1. If you have Nuget, simply enter this command in the Package Manager Console and skip steps two and three:
    Install-Package RequestReduce

    Otherwise, download the latest RequestReduce version.

  2. Extract the contents of the downloaded zip and copy RequestReduce.dll as well as optipng.exe to your website's bin directory. (If for some reason you cannot put optipng.exe in your bin, RequestReduce will function as expected but will skip the lossless compression of its sprite images.
  3. Add the RequestReduceModule to your web.config or using the IIS GUI
  4. Optimize your CSS to help RequestReduce better locate your background images
  5. Optional: Configure RequestReduce. You can control where generated css and sprites are stored, their size thresholds and specify a CDN host name to reference You may also fork the RequestReduce source code from its github site.
      For links to RequestReduce documentation, bug reports and the latest download, you can visit

http://www.RequestReduce.com

    . I’d be very interested in hearing any feedback on the tool as well as any problems you have implementing it. I plan to be continually adding to this project with more features to reduce HTTP Requests from any web site.