Installing user gems using chef by Matt Wrock

In my experience installing server infrastructure using Chef, I usually use the chef_gem resource to install a gem that's needed in order to orchestrate the setup process. These are gems that are consumed by chef, or a chef resource to converge a node. However last night I was editing a chef recipe that's included in a chef_workstation cookbook that my team at CenturyLink Cloud uses for provisioning developer vagrant boxes and our TeamCity build agents. I have bloged about that in more detail here. The recipe I was working on is responsible for installing all of the gems we use in our chef dev process. They include both publicly available knife plugins and internally authored tools as well. These gems are consumed by the user of the vagrant box and not chef directly.

The bash force approach

This was one of the first cookbooks we created when we had limited knowledge of chef, ruby and how gems worked in general - alas our knowledge still has limits but they are much less restrictive. So this recipe looked something like this:

bash "install chef gems" do
  code <<-EOS
    su - #{node["chef_workstation"]["user"]} -c "chef gem install my-gem-1"
    su - #{node["chef_workstation"]["user"]} -c "chef gem install my-gem-2"
    su - #{node["chef_workstation"]["user"]} -c "chef gem install my-gem-3"

As you can see we were just running this via bash. Really this is fine and it worked. There is alot to be said for something that works. When we were putting this cookbook together we found that using chef_gem or gem_package installed the gems into the root user's directory making them inaccesible to the vagrant user. So using su was our workaround.

Recently we started using Nexus as an internal gem repository that seems to have slightly different install behavior from rubygems or artifactory. The former repositories would always install the latest gem assuming we had no constraints. Nexus would not install anything if any version of the requested gem was already installed. This did not work well for our internal gem workflow where we expect a vagrant provision to always install the latest gem.

Using gem_package

My first thought was to use the raw ruby gem modules to check for updates, install if the gem was missing or run an update if there was a newer version. That could have worked but it just seemed like I must be reinventing a wheel so I revisited the gem_package docs. Not sure why, but I didn't find anything meeting my needs on StackOverflow or the other google results I was turning up.

After reviewing the docs and a couple of initial failed attempts I landed on the right attribute values that yielded what worked:

%w[clc-gem1 clc-gem2 clc_gem-amazing].each do |gem|
  gem_package gem do
    source node["clc_nexus"]["repo"]["localgems"]
    gem_binary "/opt/chefdk/embedded/bin/gem"
    options "--no-user-install"
    action :upgrade

The key attributes here are gem_binary and options. Because I lean toward the idiot side of the intelligence spectrum, I had initially written off the gem_binary attribute thinking it was pointing to where to install binary gems. Nope, its intended to point to the location of the gem bin you want to use. Handy when you have multiple ruby installs. We use the chefdk on our vagrant boxes so that's where I point the gem_binary attribute.

The other not so obvious thing to add is the --no-user-install option. Since chef is running as root, if this is not specified, the gems are installed in /home/root. By specifying --no-user-install, the gems are installed in the shared ruby gems location. This may not be ideal and I'm sure there must be a way to get it in the vagrant user directory, but for the purposes of our vagrant environments, this works well.

Building, signing and deploying click-once and Azure web sites without Visual Studio installed by Matt Wrock

A few weeks ago, I repaved my windows laptop with the technical preview of windows 10. As I explained in last week's post, I have not been doing a whole lot of .net development these days and therefore have not been using Visual Studio much either. So I removed Visual Studio, Sql Server and a couple other heavy installs from my boxstarter script that I do not use. However, I do still actively maintain a project that is largely a collection of powershell modules but also includes a small click-once installer and a Razor based web site hosted on Azure that serves the documentation and latest download.

Both the click-once and web application are housed within Visual Studio projects and each include a .csproj file that drives the build. In the past I have always had Visual Studio installed and therefore had all the build tools on hand such as msbuild.exe, signtool.exe and web deploy that my build script could call to build and deploy the bits.

Well it was time to push out a maintenance release the other day and I wanted to do so without installing Visual Studio. Don't get me wrong, Visual Studio is an incredible tool but if you don't need it, why install it? Its quite large and my 128GB SSD is rather greedy. This is not only sound reasoning for a developer machine but it is even more applicable to a build server. I have known lots of folks who think they need to install Visual Studio on the build server in order to build their .net apps. Perhaps there are a couple edge cases where this is so but for most build and deploy tasks, the tools you need can be downloaded separately.

I'll cover the pieces I use to build and deploy Boxstarter here and walk you through the key build tasks. You can find the entire build script here which includes tasks for building and running Pester tests as well as the one task I run that deploys the packages to Chocolatey, creates a github release with zip file and deploys the web bits and click-once installer to Azure. Let me tell you it is very convenient to deploy everything with a single command. There is no database so its easy.

Compiling with msbuild

Both my click-once install and my web project need to be compiled to a .dll. This is done via msbuild. While msbuild does come included with visual studio, it can be downloaded separately with the Microsoft Build Tools. I get this and most other build prerequisites from Chocolatey:

task Install-MSBuild {
    if(!(Test-Path "$msbuildExe")) { cinst microsoft-build-tools }

task Build-Web -depends Install-MSBuild, Install-WebAppTargets {
    exec { .$msbuildExe "$baseDir\Web\Web.csproj" /t:Clean /v:minimal }
    exec { .$msbuildExe "$baseDir\Web\Web.csproj" /t:Build /v:minimal /p:DownloadNuGetExe="true" }
    copy-Item -Path "$baseDir\packages\bootstrap.3.0.2\content\*"`
      -Destination "$baseDir\Web" -Recurse -Force `
      -ErrorAction SilentlyContinue

Note that my build uses psake which compose the build from calls to a powershell function, Task, that define a script block to specify the code that the task actually runs. These tasks can optionally include dependencies which must be run prior to the task. Here my Build-Web task depends on Install-MSBuild to check and install the msbuild tools if they are not already installed.

Web application targets

Plain msbuild comes with most of the standard target files needed to build most visual studio project types. For whatever reason, web projects reference target files located in the visual studio install. However you can obtain these files by installing the MSBuild.Microsoft.VisualStudio.Web.targets nuget package. This check and potential install is handled in the Install-WebAppTargets task of my build script:

task Install-WebAppTargets {
    if(!(Test-Path "$env:ChocolateyInstall\lib\MSBuild.Microsoft.VisualStudio.Web.targets.12.0.4\tools\VSToolsPath\WebApplications\Microsoft.WebApplication.targets")) { 
        cinst MSBuild.Microsoft.VisualStudio.Web.targets -source -version '12.0.4'

I also edit the path to Microsoft.WebApplication.targets at the end of my web project's .csproj file to reference these targets from the chocolatey lib directory:

<Import Project="$(ChocolateyInstall)\lib\MSBuild.Microsoft.VisualStudio.Web.targets.12.0.4\tools\VSToolsPath\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''" />

Code signing the click-once payload

Boxstarter can be invoked from a URL which leverages "click-once" technology. Click-once apps must be signed with a valid code signing certificate. Although I invoke the click-once build with msbuild.exe:

task Build-ClickOnce -depends Install-MSBuild, Install-Win8SDK {
    Update-AssemblyInfoFiles $version $changeset
    exec { .$msbuildExe "$baseDir\Boxstarter.ClickOnce\Boxstarter.WebLaunch.csproj" /t:Clean /v:minimal }
    exec { .$msbuildExe "$baseDir\Boxstarter.ClickOnce\Boxstarter.WebLaunch.csproj" /t:Build /v:minimal }

Boxstarter.WebLaunch.csproj includes a post build step that signs the click once payload with my certificate:

<PostBuildEvent>"$(WindowsSDK80Path)bin\x64\signtool.exe" sign /n "Open Source Developer," "$(ProjectDir)obj\debug\Boxstarter.WebLaunch.exe"</PostBuildEvent>

Note that this assumes that I have imported a certificate into my user certificate store that has the subject name: "Open Source Developer," and that it includes the private key.

As you can see, signtool.exe lives in the Windows 8.1 SDK which is not included with the build tools that provide msbuild. You will need to install the windows 8.1 SDK available here. I install it via chocolatey like most others in polite society:

task Install-Win8SDK {
  if(!(Test-Path "$env:ProgramFiles\Windows Kits\8.1\bin\x64\signtool.exe")) {
    cinst windows-sdk-8.1 

Deploying to Azure Websites

When I initially created the web project in Visual Studio, I used to use the visual studio UI to deploy the files to my azure website. That works well but I wanted to include it as a step in my command line build. This is accomplished by calling msbuild with the correct properties describing the publish properties and password to use:

task Publish-Web -depends Install-MSBuild, Install-WebDeploy {
    exec { .$msbuildExe "$baseDir\Web\Web.csproj" /p:DeployOnBuild=true /p:PublishProfile="boxstarter - Web Deploy" /p:VisualStudioVersion=12.0 /p:Password=$env:boxstarter_publish_password }

The key properties here are the profile name and password. I keep my password in an environment variable so I don't store it in clear text on github. You can get both the profile name and password from the azure portal:

This downloads an xml file with both the name and password embedded inside.

In order for this to work without visual studio installed, you will need to install Web Deploy available here. Because I'm not a barbarian, I grab this from Chocolatey:

task Install-WebDeploy {
    if(!(Test-Path "$env:ProgramW6432\IIS\Microsoft Web Deploy V3")) { 
      cinst webdeploy 

Wrapping up with psake

As mentioned above, I use psake to drive the build and manage all of the task dependencies. If you don't care for programming in xml and especially if you are comfortable with powershell, you may find psake much more palatable than dressing up csproj and target files.

I've wrapped the call to invoke psake in a simple batch file, build.bat, so that the build can be invoked either from powershell or the Fisher-Price style command line CMD.exe making it compatible with any build server.

So simply typing build followed by the task name I want to run, kicks off my build:

build deploy # runs build, tests and packages everything for deployment
build Push-Public # deploys to chocolatey, github and azure

From full microsoft stack to cross platform development by Matt Wrock

I've been wanting to write about this for some time. About 8 months ago I not only left my software engineering position at Microsoft but I pivoted technology stacks all together and plunged into an entirely new and different technology ecosphere.  This was not a calculated move to abandon the Microsoft platforms (windows, .net, powershell, visual studio, etc, etc...) nor was it a reactionary vote of no confidence in the Microsoft based developer community. Over the past few years I have developed a growing passion and involvement in automation tooling. Over time this involvement consumed my "after hours" time and while it certainly occupied a fair amount of my work at Microsoft, I was presented with a super compelling opportunity at CenturyLink Cloud where I could cultivate this passion all day long (not JUST nights and weekends). Like many companies that are not Microsoft, it so happens that we use tools, infrastructure and development stacks that are not entirely Microsoft based.

Much of our back end runs on linux, there is no hyper-v here,  and our automation leverages Chef and is therefore saturated with ruby based tools. As a result, I have not written a line of C# or SQL in the last six months and I have opened visual studio no more than a handful of times. I read and write ruby everyday and have been contributing much of my spare time to ruby based open source projects like Vagrant, Test-Kitchen, and the ruby WinRM gem, my primary IDE is a text editor (sublime text 3), and my work funded laptop natively runs an Ubuntu 14.04 desktop. Let me tell you - this is a very different world and this post will attempt to describe some of the ways in which it is different from my experiences in the Microsoft world.

Again, I want to make it clear that I do not see this "pivot" as a conversion or even evolution to something better. While I do absolutely feel that some of the technologies I am working with are "better," there is alot that I miss and other areas that are just plain "grey." For example, I love just opening a text editor and writing code in a much simpler and snappier environment but there are moments that I yearn for some of the debugging and refactoring effeciencies that visual studio (and its plugins like resharper) provided. I'll probably dedicate a separate post to C# vs ruby but I still think C# is one kick ass language and while there are areas of ruby I have learned to love (and I am appreciating it more over time) I'd prefer a statically typed compiler aided language for not all but certainly my larger projects.

Differences are more cultural than technical

As I review the nuances that strike me as most profoundly different between the cross-platform world (this may not be the best term to use here but I'm sticking with it) and the MS-centric world, these tend to gravitate more strongly toward cultural differences than technical differences. Sure there are some fundamental distinctions to be found in the underlying architecture of linux and windows. If I were writing this just 5 years ago I'd be more tempted to give these differences more discussion but Microsoft has filled in some significant gaps and while its still "different," its really the culture that captures my attention.

What do I mean by culture? Well its just not interesting to compare and contrast these tools in terms of differing feature sets (but I will). Rather, there is a distinct and unique ethos in how these tools are composed and how their users value this composition and how they expect to interact with the tooling ecosystem that I find fascinating. Why is linux better than windows, why is C# better than GO, how do ruby gems differ from nuget packages? There is definitely some juice in all of these discussions but they will all eventually lead any observer of these debates to notice key character differences between these communities and what they value in the technology they build and consume.

So I am going to point out some of the characteristics of the cross platform community that I have found personally most notable. Look, this is a broad group and it is impossible to justifiably lump them into a cohesive community like I am doing here. More so today than ever there is growing "cross over" between this "cross-platform" community and the Microsoft stack developer communities. However, anyone who has spent any significant amount of time interacting with both camps will likely agree that the experience of interacting with one compared to the other evokes similar comparisons to groups that share different political or religious affiliations.

Open Source is normal

Yes...yes...I know Microsoft has recently made several announcements regarding the open sourcing of several significant formerly proprietary closed source technologies they own. What I think is key here is how in 2014, that is such a big deal. In so many pockets of the cross-platform world, its just how teams work. I personally share this value and I applaud those at Microsoft who have worked very hard to make this happen. But trust me, cultural wars have been raging inside the walls of Microsoft to push these projects to OSS and I assure you some of the coals are still quite hot.

Microsoft is in no way alone its struggle to open source its source code. Many, particularly larger, companies find themselves needing to wage internal campaigns to do the same. But my experience in the ruby community where I am most involved right now and its java, go and node cousins is that open sourcing and in many cases even your company's core IP is common place. Now I could go off on the virtues of why I think this is a wonderful thing and maybe, just maybe, unicorns and humans would peacefully coexist within the same halls if all code were open source, but instead I want to point out that differing attitudes of open source development reflect different mindsets and ways of viewing outside developers and development teams.

The open source adverse, may see OSS projects as the fruit of the "hobbyist" or "enthusiast" developer and not necessarily the product of highly skilled developer teams. This group may also see OSS as a threat either because it would expose "secret sauce" IP or sacrifice revenue opportunities because now the very thing you would charge for is freely available. I'm not "exposing" these views as unfounded at all. One has to walk carefully here but there are going to be distinct differences in the work habits and collaboration patterns of those who embrace open source and those that resist it or even just choose not to participate.

Personally my experience here has been largely positive. Even though someone else may be technically profiting from the contributions of my "work product," my contributions give me a voice in the direction of software that may not be "mine" but provides core services to my own product. It also gives full transparency into how a body of code works and exposes to myself and everyone else any key flaws that might exist in tools that I use and provide a way for me to either fix or work around problems.

Whether I pay for this software that I do not have time to write or not, the fact that I can read the source often proves itself to be invaluable to me in order to learn how to best use it or to solve blocking problems I might be facing as I use the software.

Now I will admit that there have been times of utter frustration where I just want the software to work as advertised and I'm pissed that I have to spend hours deep diving through someone else's code to figure out why I'm getting an UnspecifiedParameterException. But you know, I'd rather do that and learn some things in the process than spend the same amount of time on phone calls and emails with customer support. It also feels good to look at the code and see my github icon on the contributor list knowing that my fix is helping others perhaps in the same bind.

More granular projects and plugins over monoliths

 This is reflective all over in the *Nix universe. You tend to see lots of tools that can be composed alongside of one another and each has a bounded scope of responsibility. Overall this is definitely a software design principle I appreciate. Historically, larger software vendors may try to sell a "one tool does everything" sort of tool chain. Sure its nice to just have to buy one thing and deal with one vendor but this often also means "settling" for inferior functionality on some part if not all of the tool.

Not just at my current gig but in several other projects I have found that its usually more efficient in the end to get the best tool for the job even if that means having to license several tools.

The Chef ecosystem is a great example here. There are literally hundreds of different pieces one can compose to build an infrastructure automation strategy. You are not hand fed a single tool to orchestrate provisioning, installations, security, testing and reporting that includes compatibility with all hypervisors, clouds, containers and bare metal.

The result can be quite mixed. At first one is overwhelmed by choice. You spend alot of time understanding the benefits of different workflows and tool chains. However in the end, you gain a deeper understanding into how the tools fit together.

What I have also found here is that this granularity can make the individual pieces much easier to understand and troubleshoot. It means I may just need to grok a few dozen files to fully understand how the core testing infrastructure works without having to worry that some Digital Ocean based provisioner is coupled to this infrastructure in such a way that is gonna mess with my VMWare driven scenario.

Finding the right abstraction boundaries can be more art than science and can potentially clutter or over complicate an API but more often than not it guides towards better flexibility and composability. It also assumes that its consumers are more technically adept. A jr. engineer is not going to be able to wire everything up to take advantage of this flexibility. Again, this can be for better or for worse. I think it also explains cultural differences in the communities that consume these different kind of products.

Compare the consumer of a large chef or puppet based infrastructure to say a vmware vcac (or whatever they are calling this at the time you are reading this) or a Microsoft system center solution. The same analogy might be drawn comparing web forms vs MVC consumers of ASP.NET. I am not saying that one is better because it has more flexibility and you should therefore NOT use the other. Especially if you do not need the flexibility, there is alot to be said for buying the "packaged" deal if it meets your needs so that you can focus your expertise elsewhere. I am stating that in large part, I have found that the cross-platform world tends to steer towards more smaller, composable parts that take some effort and know how to wire/script together than a monolithic "out of the box" solution.

Less IDE-centric programming platforms

I've already spent some time on this topic above. Certainly ruby, go and node lend themselves toward a text editor over a full blown IDE for ones primary development environment. Until recently, it was simply not practical to write C# outside of Visual Studio and it has been a long time since I wrote java but I remember having the same dependence on an IDE when I was a java developer. 

Like everything else this has its upsides and downsides. You get lighter weight tools that provide 90% of the features you may actually use in an IDE. Many may find they do just fine without that extra 10%. Personally I find that there are certain times especially when debugging that I wish I had a richer debugging toolset. Actually these toolsets (like RubyMine) are available and may augment a text editor but can be tough to get used to them if you don't use them day in/day out. An IDE and "real" Intellisense can also be a real boon in discovering APIs without reading alot of source code. Then again, there is something to be said for reading source code. I have a better understanding for the API. Also, BONUS: as a new ruby developer, this is an excellent way to learn the idioms of a new language.

The command line is king

Historically products sold by Microsoft tend to be GUI heavy and in fact, its consumer base expect this. Often you get a v1 product with a GUI and need to wait for future versions for an API. The cross-platform world is filled with tools that have no GUI and will never have a GUI. Again, the consumer base expects this and values the command line.

Now I am the slowest typist I know and I say that, sadly, sans hyperbole. However I discovered years ago that I was more efficient with git on the commandline than using a GUI. For one thing I use source control all day, every day. I eventually found that using commandline tools even in the Microsoft world had a ton of value. Lots of Microsoft stack developers find this to be the case. However, I would not say this was nearly as prevalent among Microsoft stack developers largely because tooling simply did not lend itself to this. Just like it takes descent designers to write good GUIs, good command line tools also require a special skill.

Once you spend a certain amount of time immersed in the cross platform ecosystem, you learn some common patterns shared in most CLIs that allow you to navigate most others fairly quickly. The descent CLIs either have really good command line help or point to good READMEs on github or a product website. Note that there are some that are absolutely horrible. Oddly one of the most popular, git, does not rank highly in this regard. Others will scoff, but I think powershell is excellent in providing a discoverable command line interface with quality help. Its just too easy to get by in windows without ever opening a powershell window so many never learn it or use it consistently enough to pick it up.

More mature automation tooling

One thing that I and many of my colleagues note is that everything is harder and wrought with more friction when it comes to automating windows vs. linux. This has gotten much better since powershell and is getting rapidly better but this is still very much true today on the latest windows server OS (2012 R2 at the time of this post).

On linux, SSH is "just there" and their are no odd list of exceptions when it comes to running commands locally vs. remotely. Sure you can manage anything remotely on a windows machine but may have to jump through rings of fire to do so. However have you actually ever jumped through rings of fire? Neither have I but I have watched other do so and when they do they tend to stand tall afterwards with their arms up in the air and look extremely proud. So there is that...

Package management and configuration management have long been part of the linux lexicon but are novelties to windows. At least they are now novelties and not nonexistent. There are a rich set of automation tools in the linux world. Many, if not most of these are advertised to work with windows. However, there are often tears involved. I am confident Windows is going to catch up here and that the leaders behind Windows server want to do the right thing and understand the space.

I wrote an article for InfoQ a few months back describing the state of this tooling in the Windows world today. I'm really excited about where things are heading and have been an active participant in this space.

The gap is closing but there will always be a gap

I've hinted at this several times, but in areas where I think there are solid technical shortcomings, windows is catching up. I do think there will be a day for instance where the automation story will be at least very nearly as good as it is for linux. That said, I do think there will always be a cultural gap between these two platforms and their communities.

This is not necessarily a bad thing and I suppose that depends largely on your own developer ideologies and personal style. Even regardless of those, there will always be large pockets on either side that share very different values. personally, I enjoy both platforms for different reasons and I have good friends that develop for both stacks. Besides, what fun would the world be if windows and mac/linux users didn't argue? And in this world, how would you be able to identify the hipsters?

Accessing chef node attributes from kitchen tests by Matt Wrock

This should not happen often and maybe never, but there have been a few occasions where I needed the attribute values stored in the "node under test" to verify my kitchen test verifications. For example, maybe your recipe uses the node's IP or machine names as part of writing a config file and your kitchen driver uses dhcp to obtain an IP and auto generates unique machine names. In this case, you cant know the IP or machine name at the time you are authoring your test to check that the correct text was used in the config file.

This post demonstrates a very small and simple technique to retrieve this data inside of the test in order to dynamically verify the correct values were used in your recipe that you are testing.

At CenturyLink Cloud we use this in a few serverspec tests. One place we use this is in testing our haproxy configuration. Some values we need to inject into the configuration include the subnet of the current node and its chef environment which we include in the server names. Depending on where the kitchen test is being run, a different environment may be used so we cant be sure what the names of the subnet or chef environment will be while writing the test.

We solve this by adding a recipe to the end of our kitchen run list called "export-node":

  - name: my-suite
      - recipe[cookbook-under-test]
      - recipe[export-node]

This is a very simple recipe that simply converts the current node's attributes to json and writes it to a known location in /tmp/kitchen on the node.

ruby_block "Save node attributes" do
  block do
    if Dir::exist?('/tmp/kitchen')
      IO.write("/tmp/kitchen/chef_node.json", node.to_json)

Then our test can read that file and parse its json to a hash to be used throughout the test.

require 'json'
require 'serverspec'

set :backend, :exec

describe file('/etc/haproxy/haproxy.cfg') do
  let(:node) { JSON.parse('/tmp/kitchen/chef_node.json')) }
  let(:subnet) {
    ip = node["automatic"]["ipaddress"]
  let(:env) { node["chef_environment"].upcase}

  it { should be_a_file }
  its(:content) {
    should match <<-EOS
      backend elasticsearch_backend
        mode http
        balance roundrobin
        server #{env}SRCH01 #{subnet}.121:92 weight 1 check port 92
        server #{env}SRCH02 #{subnet}.122:92 weight 1 check port 92
        server #{env}SRCH03 #{subnet}.123:92 weight 1 check port 92

      backend web_backend
        mode http
        balance roundrobin
        timeout server 5m
        server #{env}WEB01 #{subnet}.131:80 weight 1 check port 80
        server #{env}WEB02 #{subnet}.132:80 weight 1 check port 80

      backend rabbit_backend
        mode http
        balance roundrobin
        server #{env}RABBIT01 #{subnet}.141:1567 weight 1 check port 1567
        server #{env}RABBIT02 #{subnet}.142:1567 weight 1 check port 1567
        server #{env}RABBIT03 #{subnet}.143:1567 weight 1 check port 1567

Above is a serverspec test that parses the node json to a hash. That hash is then accessible to our it blocks.

I have extracted the handful of lines in the export-node recipe to its own cookbook so you can grab it here.

Windows containers - package your apps and bootstrap your chef nodes with! by Matt Wrock

I discovered a few months ago from some comment on twitter. They claimed to be bringing containers to windows. Not the ability to run linux containers on windows but actual windows containers. This perked my interest and I immediately started hacking. The technology was impressive. Creating and running these containers have many similarities to workflows I'm familiar with on docker. The file system and windows registry inside these containers are truly isolated and you can layer spoon images on top of each other as part of the container packaging process.

Despite these cool features, there were some gaps that limited their usability for my scenarios. The winsxs (god bless it) was not entirely isolated limiting MSI installations and many windows features available via server manager and DISM did not fully work in isolation. I totally respect the fact that this is not an easy problem to solve.

I've kept in touch with Kenji Obata, CEO and founder of Spoon who has been filling me in on the MSI support. Being able to isolate MSI installs is key and the team at spoon has been making lots of headway making this work. This week my daughter asked me to build a minecraft server and I thought this would be a great opportunity to play with spoon to see where it has come. Running a minecraft server is fairly straight forward. You need to install the java runtime, which installs as an MSI, and then invoke java.exe to run the minecraft server .jar which starts the process running the minecraft server. The minecraft server listens for TCP requests on a specified port to connect players with game activity.

I was not only able to create a container running a minecraft server, but was later able to create an empty container and bootstrap it as a chef node then let chef install java and the minecraft jar...and then a second container running a separate instance.

This post is going to walk you through

  1. How to package an application inside a spoon windows container.
  2. Create a base container connectable via SSH
  3. Bootstrap containers joining them to a hosted chef server

I'l be using the minecraft server as the example app because minecraft is cool. This is gonna be a whirl wind tour where we will use several popular "devops" style apps to interact with our windows containers including vagrant, chocolatey, and chef. If you would like to follow along, I am demonstrating all of this using a windows server 2012 R2 vagrant box as the container host. This box is publicly available from vagrant cloud:

vagrant init mwrock/Windows2012R2

Its the same minimal windows box described here a few weeks ago.

Getting and using spoon

Just like you need to install the docker engine to run docker containers, you need to install the spoon plugin on a host system where you intend to run spoon containers. You can download this from Lets install this on our vagrant vm now from a powershell console:

Invoke-WebRequest -Uri `
  -OutFile spoon-plugin.exe

You will need to open a new console for the changes to your path to take effect. Next you need to sign in to spoon. So make sure to sign up for a free account first. I'm now gonna sign in to the spoon CLI:

spoon login mwrockx

Now we are ready to interact with the spoon CLI, creating and running containers.

Containerizing a windows based Minecraft server

There are a few ways to containerize an app in spoon. All of these are well documented on the spoon documentation site. One way is to start with a "clean" image or a collection of base images that have already been created and have components required by your app. Then manually install and configure the rest of your application and finally commit the image. All containers are created from an image. Think of the "image" as a single snapshot that holds an app setup and configuration in a pristine state. One then "runs" containers from that image. Containers are the isolated environment actually running the application.

Since minecraft requires the java runtime and there is a oracle/jre spoon image, that's a great place to start the packaging. So we'll run the java spoon image:

PS C:\Users\Administrator> spoon run oracle/jre
Using VM 11.6.381 from local
Using jre from local
Using clean:4.0 from local
Running container 4ef7018d with visibility public (use `--private` for a private container)

Spoon just launched a new command window running inside of a new container that has java installed. See:

(4ef7018d) C:\Users\Administrator>echo %java_home%
C:\Program Files (x86)\java\jre7

(4ef7018d) C:\Users\Administrator>

Our java_home environment variable is set here. We won't see this variable in our host vm outside the container (assuming we have not installed java). Also note the hash visible at every prompt inside a container. This helps us to identify which container we are inside of. We can list all containers on the host with:

PS C:\Users\Administrator> spoon containers
ID        Images                              Command  Created                 Status   Visibility
--        ------                              -------  -------                 ------   ----------
4ef7018d  spoonbrew/clean:4.0,oracle/jre               11/30/2014 9:38:42 PM   Running  Public
c431ad42  spoonbrew/clean:4.0,freeSSHd:1.2.6           11/30/2014 12:05:16 PM  Running  Public
605f8124  spoonbrew/clean:4.0,freeSSHd:1.2.6           11/30/2014 3:11:20 AM   Running  Public

You'll see I am also running a couple of containers from a freeSSHd image. We'll get to those later. Now lets get the minecraft server .jar file. There is a different .jar for each minecraft version. I run 1.7.2 because that is the version compatible with all the minecraft mods my daughters run.

mkdir %programdata%\spoon
powershell -command wget -outfile $env:programdata\spoon\minecraft_server.jar

Great. We have all the bits we need. So now we just need to kick off the server:

(4ef7018d) C:\Users\Administrator>"%java_home%\bin\java" -Xmx1024M -Xms1024M -ja
r %programdata%\spoon\minecraft_server.jar nogui
[22:01:33] [Server thread/INFO]: Starting minecraft server version 1.7.2
[22:01:33] [Server thread/INFO]: Loading properties
[22:01:33] [Server thread/WARN]: does not exist
[22:01:33] [Server thread/INFO]: Generating new properties file
[22:01:33] [Server thread/INFO]: Default game type: SURVIVAL
[22:01:33] [Server thread/INFO]: Generating keypair
[22:01:33] [Server thread/INFO]: Starting Minecraft server on *:25565
[22:01:34] [Server thread/WARN]: Failed to load operators list:
oundException: .\ops.txt (The system cannot find the file specified)
[22:01:34] [Server thread/WARN]: Failed to load white-list:
Exception: .\white-list.txt (The system cannot find the file specified)
[22:01:34] [Server thread/INFO]: Preparing level "world"
[22:01:34] [Server thread/INFO]: Preparing start region for level 0
[22:01:35] [Server thread/INFO]: Preparing spawn area: 10%
[22:01:36] [Server thread/INFO]: Preparing spawn area: 19%
[22:01:37] [Server thread/INFO]: Preparing spawn area: 29%
[22:01:38] [Server thread/INFO]: Preparing spawn area: 39%
[22:01:39] [Server thread/INFO]: Preparing spawn area: 49%
[22:01:40] [Server thread/INFO]: Preparing spawn area: 59%
[22:01:41] [Server thread/INFO]: Preparing spawn area: 67%
[22:01:42] [Server thread/INFO]: Preparing spawn area: 80%
[22:01:43] [Server thread/INFO]: Preparing spawn area: 92%
[22:01:44] [Server thread/INFO]: Done (10.273s)! For help, type "help" or "?"

Our minecraft sever is now up and running. Lets try to find the java process running from the host vm:

PS C:\Users\Administrator> ps -Name java | select path

C:\Program Files (x86)\java\jre7\bin\java.exe

You might be wondering why java is running from that path. Afterall, this java is supposed to be isolated to our container and not hanging out in our Program Files. So we sanity check with:

PS C:\Users\Administrator> Test-Path (ps -Name java | select path)

Hmm. Its not really there. Where is it then? Pop open the windows Task Manager and look at the details tab. Notice the "Image Path Name" of java.exe. Here we find the true path:


We can also use the spoon CLI to inspect the processes running in our containers:

PS C:\Users\Administrator> spoon ps
PID   Name      Container  User
---   ----      ---------  ----
2968  java.exe  4ef7018d   vagrant
4196  cmd.exe   4ef7018d   vagrant

We should also see this java process listening on port 25565:

PS C:\Users\Administrator> netstat -a -o | select-string "Listening" | select-string "2968"

  TCP          WIN-OHS0CTBB1TK:0      LISTENING       2968
  TCP    [::]:25565             WIN-OHS0CTBB1TK:0      LISTENING       2968

OK everything looks good and we should be able to run a minecraft client against our new server connected on that port. However, we are gonna want to stop the container before saving it as a new minecraft server spoon image. That means that any time I want to run a container off of the new image in the future, I'm going to have to make that same call to java.exe to launch the server. Wouldn't it be nice if that call was simply wrapped into the container startup? We can do that by building our image with a file.

Automating spoon image builds with a file

If you are used to docker, a is the moral equivalent of a docker file. Spoon provides a small collection of scripting primitives that can be used to automate the creation of an image. This exposes some nice functionality like the "startup file" command that allows us to run a command whenever the container starts up. This also allows us to easily repeat the creation of an image and version them in source control.

Lets blow away our container where we created our minecraft server:

PS C:\Users\Administrator> spoon rm 4ef7018d
Container 4ef7018d has been removed

Now lets create a file called with the following content:

from oracle/jre

mkdir %programdata%\spoon

cmd powershell -command wget -outfile $env:programdata\spoon\minecraft_server.jar

startup file ("c:\Program Files (x86)\java\jre7\bin\java", "-Xmx1024M", "-Xms1024M", "-jar", "c:\programdata\spoon\minecraft_server.jar", "nogui")

This script will pull down the java image, download the minecraft server jar and finally ensure that any time the image is run, the minecraft jar will be launched. Lets build our image:

spoon build -n=minecraft-server:1.7.2

We now have an image that we can run with:

spoon run minecraft-server

Note how fast these containers are created from our built image. Sure is faster than downloading and installing java.

Lets take our container for a spin and run a minecraft client. First we will need to open up port 25565 on the container host:

New-NetFirewallRule -DisplayName "Minecraft" `
                    -Direction Inbount –LocalPort 25565 `
                    -Protocol TCP -Action Allow

I'm gonna assume you know how to run a minecraft client and point it at a server. All you need to know is the host IP address of your container host.

Bootstrapping and building spoon containers with chef

So I was super curious to see if I could integrate a spoon container into a chef workflow. Chef is a popular configuration management platform for building and maintaining infrastructure. In my mind there are two huge use cases for chef/container interaction:

  1. Using chef instead of (or to supplement) a file. The chef ecosystem provides a rich set of resources just as capable of building a container image as a vm.
  2. Using containers to test chef recipes that will eventually be used in a vm or bare metal. As you have probably noticed, it takes much (MUCH) less time to start a container than a fresh vm.

Working with a chef server

Sure we could just use chef-zero/chef-solo to run a recipe locally, but lets try to actually perform a remote bootstrap of a container node against a serer to get the full effect here. We could use any server. Here, I'm using a hosted chef instance that I sometimes use for personal testing. Anyone can sign up for a hosted chef account giving you access to your own publicly available chef server endpoint.

See this post for a brief run through on getting started with hosted chef. I'm not going to get into the details of setting up a chef server here but you will need:

  • you user .pem
  • your organization validator.pem
  • a knife.rb config file

Knife windows

We are going to run knife bootstrap using the knife windows gem. If you have installed chef using the ChefDK, you should have this. Otherwise you will need to run:

gem install knife-windows

You could run this directly from the container host vm. I'm just going to run it from my local os that has network access to my vm.

A minecraft cookbook with windows support

I could not find a chef cookbook for a minecraft server that runs on windows. So I wrote one and it can be found on github where you can view or clone the full cookbook. Lets take a look at the default recipe (the only recipe):

include_recipe 'chocolatey'
chocolatey 'javaruntime'

directory "#{ENV['programdata']}/minecraft"

remote_file "#{ENV['programdata']}/minecraft/minecraft_server.jar" do
  source "{node['minecraft']['version']}/minecraft_server.#{node['minecraft']['version']}.jar"

cookbook_file "minecraft.bat" do
  path "#{ENV['programdata']}/minecraft/minecraft.bat"

batch "start minecraft" do
  code <<-EOH
    powershell -command "start #{ENV['programdata']}/minecraft/minecraft.bat"
  action :run

A couple things worth noting:

  1. It uses chocolatey to install java. The java community cookbook that most use requires the java MSI to be internally hosted for windows platforms. I didn't want to bother with that and I know there is a chocolatey package that will do this job just fine. For those unfamiliar with chocolatey, it is a package management solution for windows similar to brew, yum or apt-get on mac/linux.
  2. I store the call to java.exe in a cookbook file, minecraft.bat, and use the chef batch resource to invoke that file via powershell Start-Process so that it does not hang the chef client run because that batch file will not return.

Upload the cookbooks to the chef server

You will need to upload the above mentioned minecraft cookbook as well as its dependencies which include:

  • chef-handler
  • windows
  • chocolatey

All of these dependencies are on the chef supermarket. You can use berkshelf to easily download all dependencies. Again if you use the ChefDK, you have this already, otherwise install the berkshelf gem. Using the berkshelf vendor command you can grab all dependencies and save them all to a "vendor path" you specify:

berks vendor [vendor path]

Make sure your cookbook_path in your knife.rb includes the vendor path otherwise knife upload will not find them.

Establishing SSH connectivity with your containers

Currently spoon containers do not play nice with winrm and cant isolate winrm to specific containers. Spoon is aware of this and its on their radar. However, ssh works just fine and you can remotely manage your containers as long as they have an ssh server running on the container.

I have created a spoon image that runs freeSSHd. This will be the base container we will use to bootstrap with chef. You can go ahead and start up an ssh enabled container with:

spoon run -d mwrockx/freeSSHd

The -d tells spoon to run the container in the background. Note that if you do not already have this image locally (you likely don't) then spoon will download it from the public spoon hub. Now here is another cool thing about containers. Run:

spoon images

to list the images locally stored on your container host. Take a look at the size of this freeSSHd image. Its 3.7MB. That is much better than 10+GB that most windows VMs consume.

If you will be accessing the containers outside of the container host VM, you will need to allow port 22 traffic on the windows firewall:

New-NetFirewallRule -DisplayName "SSH" `
                    -Direction Inbount –LocalPort 22 `
                    -Protocol TCP -Action Allow

I built this image giving both the Administrator and the vagrant users SSH access. Once the container is running, you can edit this in the freeSSHd settings by right clicking the desktop tray icon.

I'm assuming that you have access to an ssh client. I use the one that installs with msysgit located at C:\Program Files (x86)\git\bin\ssh.exe by default. Lets ssh to our container just to verify that everything works:

Yup. Looks good and the hash in the prompt correlates with my container id. 

One oddity that I noticed when using ssh is that upon succesfully connecting, the ssh prompt appears at the top of the console in the midst of whatever text was already there. I'm not sure why this happens but find that calling cls to clear the screen before beginning my ssh session helps.

Bootstrap the container

ok. Enough yaking. Lets install a minecraft server with chef on a windows container:

knife bootstrap windows ssh -P vagrant -x vagrant -r 'recipe[minecraft]' -N mc1

This will install the chef client on our container, add the container as a chef node on the server, download the minecraft cookbook and dependencies from the chef server and run the recipes to install java, the minecraft server jar file and launch the server. I use the default vagrant credentials which uses 'vagrant' for both user name and password. This will likely take a few minutes. The longest part is the download of the chef client.

I also see the node in my hosted chef organization:

Bootstrapping a second node

So just to be fancy, lets bootstrap a second node.

Networking considerations

We will have two containers running ssh and minecraft on what THEY THINK are ports 22 and 25565, we will need to setup proper NAT rules that redirect different ports coming into the container host to these ports on the containers. Spoon provides mechanisms to accomplish this. We will have our second container route ports 2222 and 35565 to 22 and 25565 respectfully. Spoon provides lots of options for mapping ports and DNS as well as linking containers to one another. See the spoon documentation for those details.

So first step we need to take is allow ports 2222 and 35565 on the container host:

Get-NetFirewallPortFilter | ? { 
  $_.LocalPort -eq 22 
} | Set-NetFirewallPortFilter -LocalPort @(22,2222)
Get-NetFirewallPortFilter | ? { 
  $_.LocalPort -eq 25565 
} | Set-NetFirewallPortFilter -LocalPort @(25565,35565)

Now lets run our second container informing spoon of the desired routes:

PS C:\Users\Administrator> spoon run -d --route-add=2222:22 --route-add=35565:25565 freeSSHd
Using VM 11.6.381 from local
Using freeSSHd:1.2.6 from local
Using clean:4.0 from local
Running container 9a3b62af with visibility public (use `--private` for a private container)

Just to confirm that we can ssh to the new container on port 2222:

Microsoft Windows [Version 6.3.9600]master]> ssh vagrant@ -p 2222
(c) 2013 Microsoft Corporation. All rights reserved.

(9a3b62af) C:\Users\Administrator>

Bootstrapping the second container

We are now ready to join the new container to the chef server and install the minecraft server recipe:

knife bootstrap windows ssh -P vagrant -x vagrant -r 'recipe[minecraft]' -N mc2 -p 2222

Now I can see 2 nodes on the hosted server instance:

Once a node is joined to the server, we can rerun the recipes locally using chef-client. No need to bootstrap again. Lets change one of the nodes to run a different version of minecraft. We'll change the mc2 node to run 1.7.9:

Now we'll ssh to mc2 and reconverge it with the new version:

Running a minecraft client pointed to both containers, we can see the version mismatch on the second node.

Diffing containers

Here is a feature I find really clever - the ability to see how a container differs from the host. Spoon can list all files and registry keys that have changed:

PS C:\Users\Administrator> spoon run -d freeSSHd
Using VM 11.6.381 from local
Using freeSSHd:1.2.6 from local
Using clean:4.0 from local
Running container de8d95f9 with visibility public (use `--private` for a private container)
PS C:\Users\Administrator> spoon diff de8d95f9
Using clean:4.0 from local
Using freeSSHd:1.2.6 from local
File system changes:
C C:\Program Files (x86)
C C:\Program Files (x86)\freeSSHd
C C:\Program Files (x86)\freeSSHd\FreeSSHdService.ini
C C:\ProgramData
C C:\ProgramData\Microsoft
C C:\ProgramData\Microsoft\Windows
C C:\ProgramData\Microsoft\Windows\Caches
C C:\ProgramData\Microsoft\Windows\Caches\{6AF0698E-D558-4F6E-9B3C-3716689AF493}.2.ver0x0000000000000006.db
C C:\ProgramData\Microsoft\Windows\Caches\{8290F723-67BF-4FDA-BEEA-B0AF3ADC7010}.2.ver0x0000000000000004.db
C C:\ProgramData\Microsoft\Windows\Caches\{DDF571F2-BE98-426D-8288-1A9A39C3FDA2}.2.ver0x0000000000000004.db
C C:\ProgramData\Microsoft\Windows\Caches\cversions.2.db
C C:\Users\Administrator\AppData\Local
C C:\Users\Administrator\AppData\Local\Microsoft
C C:\Users\Administrator\AppData\Local\Microsoft\Windows
A C:\Users\Administrator\AppData\Local\Microsoft\Windows\INetCache
A C:\Users\Administrator\AppData\Local\Microsoft\Windows\INetCache\counters.dat
C C:\Users\Administrator\AppData\Local\temp
A C:\Users\Administrator\AppData\Local\temp\wodCmdTerm.exe

Registry changes:
C @HKCU@\Software
C @HKCU@\Software\Microsoft
C @HKCU@\Software\Microsoft\Windows
C @HKCU@\Software\Microsoft\Windows\CurrentVersion
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer\SessionInfo
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer\SessionInfo\2
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Content
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Content\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Cookies
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Cookies\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\History
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\History\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\SavedLegacySettings
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyEnable
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\AutoDetect
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\IntranetName
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\ProxyBypass
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\UNCAsIntranet
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\OnDemandInterfaceCache
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion
A @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings
A @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
C @HKLM@\System
C @HKLM@\System\CurrentControlSet
A @HKLM@\System\CurrentControlSet\Services
A @HKLM@\System\CurrentControlSet\Services\Winsock
A @HKLM@\System\CurrentControlSet\Services\Winsock\Parameters
A @HKLM@\System\CurrentControlSet\Services\WinSock2
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\AppId_Catalog
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\NameSpace_Catalog5
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9

This can come in handy for all sorts of debugging scenarios where you need to track down changes impacted by an individual install.

Are we living with rainbows and unicorns?

I'm not quite ready to confirm that but I do feel like I just got extra sprinkles added to my ice cream. There is still much to be done to fill in some isolation gaps and provide integration and orchestration tooling.

There is incredible potential here that can be taken advantage of today. Things are going to be very interesting once Microsoft reveals more details around its own container strategy and partnership with docker. However, so far that story has been very foggy and its unclear exactly what they will deliver in the next server edition. Likely many will simply wait and see what emerges there, but I think spoon has made some great strides forward here and deserves a close look for its potential as a serious player in the windows container space.