Update: see this post for the latest update on getting up and running with Test-Kitchen on windows.
Linux automation testing has been supported for a while now using many great tools like chef, puppet, Test-Kitchen, ServerSpec, MiniTest, Bats, Vagrant, etc. If you were willing to install an SSH server on Windows, you could get most of these tools to work but if you wanted to stay “native” you were on your own.
Pictured above: Testing node convergence on an 8 inch tablet.
I’m not at all morally opposed to installing SSH on windows. I love SSH. We spoon regularly. But while SSH is “just there” on linux, it incurs an extra install step for windows that must either be done manually or included in initial provisioning or image creation. Also, for some windows-only shops, the unfamiliarity of SSH may add a layer of unwanted friction in an automation ecosystem where windows is often an after thought.
Well recent efforts to make Windows testing a first class experience are beginning to take shape. Its still early days and some of the bits are not yet “officially” released but are available to use by pulling the latest bits from source control. I know…I know…to many that will still spell “friction” in bold. However, I want to share that one can today test windows machine builds via winrm with no SSH server installed, and I also want to offer a glimpse to those who prefer to wait until everything is fully baked of what is to come, and inform you that the wheels are in motion so please keep abreast of these developments.
Note: I presented much of this material and several Boxstarter demos to the Philadelphia PowerShell User Group last week, the video is available here.
Its not automated until it is tested
I work for CenturyLink Cloud and infrastructure automation is front and center to our business. Like many shops, we have a mixed environment and central to our principals is the belief that testing our automation is just as important as building our automation. In fact they are not even two separate concepts. Untested automation is not finished being built. So I am going to share with you here how we test our Windows server infrastructure along with some other bits I have been working with on the side.
Vagrant
If you have not heard of Vagrant, just stop reading right now and mosey on over to http://vagrantup.com. Vagrant is a hypervisor agnostic way of spinning up and provisioning servers that is particularly suited for developing and testing. It completely abstracts both the VM infrastructure as well as many possible provisioning systems (chef, puppet, plain shell scripts, docker and many many more) so that one can provision and share the same machine among a team using different platforms.
To illustrate the usefulness here, where I work we have a diverse team where some prefer MACs, others work on Windows and others (like myself) run a Linux desktop. We use Chef to automate our infrastructure and anyone who needs to create or edit chef artifacts needs all sorts of dependencies installed with specific versions in order to be successful. Vagrant plays a key role here. Anyone can download our Ubuntu 12.04 base image via VirtualBox, VMWare or Hyper-V and then use its Chef provisioner plugin to build that image to a state that mirrors the one used by the entire team. all this is done by including a small file of metadata that serves as a pointer to here the base images can be found as well as the chef recipes. If this sounds interesting, again I refer you to Vagrant’s documentation for the details What I want to point out here is its windows support.
Added support for WinRM and Hyper-V
Until fairly recently, Vagrant only supported SSH as a transport mechanism to provision a VM. It also lacked official Hyper-V support as a VM provider. This changed in version 1.6 with a WinRM “Communicator” and a Hyper-V provider plugin included in the box. While I don’t really use Hyper-V at work, I have some windows based personal projects at home and I prefer to use Hyper-V. So I quickly tested out this new plugin and was happy to see it available. There are still some kinks in the current version but work is underway to improve the experience. I’m trying to to personally contribute to issues that are blocking my own work and a couple have been accepted into Vagrant Master. Overall that has been a lot of fun. Here are the issues that have come up for me:
- Only .vhdx image files are supported and .vhd files cannot be imported. I hit a wall with this when trying to use the .vhd files freely available for testing here on Technet. I have since added a patch which has been accepted to fix this.
- Generation 2 Hyper-V VMs are imported as Generation 1 VMs and fail to boot. Oddly, most .vhdx images tend to be generation 2. My PR for this issue was just accepted yesterday.
- Synced folders over SMB (this is the norm for a windows host/windows guest setup) fail. I’m hoping my PR for this issue is accepted.
If these same issues become blockers for you, the first two can be immediately fixed by pulling the latest copy of Vagrant’s master branch and copying the lib and plugin directories onto the installed version and you are welcome to pull my smb_sync branch which includes all of the fixes:
git clone -b smb_sync https://github.com/mwrock/vagrant copy-item -path vagrant\lib ` C:\HashiCorp\Vagrant\embedded\gems\gems\vagrant-1.6.3 ` -recurse -force copy-item -path vagrant\plugins ` C:\HashiCorp\Vagrant\embedded\gems\gems\vagrant-1.6.3 ` -recurse -force
Having worked with Vagrant for the past few months, I’ve been finding myself wishing there was a remote powershell equivilent to the vagrant SSH command which drops you into an ssh session on the guest box. So today I banged out a first draft of a vagrant ps command that does just that and will submit once it is more polished. You can expect it to look like this:
C:\dev\vagrant\win8.1x64> vagrant ps default: Creating powershell session to 192.168.1.14:5985 default: Username: vagrant [192.168.1.14]: PS C:\Users\vagrant\Documents>
A base box for testing
I’ve been playing with creating windows vagrant boxes. Unfortunately for Hyper-V, the vagrant package command is not yet implemented so I have to “manually” create the base box. Perhaps I’ll work on an implementation for my next contribution. My Windows 2012R2 Hyper-V box requires all the above fixes to install without error. You could use this Vagrantfile to test:
# -*- mode: ruby -*- # vi: set ft=ruby : VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "mwrock/Windows2012R2" config.vm.box_url = "https://vagrantcloud.com/mwrock/Windows2012R2/version/1/provider/hyperv.box" # Change "." below with your own folder you would like to sync config.vm.synced_folder ".", "/chocolateypackages", disabled: true config.vm.guest = :windows config.vm.communicator = "winrm" config.winrm.username = "administrator" config.winrm.password = "Pass@word1" end
Note here that you need to specify :windows as the guest. Vagrant will not infer that on its own nor will it assume you are using winrm if you are using a windows guest so make sure to add that to your boxes as well if you intend to use winrm.
Test-Kitchen
Test-Kitchen is a testing framework most often used for testing Chef recipes (hence – kitchen). However I understand it is also compatible with Puppet as well. Like many tools in this space such as Vagrant above, it is highly plugin driven. Test-Kitchen by itself doesn’t really do much. What Test-Kitchen brings to the table (Ha Ha! I said table. get it?) is the ability to bring together a provisioning configuration management system like Chef and Puppet, a myriad of different cloud and hypervisor platforms and several testing frameworks. In the end it will spin up a machine, run your provisioning code and then run your tests. Further you can integrate this in your builds providing quick feedback as to the quality of your automation upon committing changes.
“Official” support for windows guests coming soon
Currently the “official” release of Test-Kitchen does not support winrm and must go through SSH on windows. However, Salim Afiune (@afiune), a developer with Chef has been working on adding winrm support. I have plumbed this into our Windows testing at CenturyLink cloud and have also used it developing my Boxstarter cookbook, which allows one to embed boxstarter based powershell in a recipe and provides all the reboot resiliency and windows config functions available in Boxstarter core. Salim has also contributed corresponding changes to the vagrant and EC2 Test-Kitchen drivers.
At CenturyLink, we use vmware and a customized vsphere driver to test with Test-Kitchen. It was trivial to add support for Salim’s branch.. With the Boxstarter cookbook, I use his vagrant plugin without issue. According to this Chef blog post, all of this windows work will likely be pulled into the next release of Test-Kitchen.
But I just cant wait. I must try this today!
So for those interested in “kicking the tires” today, here is how you can install all the bits needed:
cinst chefdk cinst vagrant git clone -b transport https://github.com/afiune/test-kitchen git clone -b transport https://github.com/mwrock/kitchen-vagrant copy-item test-kitchen\lib ` C:\opscode\chefdk\embedded\apps\test-kitchen ` -recurse -force copy-item test-kitchen\support ` C:\opscode\chefdk\embedded\apps\test-kitchen ` -recurse -force copy-item -Path kitchen-vagrant\lib ` C:\opscode\chefdk\embedded\lib\ruby\gems\2.0.0\gems\kitchen-vagrant-0.15.0` -recurse -force cd test-kitchen gem build .\Test-kitchen.gemspec chef gem install test-kitchen-1.3.0.gem
This will install the Chef Development Kit and vagrant via Chocolatey and I’m assuming you have chocolatey installed. Otherwise you can download these from their respective download pages here and here. Then it clones the winrm based test-kitchen and kitchen-vagrant projects and copies them over the current bits.
Note that my instructions here are assuming you are testing on Windows. However, the winrm functionality is most certainly capable of running on Linux as I do at work. If you were doing this on Linux, I’d suggest running bundle install and bundle exec instead of copying over the chef directories. However this has caused me too many problems on Windows to recommend to others and purely copying the bits has not caused me any problems.
Hyper-V
Now you can pull down the boxstarter cookbook to test from https://github.com/mwrock/boxstarter-cookbook. If you run Hyper-V, you will want to install my vagrant fixes according to the instructions above since the box inside the boxstarter cookbook’s kitchen config is on a vhd file. You can then simply navigate to the boxstarer cookbook directory and run:
kitchen test
This will build a win 2012 R2 box and install and test a very simple cookbook via Test-Kitchen.
Virtual Box
If you run VirtualBox, you will need to make a couple changes. Replace the VagrantfileWinrm.erb content with this:
# -*- mode: ruby -*- # vi: set ft=ruby : VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "<%= config[:box] %>" config.vm.box_url = "<%= config[:box_url] %>" config.vm.guest = :windows config.winrm.username = "vagrant" config.winrm.password = "vagrant" config.winrm.port = 55985 end
You would also replace the .kitchen.yml content with:
The test included in the boxstarter cookbook is not very interesting but illustrates that you can indeed run kitchen tests against windows machines with no ssh installed.
---
driver:
name: vagrant
provisioner:
name: chef_zero
platforms:
- name: windows-81
transport:
name: winrm
max_threads: 1
driver:
port: 55985
username: vagrant
password: vagrant
guest: :windows
box: mwrock/Windows8.1-amd64 vagrantfile_erb: VagrantfileWinrm.erb
box_url: https://wrock.blob.core.windows.net/vhds/win8.1-vbox-amd64.box
suites:
- name: default
run_list:
- recipe[boxstarter_test::simple]
attributes:Looking at a more interesting ServerSpec test
For those reading who might want to see what a more interesting test would look like, lets take a look at this Chef recipe:
include_recipe 'boxstarter::default'
boxstarter "boxstarter run" do
password 'Pass@word1'
code <<-EOH
Update-ExecutionPolicy Unrestricted
Set-WindowsExplorerOptions -EnableShowHiddenFilesFoldersDrives `
-EnableShowProtectedOSFiles -EnableShowFileExtensions
Enable-RemoteDesktop
cinst console2
cinst IIS-WebServerRole -source windowsfeatures
#Install-WindowsUpdate -acceptEula
EOH
endThis is a sample recipe I include with the Boxstarter cookbook but I have commented out the call that runs windows updates. This recipe will run the included Boxstarter resource and perform the following:
Update the powershell execution policy
Adjust the windows explorer settings
enable remote desktop
install the console2 command line console
Install IIS
Here is a test file that will check most of the items changed by the recipe:
require 'serverspec'
include Serverspec::Helper::Cmd
include Serverspec::Helper::Windows
describe file('C:\\programdata\\chocolatey\\bin\\console.exe') do
it { should be_file }
end
describe windows_feature('Web-Server') do
it{ should be_installed.by("powershell") }
end
describe windows_registry_key(
'HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\advanced') do
it { should have_property_value('Hidden', :type_dword,'1') }
end
describe command('Get-ExecutionPolicy') do
it { should return_stdout 'Unrestricted'}
endServerspec provides a nice Ruby DSL for testing the state of a server. Although the test is pure ruby code, in most cases you don’t really need to know ruby. Familiarity with the cut and paste features will be very helpful so please review those as necessary.
The documentation on the ServerSpec.org page does a decent job of describing the different resources that can be tested. Above are just a few: a file resource, windows feature resource, a windows registry resource and a command resource that you can use to issue any powershell necessary to test your server.
All of these tests, as we do at CenturyLink can be fed into a Continuous Integration server (Jenkins, TeamCity, TFS, etc.) to give your team speedy feedback on the state of your automation codebase.
I hope you find this helpful and I look forward to these features making it into the official vagrant and test-kitchen installs soon.
As of last week I am no longer working at Microsoft. I worked at Microsoft for the last four and a half years and it was an amazing experience where I learned a lot from many very smart people. I am now a Software Engineer focusing on data center automation at 
Just a month after releasing integration with Hyper-V, I am pleased to announce support for auto connection configuration and first class Checkpointing in Windows Azure VMs. 
