Windows VM Provisioning Part 1: Inject a ‘startup on boot ‘ script into a VHD.

I’m currently in the process of adding a Virtualization module to my Boxstarter project. This post is part of a three part series covering some of the technicalities involved. Although these posts will document how Boxstarter provisions a Windows VM, I think that there will be information covered to accommodate a wide range of scenarios whether you are interested in using Boxstarter or not.

The Boxstarter Scenario

Boxstarter is a set of Powershell modules (or you can simply invoke from a URL) that can deploy a single application or standup a complete environment from a script leveraging Chocolatey and Nuget packaging technologies. Its target scenario is to bring a Windows machine from bare OS to an environment that is fully patched and has everything you need to get stuff done. Yes, there are lots of cool solutions that can do this at enterprise scale but Boxstarter is designed to be light weight and very simple. It can perform this provisioning on a physical machine but what about VMs?

You can of course log on to a VM and use Boxstarter just as you would on any physical machine, but I want to eliminate the need to have to setup network settings and manually RDP or use a Hyper-V console to connect to the VM. Deploying a Boxstarter Install to a VM should be just as simple as any other environment.

How will Boxstarter provision a VM without the need to manually prepare it

Here is the basic flow:

1. Inject a script into a VHD that will run under the local machine account with administrative privileges on boot. This script will add a firewall rule and edit the Local Account Token Filter Policy.
2. Use PSEXEC to invoke another script from a user account that will enable powershell Remoting on the VM with credssp authentication so that your credentials in the remote session can be used to access other remote resources from the VM.
3. Use powershell remoting to Invoke a Boxstarter package from the VM Host but that will run on the VM Guest.
4. Wrap all of this up into a single, simple command. that can be extended to be VM Vendor agnostic but will work with Hyper-V and Windows Azure VMs out of the box.

Ideally, this could even be leveraged to create a Vagrant Provisioner.

This post will cover the zeroth point above. The use cases of plugging a startup script right into a VHD span well beyond Boxstarter and the means of doing it is not particularly difficult but it did take me a while to figure out how to get it done right to accommodate both simple workstation environments as well as Domain topologies.

Requirements on the Host and Guest

The Goal is that this should work on any “vanilla” bare OS guest install with access to the internet and no “special” networking configuration. No Firewall tweaking, no need to enable powershell remoting on the host or guest and no installation of software on the guest beyond the operating system. That said, the following are required:

1. The VM guest must be able to access the internet unless your boxstarter package installs everything from a local source.
2. The Host must be running at least Powershell v.3 and have the Hyper-V module available.
3. The Guest must be running windows 7, 8, server 2008 R2 or server 2012.
4. The VHD where the script is injected must contain the system volume of the VM (windows\system32).

The Script Script (the script that installs the script)

The script lives in the Boxstarter.VirtualMachine module and can be called like so:

$vhd=Get-VMHardDiskDrive -VMName "MyVMName"Add-VHDStartupScript $vhd.Path -FilesToCopy "c:\myFiles\file.ps1","..\MyOtherFiles" {    $here = Split-Path -Parent $MyInvocation.MyCommand.Path    . "$here\file.ps1"}

This will take the VHD used by the VM on the host named MyVMName, the file c:\myfiles\file.ps1 as well as all files in ..\MyOtherFiles will be copied to the VHD. Furthermore, the script block above will be stored in a file in the same directory as the copied files. A local Group Policy will be added to the Registry stored in the VHD that will call the above script when the VM next boots. To be clear, the script runs at boot time and not login time so that no separate login is necessary to kick things off. The script will run under the local machine account.

Validate and Mount the VHD

function Add-VHDStartupScript {[CmdletBinding()]param(    [Parameter(Position=0,Mandatory=$true)]    [ValidateScript({Test-Path $_})]    [ValidatePattern("\.(a)?vhd(x)?$")]    [string]$VHDPath,    [Parameter(Position=1,Mandatory=$true)]    [ScriptBlock]$Script,    [Parameter(Position=2,Mandatory=$false)]    [ValidateScript({ $_ | % {Test-Path $_} })]    [string[]]$FilesToCopy = @())if((Get-ItemProperty $VHDPath -Name IsReadOnly).IsReadOnly){    throw New-Object -TypeName InvalidOperationException `      -ArgumentList "The VHD is Read-Only"}    $volume=mount-vhd $VHDPath -Passthru | get-disk | Get-Partition | Get-Volumetry{    Get-PSDrive | Out-Null    $winVolume = $volume | ? {        Test-Path "$($_.DriveLetter):\windows\System32\config"    }    if($winVolume -eq $null){        throw New-Object -TypeName InvalidOperationException `          -ArgumentList "The VHD does not contain system volume"    }

In the beginning of the script we validate the user input and Mount the VHD. Pretty straight forward stuff.

Copy files and create startup script file

$TargetScriptDirectory = "Boxstarter.Startup"mkdir "$($winVolume.DriveLetter):\$targetScriptDirectory" -Force | out-nullNew-Item "$($winVolume.DriveLetter):\$targetScriptDirectory\startup.bat" -Type File `  -Value "@echo off`r`npowershell -ExecutionPolicy Bypass -NoProfile -File `"%~dp0startup.ps1`""`  -force | out-nullNew-Item "$($winVolume.DriveLetter):\$targetScriptDirectory\startup.ps1" -Type File `  -Value $script.ToString() -force | out-nullForEach($file in $FilesToCopy){    Copy-Item $file "$($winVolume.DriveLetter):\$targetScriptDirectory" -Force}

Here we copy the files provided in the FilesToCopy parameter and create a powershell file to hold the script in the script block and a batch file that will invoke the powershell file.

Load the registry hive in the VHD

reg load HKLM\VHDSYS "$($winVolume.DriveLetter):\windows\system32\config\software" | out-null

This takes the file in the VHD that contains HKLM\Software, which is where the computer startup group policies reside and Loads its keys into a new hive referencable from HKLM:\VHDSYS. Now we can query and modify the values in the registry as easily as we can any of our local registry information.

Add the Group Policy

Now that the VHD Registry is loaded, we need to add a Local Group Policy that will invoke our startup.bat file upon boot. This is a bit involved to account for various scenarios such as:

• What if you already have different startup scripts
• What if you have already have added a startup script and do not want to add a duplicate
• What if you have one or more domain group policies

To try and keep things at least somewhat tidy, we will place this logic in a separate function:

function Get-RegFile {    $regFileTemplate = "$($boxstarter.BaseDir)\boxstarter.VirtualMachine\startupScript.reg"    $startupRegFile = "$env:Temp\startupScript.reg"    $policyKey = "HKLM:\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy"    $scriptNum=0    $localGPONum=0    if(Test-Path "$policyKey\Scripts\Startup\0\0"){        $localGPO = Get-ChildItem "$policyKey\Scripts\Startup" | ? {            (GCI -path $_.PSPath -Name DisplayName).DisplayName -eq "Local Group Policy"        }        if($localGPO -ne $null) {            $localGPONum = $localGPO.PSChildName            $localGPO=$null #free the key for GC so it can be unloaded        }        else{            Shift-OtherGPOs "$policyKey\Scripts\Startup"            Shift-OtherGPOs "$policyKey\State\Machine\Scripts\Startup"        }        if(test-path "$policyKey\Scripts\Startup\$localGPONum"){            $scriptDirs = Get-ChildItem "$policyKey\Scripts\Startup\$localGPONum"            $existingScriptDir = $scriptDirs | ? {                 (Get-ItemProperty -path $_.PSPath -Name Script).Script `                  -like "*\Boxstarter.Startup\startup.bat"            }            if($existingScriptDir -eq $null){                [int]$scriptNum = $scriptDirs[-1].PSChildName                $scriptNum += 1            }            else {                $scriptNum = $existingScriptDir.PSChildName                $existingScriptDir = $null #free the key for GC so it can be unloaded            }        }        $scriptDirs=$null    }    (Get-Content $regFileTemplate) | % {        $_ -Replace "\\0\\0", "\$localGPONum\$scriptNum"    } | Set-Content $startupRegFile -force    return $startupRegFile}

function Shift-OtherGPOs($parentPath){    Get-ChildItem $parentPath | Sort-Object -Descending | % {        [int]$num = $_.PSChildName        $oldName = $_.Name.Replace("HKEY_LOCAL_MACHINE","HKLM:")        [string]$newName = "$($num+1)"        try {Rename-Item -Path $oldName -NewName $newName} catch [System.InvalidCastException] {            #possible powershell bug when renaming reg keys that are numeric            #the key is copied but the old key remains            Remove-Item $oldName -Recurse -force        }    }}

A couple things to note here. The script Group Policies appear to be mirrored at both:

• HKLM:\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts\Startup
• HKLM:\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup
  I honestly do not know why or what the significance is of the different locations. These are just the keys I saw that were affected when I manually played with creating startup scripts inside GPEDIT.MSC.

The Script keys maintain the following subkey structure:

\Scripts\Startup\{policy node}\{script node}

The policy and script nodes are each simple integers starting at 0 and increment for each additional policy scope or script. So a machine with a Local policy containing 1 script and a domain policy containing 2 scripts would look like:

\Scripts\Startup\0\0 – Local policy script

\Scripts\Startup\1\0 – First domain Policy script

\Scripts\Startup\1\1 – Second domain Policy script

From what I could tell in my experimentation, the Local Policy always occupied position 0.

Here are some surprising and unintuitive findings to be aware of:

• When you are done with the registry you will need to unload it just as we loaded it in order to free up the file. That is not surprising. What is surprising is that if you save any keys to a variable as you are navigating its values, you will need to dereference those variables. It was fun in VB6 and it is still fun today! To add icing to this cake, you also need to do the thing that you should really never do: call GC::Collect() before unloading. Yep, that’s right. Unless of coarse you like Access Exceptions.
• There seems to be a bug in the powershell registry provider when renaming keys that have a numeric value. Doing so raises a invalid cast exception. It also goes ahead and creates the renamed key but it does not delete the old name. This is why I delete it inside of the catch block.

The call to the above Get-RegFile function and the surrounding import code looks like:

reg load HKLM\VHDSYS "$($winVolume.DriveLetter):\windows\system32\config\software" | out-null$startupRegFile = Get-RegFilereg import $startupRegFile 2>&1 | out-nullRemove-Item $startupRegFile -force

We use the reg command to import the temp file with the altered template and then dispose of the temp file.

 

The registry import template

I found it easiest to create a .reg file containing all of the registry modifications instead of using the powershell registry provider to individually modify the tree. I simply needed to determine the correct policy and script nodes and then inject those into the template. Here is the template:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts\Shutdown]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts\Startup]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts\Startup\0]"GPO-ID"="LocalGPO""SOM-ID"="Local""FileSysPath"="%SystemRoot%\\System32\\GroupPolicy\\Machine""DisplayName"="Local Group Policy""GPOName"="Local Group Policy""PSScriptOrder"=dword:00000001

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\State\Machine\Scripts\Startup\0\0]"Script"="%SystemDrive%\\Boxstarter.Startup\\startup.bat""Parameters"="""ExecTime"=hex(b):00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup]

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup\0]"GPO-ID"="LocalGPO""SOM-ID"="Local""FileSysPath"="%SystemRoot%\\System32\\GroupPolicy\\Machine""DisplayName"="Local Group Policy""GPOName"="Local Group Policy""PSScriptOrder"=dword:00000001

[HKEY_LOCAL_MACHINE\VHDSYS\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup\0\0]"Script"="%SystemDrive%\\Boxstarter.Startup\\startup.bat""Parameters"="""IsPowershell"=dword:00000000"ExecTime"=hex(b):00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00

Cleaning up

Finally, we unload the registry hive and dismount the VHD:

}finally{    [GC]::Collect()    reg unload HKLM\VHDSYS 2>&1 | out-null    Dismount-VHD $VHDPath}

Again as noted before, we have the unattractive GC Collect. If anyone discovers a way to avoid this please comment but I was not able to find any other way around this. Failing to call Collect results in an Access Exception when unloading the registry. This is only the case if you have used the powershell registry provider to navigate the loaded hive. Also as noted before, if you have referenced any keys in a variable, you must deallocate those variables before the call to GC::Collect().

Reboot a VM attached to the VHD and see the script run

That’s it. Now you can reboot a VM attached to this VHD and its script will execute under the local machine account’s credentials with administrative privileges. Since the VHD format is supported by almost all of the major Virtualization vendors, you should be able to leverage this script on most virtualization platforms.

Next: Enable Powershell Remoting

The next post will explore how to use this script to enable powershell remoting on a VM guest. It is not enough to simply have the script enable remoting since remoting must be enabled by a user account. I’ll show you the approach I found that works to set that up.

Understanding networking options with a wireless Hyper-V host and solving Catastrophic failure (0x8000FFFF)

I have been experimenting lately with different network configurations and exploring ways to automate the setup and deployment of VMs in conjunction with Boxstarter. Along the way I have bumped up against some problems and discovered more than one way that many on the net recommend configuring a Windows 8 Hyper-V setup on a wireless network. In the process of this experimentation I have also managed to, more than once, screw things up. I’ll start with the screwing things up part.

Problem: cant create a Hyper-V network due to catastrophic failures

A few months ago my laptop ended up in a state where I could not create an external network on my Hyper-V virtual switch. All attempts resulted in a catastrophic failure which included the guid of the Network Interface and the hex code 0x8000FFFF.

Ethernet port '{C7DB406D-4F2A-44A9-8C10-7952707F7498}' bind failed: Catastrophic failure (0x8000FFFF).

After hours of futile random attempts to fix this and little to no clues from google, I arrived at a hypothesis that this was somehow connected to my VirtualBox setup that I had since uninstalled but there were some Network interfaces left over that I could not seem to remove. I ended up repaving my OS to get back to a good state.That was clearly not the solution I was hoping for.

SIDEBAR: I have been working on a project called Boxstarter off and on for the past year. This allows one to rebuild a machine in a scripted manner. It leverages the functionality of Chocolatey and adds an environment that is reboot resilient and also exposes some handy Cmdlets that tweak windows configuration and install updates. So this was not such a big deal since I can have my familiar dev environment back in about an hour.

Last weekend I started running into the exact same scenario and error. However, I have been doing some experimenting with different network and Hyper-V configurations lately and found a clue that helped me to recover without reinstalling my OS.

The Solution

For those that do not care to read on, this was the issue: I had previously setup an “Internal Network” in Hyper-V and “Shared” my wireless adapters internet connection with my Hyper-V Internal Network. I later removed the Internal Network but did not turn off sharing on my wireless adapter first. A Shared network interface cannot be bridged, which is what is used when you create an External network on a wireless connection. Oddly even attempting to create a new Private hyper-v network was failing but with a different but equally unhelpful error. The key problem here is now that I only had one network adapter, the ability to turn on or off sharing was no longer available. The adapter was shared:

but I could not go to the properties page, go to the “Sharing”  tab and then turn off sharing. The Sharing tab was no longer present.

This problem is further complicated by the fact that even on Powershell 3 and 4 with all of its new Networking cmdlets, there is no cmdlet that can turn on or off Internet Connection Sharing. However, there are a set of COM interfaces that expose the sharing properties of a network interface. This script disabled  my internet sharing and allowed me to then create a new Hyper-V network:

function Disable-ICS{    $cfg = New-Object -ComObject HNetCfg.HNetShare.1    $all = $cfg.EnumEveryConnection    foreach($conn in $all){        $shareCfg=$cfg.INetSharingConfigurationForINetConnection($conn)        $props=$cfg.NetConnectionProps($conn)        $props        $shareCfg.DisableSharing()    }}

Disable-ICS

Hyper-V, Wireless Adapters, and a connected VM

One of the great things about Windows 8 is that one can run a Hyper-V server and host guest VMs. Given that I use my personal laptop to develop Boxstarter. Being able to test machine installs makes running VMs a necessity and I much prefer running the lighter weight Windows 8 over Server 2012. Also given that I am working on a laptop, I am always using a wireless connection.

The challenge of a wireless adapter

Prior to the Windows 8 implementation of Hyper-V, a Hyper-V VM could only connect to an external network through a wired network adapter. Since Hyper-V supports a Layer-2 switch, it routes traffic based on the MAC address of the virtual network adapters. A wired adapter can run in promiscuous mode and allow packets to be transmitted with the MAC address of the virtual NIC. A wireless adapter can only send traffic with its own MAC address. To mitigate this issue, the Windows 8 Hyper-V uses a bridged adapter to bridge the wireless NIC and the virtual NIC. Although the virtual MAC address is replaced by the wireless adapter’s MAC address when a packet is sent, the bridge maintains a mapping of virtual IP addresses to their MAC addresses so that traffic can be properly routed back to the correct VM.

Two ways to connect your Windows 8 VMs to the internet

#1: Using an External Network Switch

This is essentially exactly what I have just outlined above. In the Hyper-V manager (this can also be accomplished using the Hyper-V powershell module), go to the Virtual Switch Manager and create an External network binding to your wireless network adapter:

As previously described, this will create a new network adapter (the virtual adapter) and a network bridge that connects the virtual and wireless adapters.

Finally go to the “Add Hardware” section of your VM while it is turned off and add a Network Adapter assigning it to the switch you just created. Now your VMs access the same external network as the host as if they are simply another device attached to your wireless router and assigned their own IP from your router’s DHCP server.

#2: A Shared Internal Network

There is another option to connect your VMs to the internet besides using an External Virtual switch as described above. This includes the creation of an Internal virtual switch instead of an external one. An Internal switch establishes a connection between the guest VM and the host but not to the outside world. Ok. So how in the heck will that connect a VM to my internet? Well after you create the Internal switch:

You will “Share” your wireless adapter with the internal switch.

Note that when you have one or more internal network adapters, the “sharing” tab becomes available in your WIFI adapter’s properties. If you had more than one internal adapter, you would also see a drop down menu allowing you to choose which internal adapter to share. Once you add the internal adapter to your VM, you should be able to access your host internet connection from your guest. Maybe…

Which method should I u…Maybe?…What does that mean?

If you have done much if any investigation already on setting up a windows 8 VM client on a wireless network, you have likely run across accounts of others having issues getting connectivity setup just right and also seen others advising to follow one of the two above methods. Sometimes the recommendations are reported to work but other times the individual seeking help states that they still cannot establish connectivity.

I’ve been tinkering with this for months and have several times fallen into a state where I have followed every step but things fail to work. This technology is fairly young and still, IMHO rather fragile. Usually the fix is something like rebooting the host or guest, disabling/enabling one of the adapters, fiddling with the network bindings (YUCK!), or toggling the network sharing on or off. None of these solutions inspire much confidence, but sometimes the answer to which approach works best is whichever approach simply works for you.

Fragility Aside, what are some of the nuances of each

The Bridged External Switch is the more stable

My personal experience is that the bridged network is more stable and the connectivity is the most “full featured.” I did go through a bout of having to reset the bridged connection and rebinding its IPv4 and IPv6 bindings after each reboot or sleep cycle. That was quite annoying and I eventually wrote this script to automate the process:

function Reset-NetAdapters {    $bindings = @(        "Client for Microsoft Networks",        "Internet Protocol Version 6 (TCP/IPv6)",        "Internet Protocol Version 4 (TCP/IPv4)"        )    $bindings | %{ Enable-NetAdapterBinding -Name "Network Bridge" -DisplayName $_ }    Disable-NetAdapter "vEthernet (Virtual Switch)"    Enable-NetAdapter "vEthernet (Virtual Switch)"}

When I installed the Windows 8.1 preview that went away but I honestly think that was more the fault of an OS overhaul than windows 8.1 specifically.

This also seems to be the “official” or “supported” approach. It was even mentioned in Stephen Sinofsky’s blog.

The Internal switch can be unreliable but adds some privacy

If you simply want to access the web or access files on the host, the connectivity provided here should be fine. However I did run into some odd bumps. For example, windows updates did not work at first until I discovered this very cryptic fix: on the guest NIC, go to the advanced configuration settings and disable the “Large Send Offload” settings for both IPv4 and IPv6.

Also, while this allows your guest VM to access the outside world, the outside world cannot see it. This can be good or bad depending on how you want to use the VM. The benefit is some added security. The VM is not actually on the external network and therefore others in your network cannot ping it or access its shares. But then maybe that is exactly what you want to do. This is basically using NAT (network address translation) and assigns the guest VM its own address when transmitting communication outside of the internal network. This is the default networking method used by VirtualBox and is most convenient if you have a limited number of IP addresses and don’t want to assign a separate address to every device.

Hopefully you have found this usefull and especially if you are suffering from good old 0x8000FFFF, the SEO gods have provided you a solution more quickly than it took me to find.

Automate Codeplex Releases in your Powershell build and deployment scripts

I maintain a few projects on Codeplex.com and one that I have been working on actively lately is Boxstarter. A sort of “Add-On” to Chocolatey that allows you to run complex (or simple) environment scripts completely unattended in a reboot/logon resilient manner.

Whenever I startup any project that I plan to develop beyond a hack or a spike, the very first thing I do is lay down test infrastructure along with build and deployment automation. These are NOT “nice to haves” and especially since I devote my “free” time to these projects, it is important that I make the most of that time and spend as little time as possible chasing regression bugs or mucking with the Release gymnastics of packaging things up and copying/uploading files, etc. It is too easy to get that stuff wrong and takes too long to do and test every detail by hand every time I want to push bits - not to mention that it is just not fun.

When I push a Boxstarter release, my download zip, click-once app  and home page are updated on my Azure web site by doing a GIT Push to Master. My Chocolatey packages are packed and pushed by a deployment script. So the one manual step has been uploading ,my download zip file to Codeplex and creating a Codeplex Release for it. Well it ends up there is a web service that codeplex maintains to facilitate the automation of this step. It is a SOAP service and therefore a bit of a pain to call directly, but Codeplex also has a C# SDK that makes working with this service very easy.

While the SDK is a .Net Assembly that was likely intended for C# based clients, I prefer to write my build and deployment scripts in Powershell. I believe that what happens in a deployment should be easily discoverable and that the script should serve as the “deployment document.” Compiled code does not lend itself well to that kind of transparency. I also don’t like to use XML for control flow and therefore avoid msbuild for all but the basic compile tasks. PSake is a great powershell tool I have been using for years to organize and execute build and deployment scripts. Every build task is encapsulated by a Powershell function called Task. Here is my Push-Codeplex task:

Task Push-Codeplex { Add-Type -Path ` "$basedir\BuildScripts\CodePlexClientAPI\CodePlex.WebServices.Client.dll" $releaseService=New-Object CodePlex.WebServices.Client.ReleaseService $releaseService.Credentials = Get-Credential `   -Message "Codeplex credentials"`   -username "mwrock" $releaseService.CreateARelease(`     "boxstarter",`     "Boxstarter $version",`     "Running the Setup.bat file will install Chocolatey if not present `      and then install the Boxstarter modules.", `     [DateTime]::Now,[CodePlex.WebServices.Client.ReleaseStatus]::Beta,`     $true,`     $true) $releaseFile = New-Object CodePlex.WebServices.Client.releaseFile $releaseFile.Name="Boxstarter $version" $releaseFile.MimeType="application/zip" $releaseFile.FileName="boxstarter.$version.zip" $releaseFile.FileType=`   [CodePlex.WebServices.Client.ReleaseFileType]::RuntimeBinary $releaseFile.FileData=[System.IO.File]::ReadAllBytes(`   "$basedir\BuildArtifacts\Boxstarter.$version.zip") $fileList=new-object `   "System.Collections.Generic.List``1[[CodePlex.WebServices.Client.ReleaseFile]]" $fileList.Add($releaseFile) $releaseService.UploadReleaseFiles("boxstarter", "Boxstarter $version", $fileList)}

This adds the types in the Codeplex.Webservices.Client assembly to the current Powershell session. I have downloaded  the SDK and put it in my source tree. Its too bad this is not packaged in its own Nuget package. Then I could use Package Restore to grab the bits from Nuget.org.

Note that this assembly has a dependency on a couple Build assemblies. You could try to track these down separately but since I have Visual Studio installed, this gives me these dependencies.

The parent object handling all codeplex project commands takes a ICredentials to authenticate the client. I can simply use Get-Credential for this. This does require me to manually enter my password but that is fine for my script.If you had a more enterprise grade deployment that needed to fully script the authentication, that is very possible. After that I am free to create my release, and upload my file. This worked flawlessly for my project and now I can deploy my project with a single command.

Setup a new machine with just a URL and Chocolatey package

When I first started working on Boxstarter, a set of scripts that can stand up a complete, customized windows environment in a completely unattended manner, one of my key objectives was to be able to spin up a new environment with as few bits on hand as possible. Until now, Boxstarter achieved this by allowing you to create a portable Chocolatey package on a thumb drive or network share. I have never really been satisfied with that. It always seems when the time comes and I want to repave, that thumb drive or share is not around or in an up to date state. What I really want is just to type something brief and rememberable that would bring down all the bits I need to perform the setup (at least solving the “not around” part). This is now a reality.

Introducing The Boxstarter Click-Once Web Launcher

You can use any browser that supports Click Once applications. This is limited to IE unless you add extensions to Chrome or Firefox. You can also use either the Command Line or a Powershell console, both using the same command as long as IE is your default browser. The fact that this method is rather IE-centric is not much of a drawback since every Windows machine has IE as its default browser in the base install which is the key Boxstarter scenario.

All you need to remember is the Boxstarter domain and the Chocolatey package that drives your machine setup. In the above example, the package is called…well…example. Boxstarter probes both the public Chocolatey.org gallery and the Myget.org community boxstarter feed for a package ID named Example.

This first installs a Boxstarter Click-Once Bootstrapper.

Be aware that on Windows 8, the Smart Screen filter will supply a somewhat frightening message stating that you have downloaded software that is not yet trusted.

Even though the Boxstarter Web Launcher is signed with an Authoritative certificate, it has not “gained enough reputation” to be considered safe. In theory, this should eventually go away. In the meantime, you need to click the “More info” link to get this dialog:

Click the “Run Anyway” link to proceed. Next, unless you have UAC disabled, you need to give Boxstarter permissions to run with administrative privileges. Boxstarter needs these privileges in order to run all of the application installs, reboot and perform all actions you want it to to tweak your windows environment.

Next the normal Boxstarter session begins and prompts you for your windows account password:

Boxstarter stores the password you enter in a special place in windows designed specifically for this purpose. The password is stored in encrypted form and is never sent over the network or stored in plain text. This allows you to be automatically logged on whenever Boxstarter needs to reboot your system throughout the system installation.

If you want to install a package without the risk of reboot and without this prompt, you can invoke boxstarter using this URL:

HTTP://Boxstarter.org/package/nr/example

Its all in the “/nr/”. This might be handy if you know no reboot is needed and perhaps you want to use Boxstarter because you know Chocolatey is not installed.

What happens next?

Boxstarter now installs Chocolatey and (if you are on windows 7 or server 2008 R2) .net4.0 if these are not already installed. It then proceeds to run the package you specified in the original URL. This is just a normal Chocolatey package. Boxstarter can run any Chocolatey package. What’s different when running the package in Boxstarter is the following:

• Boxstarter shuts down the windows update service and if installed, the SCCM client. These can often interfere with software installations.
• Boxstarter intercepts all Chocolatey install calls. If there is a pending reboot, boxstarter invokes a system reboot and ensures that it will be reinvoked first thing on startup.
• Boxstarter examines all Chocolatey failures and if these are standard MSI Pending reboot failures, it will like above, restart your system and try again after restarting.
• Boxstarter runs the entire installation as admin to avoid prompts for permissions elevation or errors complaining about insufficient permissions.
• Boxstarter imports the Boxstarter.Winconfig powershell module making all of its CmdLets available that provide various helpers for customizing windows settings (enabling remote desktop, customizing windows explorer defaults, icon sizes, etc.). The most important feature in this module is the ability to run all critical windows updates. A routine requirement when setting up a new system.

What might a Chocolatey package look like for a Boxstarter install?

Here is an example:

Update-ExecutionPolicy Unrestricted Move-LibraryDirectory "Personal""$env:UserProfile\skydrive\documents" Set-ExplorerOptions -showHidenFilesFoldersDrives -showProtectedOSFiles -showFileExtensions Enable-RemoteDesktop cinstm VisualStudioExpress2012Web cinstm fiddler cinstm mssqlserver2012express cinstm git-credential-winstore cinstm console-devel cinstm skydrive cinstm poshgit cinst Microsoft-Hyper-V-All -source windowsFeatures cinst IIS-WebServerRole -source windowsfeatures cinst TelnetClient -source windowsFeatures Install-ChocolateyPinnedTaskBarItem "$env:programfiles\console\console.exe" copy-item (Join-Path (Get-PackageRoot($MyInvocation)) 'console.xml') -Force $env:appdata\console\console.xml Install-ChocolateyVsixPackage xunit http://visualstudiogallery.msdn.microsoft.com/463c5987-f82b-46c8-a97e-b1cde42b9099/file/66837/1/xunit.runner.visualstudio.vsix Install-WindowsUpdate -AcceptEula

This is a relatively simple setup. Here is what will happen:

• The user’s powershell execution policy will be permanently set to unrestricted.
• The MyDocuments folder is moved to Skydrive (I always do this and it is incredibly convenient since my docs (including things like powershell profile) are synced between machines and makes the risk of a machine failure less ominous).
• The Windows explorer settings are set to a developer sane level.
• Remote Desktop is enabled.
• Several typical developer applications are installed including Visual Studio and Sql server. When installed together these two often make for a “reboot perfect storm.” All of their dependent packages will be installed as well like GIT for poshgit and the .net 4.5 framework for Visual Studio.
• Hyper-V, IIS and the Telnet client are installed
• I “pin” Console, my favorite command line console, to the task bar and copy its configuration file with my preferred settings that I embedded in the chocolatey package to the directory where it expects to find it.
• I install the XUnit test runner Visual Studio extension.
• Lastly I install all critical windows updates.

Chances are this will include at least one or two reboots in windows 8 and several more in Windows 7. Boxstarter will reboot when needed and then simply rerun the package after restart. Chocolatey helps us from installing packages more than once. On subsequent runs it knows if we installed something previously and will just skip to the next install. This does mean that any custom script we add, and we can add any powershell we want, must be repeatable.

I thought Powershell 4’s Desired State Configuration solves this

Desired state configuration is the flagship feature shipping with Powershell 4 on Windows 8.1 and Server 2012 R2. It is no less than very cool and will indeed change the way we do machine configuration in Windows at last. It allows you to create declarative configuration files hat describe exactly how you want a machine to look and then tells one or more machines in your enterprise, look like this and do whatever it takes to make that happen.

The gaps that Boxstarter tries to fill here are:

• Backward compatibility back to windows 7 and server 2008 R2.
• Support for client systems. DSC is currently strongly targeting server infrastructure
• Tooling. Eventually, I want people to be able to use Boxstarter who have no knowledge of powershell or any scripting for that matter.

What's next for Boxstarter?

I’ve been on a bit of a oss hiatus lately with my day job keeping me quite busy during non daylight hours, but I have had some time lately to do some polishing and also make some headway into another goal I have for Boxstarter: Easily deploying a Boxstarter install to a new VM. If you are familiar with Vagrant, a great VM environment setup platform for VirtualBox, VMWare and AWS then you can picture where I am going with this. I’d like to bring the same (or atleast very similar) capabilities to Hyper-V and windows Azure VMs.

This week I wrote a test script to help me automate testing Boxstarter packages and Boxstarter itself. Its just part of my PSake build file now:

task Test-VM -requiredVariables "VmName","package"{ $vm= Get-VM $VmName Restore-VMSnapshot $vm-Name $vm.ParentSnapshotName -Confirm:$false Start-VM $VmName $creds= Get-Credential -Message "$vmName credentials"-UserName "$env:UserDomain\$env:username" $me=$env:computername $remoteDir=$baseDir.replace(':','$') $encryptedPass= convertfrom-securestring -securestring $creds.password $modPath="\\$me\$remoteDir\Boxstarter.Chocolatey\Boxstarter.Chocolatey.psd1" $script= { Import-Module $args[0] Invoke-ChocolateyBoxstarter $args[1]-Password $args[2] } Write-Host "Waiting for $vmName to start..." do {Start-Sleep -milliseconds 100} until ((Get-VMIntegrationService $vm | ?{$_.name -eq"Heartbeat"}).PrimaryStatusDescription -eq"OK") Write-Host "Importing Module at $modPath" Invoke-Command -ComputerName $vmName-Credential $creds-Authentication Credssp -ScriptBlock $script-Argumentlist $modPath,$package,$creds.Password }

This takes a given VM and reverts it to its parent snapshot and then uses Powershell remoting to run the install. Take a look at this gist I wrote with a function to test pure Chocolatey packages. It includes some comments on how to setup the remoting.

I’d like to improve on this by cloning a VM from an existing VHD, then run the Boxstarter install on the VM taking over the mess of setting up the remoting and VM networking. We’ll see how that shapes up.

Feedback

If you have the chance to try Boxstarter out, I’m very interested to hear how he experience is. What is confusing, what does not work or what you would really like to see added as a feature. Please feel free to utilize the Codeplex Discusion and Issues features to voice these. You can also find complete documentation on the Boxstarter.Codplex.com documentation wiki.

Extract TFS Pending Changes to a zip file

Our TFS server was down today and I needed to get a Shelveset to a tester. Playing with the Power Tools PowerShell CmdLets I was able to basically pipe my pending changes to a zip file and give that to the tester.

Getting the Power Tools PowerShell Cmdlets

If you have Visual Studio Update 1 or Update two, you can download the TFS Power Tools from the Visual Studio Gallery here. If you have an older version of Visual Studio, download the Power Tools from the Microsoft Download Center here.

When you install the MSI, make sure to opt in to the PowerShell Cmdlets. The default options will not include the PowerShell Cmdlets!

Accessing the Cmdlets

After you have installed the power tools, you can launch the Power Tools Powershell console from the start menu item created during install. However if you are like me and have your own shell preferences, simply import them into your shell:

Import-Module "${env:ProgramFiles(x86)}\Microsoft Team Foundation Server 2012 Power Tools\  Microsoft.TeamFoundation.PowerTools.PowerShell.dll"

You may want to add this to your PowerShell Profile.

Extracting Pending Changes

Run the following commands from the root of your workspace to pipe all pending changes to a zip.

Add-Type -AssemblyName System.IO.Compression.FileSystem,System.IO.Compression$here= (Get-Item .).FullName$archive = [System.IO.Compression.ZipFile]::Open(  (Join-Path $here "archive.zip"), [System.IO.Compression.ZipArchiveMode]::Create)get-tfsPendingChange | % {   [System.IO.Compression.ZipFileExtensions]::CreateEntryFromFile(    $archive, $_.LocalItem, $_.LocalItem.Substring($here.Length+1)  ) }$archive.Dispose()

This simply adds the .net types to the powershell session and uses them to create a ZipArchive and then calls the Get-TfsPendingChanges to get a list of all files with pending changes. This is sent to a Zip file which in this case is called archive.zip and located in your current folder.

Requires .NET 4.5 and Powershell 3

The above zip file api makes use of the new, more friendly api for creating zip files. This will not work if you have .net 4.0 or lower. Also, since Powershell versions prior to 3.0 use the .NET 2 runtime, they will not be able to load .net 4.5 types. PowerShell 3.0 comes automatically unstalled on windows 8 and Server 2012. You may download and install the Windows Management Framework 3.0 here on Windows 7 or Server 2008 R2 to get Powershell 3.0 on those operating systems. You may get the .net 4.5 runtime here.