Whats new in the ruby WinRM gem 1.5 by Matt Wrock

Its been well over a year since Salim Afiune started the effort of making Test-Kitchen (a popular infrastructure testing tool) compatible with Windows. I got involved in late 2014 to improve the performance of copying files to Windows Test instances from Windows or Linux hosts. There was a lot of dust flying in the air as this work was nearing completion in early 2015. Shawn Neal nicely refactored some of the work I had submitted to the winrm gem into a spin off gem called winrm-fs. At the same time, Fletcher Nichol was feverishly working to polish off the Test-Kitchen work by Chef Conf 2015 and was making some optimizations on Shawn's refactorings and those found their way into a new gem winrm-transport.

Today we are releasing a new version of the winrm gem that pulls in alot of this work into the core winrm classes and will make it possible to pull the rest into winrm-fs, centralizing ruby winrm client implementations and soon deprecating the winrm-transport gem. If you use the winrm gem, there are now some changes available that improve the performance of cross platform remote execution calls to windows along with a couple other new features. This post summarizes the changes and explains how to take advantage of them.

Running multiple commands in one shell

The most common pattern for invoking a command in a windows shell has been to use the run_cmd or run_powershell_script methods of the WinRMWebService class.

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
winrm.run_cmd('ipconfig /all') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

winrm.run_powershell_script('Get-Process') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

Under the hood both run_cmd and run_powershell_script each make about 5 round trips to execute the command and return its output:

  1. Open a "shell" (equivalent of launching a cmd.exe instance)
  2. Create a command
  3. Request command output (potentially several calls for long running streamed output or long blocking calls)
  4. Terminate the command
  5. Close the shell

The first call, opening the shell, can be very expensive. It has to spawn a new process and involves authenticating the credentials with windows. If you need to make several calls using these methods, a new shell is spawned for each one. And things are even worse with powershell since that spawns yet another process (powershell.exe) from the command shell incurring the cost of creating a new runspace on each call. Stay tuned for a future winrm release (likely 1.6) that implements the powershell remoting protocol (psrp) to avoid that extra overhead.

While there is currently a way to make several calls in the same shell, its not very friendly or safe (you could easily end up with orphaned processes running on the remote windows machine). Typically this is not such a big deal because most will batch up several calls into one larger script. But here's the kicker: you are limited to a 8000 character command in a windows command shell (again stay tuned for the psrp implementation to avoid that). So imagine you are copying a 100MB file over the command line. Well, you will have to break that up into 8k chunks. While this may provide an excellent opportunity to review the six previous Star Wars episodes as you wait to transfer your music library, its far from ideal.

Using the CommandExecutor to stream commands

This 1.5 release, exposes a new class, CommandExecutor that you can use to make several commands from the same shell. The CommandExecutor provides run_command and run_powershell_script methods but these simply run a command, collect the output and terminate the command. You get a CommandExecutor by calling create_executor from a WinRMWebService instance. There are two usage patterns:

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
winrm.create_executor do |executor|
  executor.run_cmd('ipconfig /all') do |stdout, stderr|
    STDOUT.print stdout
    STDERR.print stderr
  end
  executor.run_powershell_script('Get-Process') do |stdout, stderr|
    STDOUT.print stdout
    STDERR.print stderr
  end
end

This yields an executor to a block that uses the executor to make calls. When the block completes, the shell opened by the executor will be closed.

The other pattern:

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
executor = winrm.create_executor

executor.run_cmd('ipconfig /all') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end
executor.run_powershell_script('Get-Process') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

executor.close

Here we are responsible for the executor and this closing the shell it owns. Its important to close the shell because if it remains open, that process will continue to live on the remote machine. Will it dream?...we may never know.

Self signed SSL certificates and ssl_peer_fingerprint

If you are using a self signed certificate, you can currently use the :no_ssl_peer_verification option to disable verification of the certificate on the client:

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :basic_auth_only => true, :no_ssl_peer_verification => true)

This is not ideal since it still risks "Man in the Middle" attacks. Still not completely ideal but better than completely ignoring validation is using a known fingerprint and passing that to the :ssl_peer_fingerprint option:

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :ssl_peer_fingerprint => '6C04B1A997BA19454B0CD31C65D7020A6FC2669D')

This ensures that all messages are encrypted with a certificate bearing the given fingerprint. Thanks here is due to Chris McClimans (HippieHacker) for submitting this feature.

Retry logic added to opening a shell

Especially if provisioning a new machine, it's possible the winrm service is not yet running when first attempting to connect. The WinRMWebService now accepts new options :retry_limit and :retry_delay to specify the maximum number of attempts to make and how long to wait in between. These default to 3 attempts and a 10 second delay.

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :retry_limit => 30, :retry_delay => 10)

Logging

The WinRMWebService now exposes a logger attribute and uses the logging gem to manage logging behavior. By default this appends to STDOUT and has a level of :warn, but one can adjust the level or add additional appenders.

winrm = WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass)

# suppress warnings
winrm.logger.warn = :error

# Log to a file
winrm.logger.add_appenders(Logging.appenders.file('error.log'))

If a consuming application uses its own logger that complies to the logging API, you can simply swap it in:

winrm.logger = my_logger

Up next: PSRP

Thats it for WinRM 1.5 but I'm really looking forward to productionizing a spike I recently completed getting the powershell remoting protocol working, which promises to improve performance and opens up other exciting cross platform scenarios.

Vagrant Powershell - The closest thing to vagrant ssh yet for windows by Matt Wrock

 Using "vagrant powershell" to remote to my windows nano server

Using "vagrant powershell" to remote to my windows nano server

A couple years ago when I started playing with vagrant to setup both windows and linux VMs, I wished there was a powershell equivalent of Vagrant's ssh command that would drop me into a powershell session on the windows vagrant box. Sure its simple enough to find the IP and winrm port of the box and run Enter-PSSession with the credentials given to vagrant, but a simple "vagrant powershell" just makes it so much more seamless.

After gaining some familiarity with ruby, I submitted  a PR to the vagrant github repo implementing this command. The command essentially shells out to powershell.exe and runs Enter-PSSession pointing to the vagrant guest using the same credentials that the vagrant's winrm communicator uses. Additionally, it temporarily adds the vagrant winrm endpoint to the host's Trusted Host entries, restoring the original entries once the command exits.

Its been a good while since that submission, but I'm excited to see it released in this week's first minor version update since that time.

Small bug/annoyance

Note that the up and down arrow keys do not cycle through command history. While I thought that was working a ways back, its possible that other changes around vagrant's handling of subprocesses caused this to pop up. At any rate, I just submitted a fix for that today and hope it is merged by the next bug fix release.

Only works on windows

The vagrant powershell command is only available on Windows hosts. Currently there is no cross platform implementation of the Powershell Remoting Protocol. There is a ruby based winrm repl but it is extremely limited compared to a true powershell session. However, the vagrant powershell command does provide a --command argument for running ad hoc non-interactive commands on the vagrant box. It would be easy to at least support that on non windows boxes.

Hosting a .Net 4 runtime inside Powershell v2 by Matt Wrock

My wish for the readers of this post is that you find this completely irrelevant and wonder why folks would wish to inflict powershell v2 on themselves now that we are on a much improved v5. However the reality is that many many machines are still running windows 7 and server 2008R2 without an upgraded powershell.

As I was working on Boxstarter 2.6 to support Chocolatey 0.9.9 which now ships as a .net 4 assembly, I had to be able to load it inside of Powershell 2 since I still want to support virgin win7/2008R2 environments. Without "help", this will fail because Powershell 2 hosts .Net 3.5. I really don't want to ask users to install an updated WMF prior to using Boxstarter because that violates the core mission of Boxstarter which is to setup a machine from scratch.

Adjusting CLR version system wide

So after some investigation I found several posts telling me what I already knew which included the following solutions:

  1. Upgrade to a WMF 3 or higher
  2. Create or edit a Powershell.exe.config file in C:\WINDOWS\System32\WindowsPowerShell\v1.0 setting the supportedRuntime to .net 4
  3. Edit the  hklm\software\microsoft\.netframework registry key to only use the latest CLR

I have already mentioned why option 1 was not an option. Options 2 and 3 are equally unpalatable if you do not "own" the system since both change system wide behavior. I just want to change the behavior when my application is running.

An application scoped solution

So after more digging I found an obscure, and seemingly undocumented environment variable that can impact the version of the .net runtime loaded: $env:COMPLUS_version. If you set this variable to "v4.0.30319" and then spawn a new process, that process will use the specified version of the .net runtime.

PS C:\Users\Administrator> $PSVersionTable

Name                           Value
----                           -----
CLRVersion                     2.0.50727.5420
BuildVersion                   6.1.7601.17514
PSVersion                      2.0
WSManStackVersion              2.0
PSCompatibleVersions           {1.0, 2.0}
SerializationVersion           1.1.0.1
PSRemotingProtocolVersion      2.1


PS C:\Users\Administrator> $env:COMPLUS_version="v4.0.30319"
PS C:\Users\Administrator> & powershell { $psVersionTable }

Name                           Value
----                           -----
PSVersion                      2.0
PSCompatibleVersions           {1.0, 2.0}
BuildVersion                   6.1.7601.17514
CLRVersion                     4.0.30319.17929
WSManStackVersion              2.0
PSRemotingProtocolVersion      2.1
SerializationVersion           1.1.0.1

A script that runs commands in .net 4

So given that this works, I created a Enter-DotNet4 command that allows one to run ad hoc scripts inside .net 4. Here it is:

function Enter-Dotnet4 {
<#
.SYNOPSIS
Runs a script from a process hosting the .net 4 runtile

.DESCRIPTION
This function will ensure that the .net 4 runtime is installed on the
machine. If it is not, it will be downloaded and installed. If running
remotely, the .net 4 installation will run from a scheduled task.

If the CLRVersion of the hosting powershell process is less than 4,
such as is the case in powershell 2, the given script will be run
from a new a new powershell process tht will be configured to host the
CLRVersion 4.0.30319.

.Parameter ScriptBlock
The script to be executed in the .net 4 CLR

.Parameter ArgumentList
Arguments to be passed to the ScriptBlock

.LINK
http://boxstarter.org

#>
    param(
        [ScriptBlock]$ScriptBlock,
        [object[]]$ArgumentList
    )
    Enable-Net40
    if($PSVersionTable.CLRVersion.Major -lt 4) {
        Write-BoxstarterMessage "Relaunching powershell under .net fx v4" -verbose
        $env:COMPLUS_version="v4.0.30319"
        & powershell -OutputFormat Text -ExecutionPolicy bypass -command $ScriptBlock -args $ArgumentList
    }
    else {
        Write-BoxstarterMessage "Using current powershell..." -verbose
        Invoke-Command -ScriptBlock $ScriptBlock -argumentlist $ArgumentList
    }
}

function Enable-Net40 {
    if(!(test-path "hklm:\SOFTWARE\Microsoft\.NETFramework\v4.0.30319")) {
        if((Test-PendingReboot) -and $Boxstarter.RebootOk) {return Invoke-Reboot}
        Write-BoxstarterMessage "Downloading .net 4.5..."
        Get-HttpResource "http://download.microsoft.com/download/b/a/4/ba4a7e71-2906-4b2d-a0e1-80cf16844f5f/dotnetfx45_full_x86_x64.exe" "$env:temp\net45.exe"
        Write-BoxstarterMessage "Installing .net 4.5..."
        if(Get-IsRemote) {
            Invoke-FromTask @"
Start-Process "$env:temp\net45.exe" -verb runas -wait -argumentList "/quiet /norestart /log $env:temp\net45.log"
"@
        }
        else {
            $proc = Start-Process "$env:temp\net45.exe" -verb runas -argumentList "/quiet /norestart /log $env:temp\net45.log" -PassThru 
            while(!$proc.HasExited){ sleep -Seconds 1 }
        }
    }
}

This will install .net 4.5 if not already installed and then spawn a new powershell process to run the given commands with the .net 4 runtime hosted.

Does not work in a remote shell

One scenario where this does not work is if you are remoted on a Powershell v2 machine. The .net4 CLR will almost immediately crash. My guess is that this is related to the fact that remote shells have an inherently different hosting model and run under wsmprovhost.exe or winrshost.exe.

The workaround for this in Boxstarter is to call the chocolatey.dll in a Scheduled Task instead of using Enter-DotNet4 when running remote.

Released Boxstarter 2.6 now with an embedded Chocolatey 0.9.10 Beta by Matt Wrock

Today I released Boxstarter 2.6. This release brings Chocolatey support up to the latest beta release of the Chocolatey core library. In March of this year, Chocolatey released a fully rewritten version 0.9.9. Prior to this release, Chocolatey was released as a Powershell module. Boxstarter intercepted every Chocolatey call and could easily maintain state as both chocolatey and boxstarter coexisted inside the same powershell process. With the 0.9.9 rewrite, Chocolatey ships as a .Net executable and creates a separate powershell process to run each package. So there has been lot of work to create a different execution flow requiring changes to almost every aspect of Boxstarter. While this may not introduce new boxstarter features, it does mean one can now take advantage of all new features in Chocolatey today.

A non breaking release

This release should not introduce any breaking functionality from previous releases. I have tested many different usage scenarios. I also increased the overall unit and functional test coverage of boxstarter in this release to be able to more easily validate the impact of the Chocolatey overhaul. That all said, there has been alot of changes to how boxstarter and chocolatey interact and its always possible that some bugs have hidden themselves away. So please report issues on github as soon as you encounter problems and I will do my best to resolve problems quickly. Pull requests are welcome too!

Where is Chocolatey?

One thing that may come as a surprise to some is that Boxstarter no longer installs Chocolatey. One may wonder, how can this be? Well, Chocolatey now exposes its core functionality via an API (chocolatey.dll). So if you are setting up a new machine with boxstarter, you will still find the Chocolatey repository in c:\ProgramData\Chocolatey, but no choco.exe. Further the typical chocolatey commands: choco, cinst, cup, etc will not be recognized commands on the command line after the Boxstarter run unless you explicitly install Chocolatey.

You may do just that: install chocolatey inside a boxstarter package if you would like the machine setup to include a working chocolatey command line.

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))

You'd have to be nuts NOT to want that.

Running 0.9.10-beta-20151210

When I say Boxstarter is running the latest Chocolatey, I really mean the latest prerelease. Why? That has a working version of the WindowsFeatures chocolatey feed. When the new version of Chocolatey was released, The WindowsFeature source feed did not make it in. However it has been recently added and because it is common to want to toggle windows features when setting up a machine and many Boxstarter packages make use of it, I consider it an important feature to include.


Fixing - WinRM Firewall exception rule not working when Internet Connection Type is set to Public by Matt Wrock

You may have seen the following error when either running Enable-PSRemoting or Set-WSManQuickConfig:

Set-WSManQuickConfig : <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2150859113"
Machine="localhost"><f:Message><f:ProviderFault provider="Config provider"
path="%systemroot%\system32\WsmSvc.dll"><f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault"
Code="2150859113" Machine="win81"><f:Message>WinRM firewall exception will not work since one of the network
connection types on this machine is set to Public. Change the network connection type to either Domain or Private and
try again. </f:Message></f:WSManFault></f:ProviderFault></f:Message></f:WSManFault>
At line:1 char:1
+ Set-WSManQuickConfig -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Set-WSManQuickConfig], InvalidOperationException
    + FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.SetWSManQuickConfigCommand

This post will explain how to get around this error. There are different ways to do this depending on your operating system version. Windows 8/2012 workarounds are fairly sane while windows 7/2008R2 may seem a bit obtuse.

This post is inspired by an experience I had this week trying to get a customer's Chef node able to connect via WinRM over SSL. Her test node was running Windows 7 and she was getting the above error when trying to enable WinRM. Windows 7 has no way to change the connection Type via a native powershell cmdlet. I had done this via the commandline before on Windows 7 but did not have the commands handy. Further, it had been so long since changing the connection type on windows 7 via the GUI, I had to fire up my own win 7 VM and run through it just so I could relay propper instructions to this very patient customer.

So I write this to run through the different nuances of connection types on different operating systems but primarily to have a known place on the internet where I can stash the commands.

TL;DR for Windows 7 or 2008R2 Clients:

If you dont care about anything else other than getting past this error on windows 7 or 2008R2 without ceremonial pointing and clicking, simply run these commands:

$networkListManager = [Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]"{DCB00C01-570F-4A9B-8D69-199FDBA5723B}")) 
$connections = $networkListManager.GetNetworkConnections() 

# Set network location to Private for all networks 
$connections | % {$_.GetNetwork().SetCategory(1)}

This works on windows 8/2012 and up as well but there are much friendlier commands you can run instead.  Unless you are one partial to GUIDs.

Connection Types - what does it mean?

Windows provide different connection type profiles (or Network Locations) each with different levels of restrictions on what connections can be granted to the local computer on the network.

I have personally always found these types to be confusing yet well meaning. You are perhaps familiar with the message presented the first time you logon to a machine asking you if you would like the computer to be discoverable on the internet. If you choose no, the network interface is given a public internet connection profile. If you choose "yes" then it is private. For me the confusion is that I equate "public" with "publicly accessible" but here the opposite applies.

Public network locations have Network Discovery turned off  and restrict your firewall for some applications. You cannot create or join Homegroups with this setting. WinRM firewall exception rules also cannot be enabled on a public network. Your network location must be private in order for other machines to make a WinRM connection to the computer.

Domain Networks

If your computer is on a domain, that is an entirely different network location type. On a domain network, the accessibility of the machine is governed by your domain policies. This network location type is automatically chosen if your machine is part of an Active Directory domain.

Working around Public network restrictions on Windows 8 and up

When enabling WinRM, client SKUs of windows (8, 8.1, 10) expose an additional setting that allow the machine to be discoverable over WinRM publicly but only on the same subnet. By using the -SkipNetworkProfileCheck switch of Enable-PSRemoting or Set-WSManQuickConfig you can still allow connections to your computer but those connections must come from other machines on the same subnet.

Enable-PSRemoting -SkipNetworkProfileCheck

So this can work for local VMs but will still be restrictive for cloud based VMs.

Changing the Network Location

Once you answer yes or no the initial question of whether you want to be discovered, you are never prompted again, but you can change this setting later.

This is a family friendly blog so I am not going to cover how to change the Network Location via the GUI. It can be done but you are a dirty person for doing so (full disclosure - I have been guilty of doing this).

Windows 8/2012 and up

Powershell version 3 and later expose cmdlets that allow you to see and change your Network Location. Get-NetConnectionProfile shows you the network location of all network interfaces:

PS C:\Windows\system32> Get-NetConnectionProfile


Name             : Network  2
InterfaceAlias   : Ethernet
InterfaceIndex   : 3
NetworkCategory  : Private
IPv4Connectivity : Internet
IPv6Connectivity : LocalNetwork

Note the NetworkCategory above. The Network Location is private.

Use the Set-NetConnectionProfile to change the location type:

Set-NetConnectionProfile -InterfaceAlias Ethernet -NetworkCategory Public

or

Get-NetConnectionProfile | Set-NetConnectionProfile -NetworkCategory Private

The later is convenient if you want to ensure that all network interfaces are set to a particular location.

Windows 7 and 2008R2

You will not have the above cmdlets available on Windows 7 or 2008R2. You can still change the location on the command line but the commands are far less intuitive. As shown in the tl;dr, here is the command:

$networkListManager = [Activator]::CreateInstance([Type]::GetTypeFromCLSID([Guid]"{DCB00C01-570F-4A9B-8D69-199FDBA5723B}")) 
$connections = $networkListManager.GetNetworkConnections() 

# Set network location to Private for all networks 
$connections | % {$_.GetNetwork().SetCategory(1)}

First we get a reference to a COM instance of an INetworkListManager which naturally has a Class ID of DCB00C01-570F-4A9B-8D69-199FDBA5723B. We then grab all the network connections and finally set them all to the desired location:

  • 0 - Public
  • 1 - Private
  • 2 - Domain