Certificate (password-less) based authentication in WinRM by Matt Wrock

This week the WinRM ruby gem version 1.8.0 released adding support for certificate authentication. Many thanks to the contributions of @jfhutchi and @elpetak that make this possible. As I set out to test this feature, I explored how certificate authentication works in winrm using native windows tools like powershell remoting. My primary takeaway was that it was not at all straightforward to setup. If you have worked with similar authentication setups on linux using SSH commands, be prepared for more friction. Most of this is simply due to the lack of documentation and google results (well now there is one more). Regardless, I still think that once setup, authentication via certificates is a very good thing and many are not aware that this is available in WinRM.

This post will walk through how to configure certificate authentication, enumerate some of the "gotchas" and pitfalls one may encounter along the way and then explain how to use certificate authentication using Powershell Remoting as well as via the WinRM ruby gem which opens up the possibility of authenticating from a linux client to a Windows WinRM endpoint.

Why should I care about certificate based authentication?

First lets examine why certificate authentication has value. What's wrong with usernames and passwords? In short, certificates are more secure. I'm not going to go too deep here but here are a few points to consider:

  • Passwords can be obtained via brute force. You can protect against this by having longer, more complex passwords and changing them frequently. Very few actually do that, and even if you do, a complex password is way easier to break than a certificate. Its nearly impossible to brute force a private key of sufficient strength.
  • Sensitive data is not being transferred over the wire. You are sending a public key and if that falls into the wrong hands, no harm done.
  • There is a stronger trail of trust establishing that the person who is seeking authentication is in fact who they say they are given the multi layered process of having a generated certificate signed by a trusted certificate authority.

Its still important to remember that nothing may be able to protect us from sophisticated aliens or time traveling humans from the future. No means of security is impenetrable.

Not as convenient as SSH keys

So one reason some like to use certificates over passwords in SSH scenarios is ease of use. There is a one time setup "cost" of sending your public key to the remote server, but:

  1. SSH provides command line tools that make this fairly straight forward to setup from your local environment.
  2. Once setup, you just need to initiate an ssh session to the remote server and you don't have to hassle with entering a password.

Now the underlying cryptographic and authentication technology is no different using winrm, but both the initial setup and the "day to day" use of using the certificate to login is more burdensome. The details of why will become apparent throughout this post.

One important thing to consider though is that while winrm certificate authentication may be more burdensome, I don't think the primary use case is for user interactive login sessions (although that's too bad). In the case of automated services that need to interact with remote machines, these "burdens" simply need to be part of the connection automation and its just a non sentient cpu that does the suffering. Lets just hope they won't hold a grudge once sentience is obtained.

High level certificate authentication configuration overview

Here is a run down of what is involved to get everything setup for certificate authentication:

  1. Configure SSL connectivity to winrm on the endpoint
  2. Generate a user certificate used for authentication
  3. Enable Certificate authentication on the endpoint. Its disabled by default for server auth and enabled on the client side.
  4. Add the user certificate and its issuing CA certificate to the certificate store of the endpoint
  5. Create a user mapping in winrm with the thumbprint of the issuing certificate on the endpoint.

I'll walk through each of these steps here. When the above five steps are complete, you should be able to connect via certificate authentication using powershell remoting or using the ruby or python open source winrm libraries.

Setting up the SSL winrm listener

If you are using certificate authentication, you must use a https winrm endpoint. Attempts to authenicate with a certificate using http endpoints will fail. You can setup SSL on the endpoint with:

$ip="192.168.137.169" # your ip might be different
$c = New-SelfSignedCertificate -DnsName $ip `
                               -CertStoreLocation cert:\LocalMachine\My
winrm create winrm/config/Listener?Address=*+Transport=HTTPS "@{Hostname=`"$ip`";CertificateThumbprint=`"$($c.ThumbPrint)`"}"
netsh advfirewall firewall add rule name="WinRM-HTTPS" dir=in localport=5986 protocol=TCP action=allow

Generating a client certificate

Client certificates have two key requirements:

  1. An Extended Key Usage of Client Authentication
  2. A Subject Alternative Name with the UPN of the user.

Only ADCS certificates work from Windows 10/2012 R2 clients via powershell remoting

This was the step that I ended up spending the most time on. I continued to receive errors saying my certificate was malformed:

new-PSSession : The WinRM client cannot process the request. If you are using a machine certificate, it must contain a DNS name in the Subject Alternative Name extension or in the Subject Name field, and no UPN name. If you are using a user certificate, the Subject Alternative Name extension must contain a UPN name and must not contain a DNS name. Change the certificate structure and try the request again.

I was trying to authenticate from a windows 10 client using powershell remoting. I don't typically work or test in a domain environment and don't run an Active Directory Certificate Services authority. So I wanted to generate a certificate using either New-SelfSignedCertificate or OpenSSL.

In short here is the bump I hit: powershell remoting from a windows 10 or windows 2012 R2 client failed to authenticate with certificates generated from OpenSSL or New-SelfSignedCertificate. However these same certificates succeed to authenticate from windows 7 or windows 2008 R2. They only worked on Windows 10 and 2012 R2 if I used the ruby WinRM gem instead of powershell remoting. Note that while I tested on windows 10 and 2012 R2, I'm sure that windows 8.1 suffers the same issue. The only certificates I got to work on windows 10 and 2012 R2 via powershell remoting were created via an Active Directory Certificate Services Enterprise CA .

So unless I can find out otherwise, it seems that you must have access to an Enterprise root CA in Active Directory Certificate Services and have client certificates issued in order to use certificate authentication from powershell remoting on these later windows versions. If you are using ADCS, the stock client template should work.

Generating client certificates via OpenSSl

As stated above, certificates generated using OpenSSL or New-SelfSignedCertificate did not work using powershell remoting from windows 10 or 2012 R2. However, if you are using a previous version of windows or if you are using another client (like the ruby or python libraries), then these other non-ADCS methods will work fine and do not require the creation of a domain controller and certificate authority servers.

If you do not already have OpenSSL tools installed, you can get them via chocolatey:

cinst openssl.light -y

Then you can run the following powershell to generate a correctly formatted user certificate which was adapted from this bash script:

function New-ClientCertificate {
  param([String]$username, [String]$basePath = ((Resolve-Parh .).Path))

  $OPENSSL_CONF=[System.IO.Path]::GetTempFileName()

  Set-Content -Path $OPENSSL_CONF -Value @"
  distinguished_name = req_distinguished_name
  [req_distinguished_name]
  [v3_req_client]
  extendedKeyUsage = clientAuth
  subjectAltName = otherName:1.3.6.1.4.1.311.20.2.3;UTF8:$username@localhost
"@

  $user_path = Join-Path $basePath user.pem
  $key_path = Join-Path $basePath key.pem
  $pfx_path = Join-Path $basePath user.pfx

  openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out $user_path -outform PEM -keyout $key_path -subj "/CN=$username" -extensions v3_req_client 2>&1

  openssl pkcs12 -export -in $user_path -inkey $key_path -out $pfx_path -passout pass: 2>&1

  del $OPENSSL_CONF
}

This will output a certificate and private key file both in base64 .pem format and additionally a .pfx formatted file.

Generating client certificates via New-SelfSignedCertificate

If you are on windows 10 or server 2016, then you should have a more advanced version of the New-SelfSignedCertificate cmdlet - more advaced than what shipped with windows 2012 R2 and 8.1. Here is the command to generate the script:

New-SelfSignedCertificate -Type Custom `
                          -Container test* -Subject "CN=vagrant" `
                          -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.2","2.5.29.17={text}upn=vagrant@localhost") `
                          -KeyUsage DigitalSignature,KeyEncipherment `
                          -KeyAlgorithm RSA `
                          -KeyLength 2048

This will add a  certificate for a vagrant user to the personal LocalComputer folder in the certificate store.

Enable certificate authentication

This is perhaps the simplest step. By default, certificate authentication is enabled for clients and disabled for server. So you will need to enable it on the endpoint:

Set-Item -Path WSMan:\localhost\Service\Auth\Certificate -Value $true

Import the certificate to the appropriate certificate store locations

If you are using powershell remoting, the user certificate and its private key should be in the My directory of either the LocalMachine or the CurrentUser store on the client. If you are using a cross platform library like the ruby or python library, the cert does not need to be in the store at all on the client. However, regardless of client implementation, it must be added to the server certificate store.

Importing on the client

As stated above, this is necessary for powershell remoting clients. If you used ADCS or New-SelfSignedCertificate, then the generated certificate is added automatically. However if you used OpenSSL, you need to import the .pfx yourself:

Import-pfxCertificate -FilePath user.pfx `
                      -CertStoreLocation Cert:\LocalMachine\my

Importing on the server

There are two steps to timporting the certificate on the endpoint:

  1. The issuing certificate must be present in the Trusted Root Certification Authorities of the LocalMachine store
  2. The client certificate public key must be present in the Trusted People folder of the LocalMachine store

Depending on your setup, the issuing certificate may already be in the Trusted Root location. This is the certificate used to issue the client cert. If you are using your own enterprise certificate authority or a publicly valid CA cert, its likely you already have this in the trusted roots. If you used OpenSSL or New-SelfSignedCertificate then the user certificate was issued by itself and needs to be imported.

If you used OpenSSL, you already have the .pem public key. Otherwise you can export it:

Get-ChildItem cert:\LocalMachine\my\7C8DCBD5427AFEE6560F4AF524E325915F51172C |
  Export-Certificate -FilePath myexport.cer -Type Cert

This assumes that 7C8DCBD5427AFEE6560F4AF524E325915F51172C is the thumbprint of your issuing certificate. I guarantee that is an incorrect assumption.

Now import these on the endpoint:

Import-Certificate -FilePath .\myexport.cer `
                   -CertStoreLocation cert:\LocalMachine\root
Import-Certificate -FilePath .\myexport.cer `
                   -CertStoreLocation cert:\LocalMachine\TrustedPeople

Create the winrm user mapping

This will declare on the endpoint: given a issuing CA, which certificates to allow access. You can potentially add multiple entries for different users or use a wildcard. We'll just map our one user:

New-Item -Path WSMan:\localhost\ClientCertificate `
         -Subject 'vagrant@localhost' `
         -URI * `
         -Issuer 7C8DCBD5427AFEE6560F4AF524E325915F51172C `
         -Credential (Get-Credential) `
         -Force

Again this assumes the issuing certificate thumbprint of the certificate that issued our user certificate is 7C8DCBD5427AFEE6560F4AF524E325915F51172C and we are allowing access to a local account called vagrant. Note that if your user certificate is self-signed, you would use the thumbprint of the user certificate itself.

Using certificate authentication

This completes the setup. Now we should actually be able to login remotely to the endpoint. I'll demonstrate this first using powershell remoting and then ruby.

Powershell remoting

C:\dev\WinRM [master]>Enter-PSSession -ComputerName 192.168.137.79 `
>> -CertificateThumbprint 7C8DCBD5427AFEE6560F4AF524E325915F51172C
[192.168.137.79]: PS C:\Users\vagrant\Documents>

Ruby WinRM gem

C:\dev\WinRM [master +3 ~0 -0 !]> gem install winrm
WARNING:  You don't have c:\users\matt\appdata\local\chefdk\gem\ruby\2.1.0\bin in your PATH,
          gem executables will not run.
Successfully installed winrm-1.8.0
Parsing documentation for winrm-1.8.0
Done installing documentation for winrm after 0 seconds
1 gem installed
C:\dev\WinRM [master +3 ~0 -0 !]> irb
irb(main):001:0> require 'winrm'
=> true
irb(main):002:0> endpoint = 'https://192.168.137.169:5986/wsman'
=> "https://192.168.137.169:5986/wsman"
irb(main):003:0> puts WinRM::WinRMWebService.new(
irb(main):004:1*   endpoint,
irb(main):005:1*   :ssl,
irb(main):006:1*   :client_cert => 'user.pem',
irb(main):007:1*   :client_key => 'key.pem',
irb(main):008:1*   :no_ssl_peer_verification => true
irb(main):009:1> ).create_executor.run_cmd('ipconfig').stdout

Windows IP Configuration


Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : mshome.net
   Link-local IPv6 Address . . . . . : fe80::6c3f:586a:bdc0:5b4c%12
   IPv4 Address. . . . . . . . . . . : 192.168.137.169
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.137.1

Tunnel adapter Local Area Connection* 12:

   Connection-specific DNS Suffix  . :
   IPv6 Address. . . . . . . . . . . : 2001:0:5ef5:79fd:24bc:3d4c:3f57:7656
   Link-local IPv6 Address . . . . . : fe80::24bc:3d4c:3f57:7656%14
   Default Gateway . . . . . . . . . : ::

Tunnel adapter isatap.mshome.net:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : mshome.net
=> nil
irb(main):010:0>

Interested?

This functionality just became available in the winrm gem 1.8.0 this week. This gem is used by Vagrant, Chef and Test-Kitchen to connect to remote machines. However, none of these applications provide configuration options to make use of certificate authentication via winrm. My personal observation has been that nearly no one uses certificate authentication with winrm but that may be a false observation or a result of the fact that few no about this possibility.

If you are interested in using this in Chef, Vagrant or Test-Kitchen, please file an issue against their respective github repositories and make sure to @ mention me (@mwrock) and I'll see what I can do to plug this in or you can submit a PR yourself if so inclined.

Installing and running a Chef client on Windows Nano Server by Matt Wrock

I've been really excited about Nano Server ever since it was first talked about last year. It's a major shift in how we "do windows" and has the promise of addressing many pain points that exist today for those who run Windows Server farms at scale. As a Chef enthusiast and employee I am thrilled to be apart of the Journey to making Chef on Nano a delight. Now note the word "journey" here. We are on the road to awesome  but we've just laid down the foundation and scaffolding. Heck, even Nano itself is still evolving as I write this and is on it's own journey to RTM. So while today I would not characterize the experience of Chef on Nano as "delightful," I can say that it is "possible." For those who enjoy living on the bleeding edge, "possible" is a significant milestone on the way to "delight."

In this post I will share what Nano is, how to get it and run it on your laptop, install a chef client and run a basic recipe. I'll also point out some unfinished walls and let you know some of the rooms that are entirely missing. Also bear in mind that I have not toured the entire house so you will certainly find other gaps between here and delight and hopefully you will share. Keep in mind that I am a software developer and not a product owner at Chef. So I won't be talking about road maps or showing any gnat charts here. I'm really speaking more as a enthusiast here and as someone who can't wait to use it more and make the experience of running Chef on Nano awesome like a possum because possums are awesome. (They really are. We have one living under our front porch and its just so gosh darn cute!)

What is Nano and why should I care?

Some of you may be thinking, "finally a command line text editor for windows!" Yeah that would be great but no I'm not talking about the GNU Nano text editor familiar to many linux users, I am talking about Windows Nano Server. You have likely heard of Windows Server Core. Well this takes core to the next level (or two). While Server Core does not support GUI apps (except the ones it does) but still supports all applications and APIs available on GUI based Windows Server, Nano throws backwards compatibility to the wind and slashes vast swaths of what sits inside a Windows server today widdling it down to a few hundred megabytes.

There is no 32 bit subsystem only 64 bit is supported. While traditional Windows server has multiple APIs for doing the same thing, Nano pairs much of that down throwing out many APIs where you can accomplish the same tasks using another API. Nope, no VBScript engine here. All those windows roles and features that lie dormant on a traditional Windows server simply don't exist on Nano. You start in a bare core and then add what you need.

Naturally this is not all roses and sunshine. The road to Nano will have moments of pain and frustration. We have grown accustomed to our bloated Windows and will need to adjust our workflows and in some case our application architecture to exist in this new environment. Here is one gotcha many may hit pretty quickly. Maybe you pull down the Couchbase MSI and try to run MSIEXEC on that thing. Well guess what...on Nano "The term 'msiexec' is not recognized as the name of a cmdlet, function, script file, or operable program." Nano will introduce new ways to install applications.

But before we get all cranky about the loss of MSIs (yeah the "packaging" system that wasn't really) lets think about what this smaller surface area gets us. First an initial image file that is often less than 300MB compared to what would be 4GB on 2012R2. With less junk in the trunk, there is less to update, so no more initial multi hour install of hundreds of updates. Fewer updates will mean fewer reboots. All of this is good stuff. Nano also boots up super duper quick. The first time you provision a Nano server, you may think you have done something wrong - I did - because it all happens so much more quickly than what you are used to.

Info graphic provided by Microsoft

Info graphic provided by Microsoft

Sounds great! How do I go to there?

There are several ways to get a nano server up and running and I'll outline several options here:

  • Download the latest Windows Server 2016 Technical Preview and follow these instructions to extract and install a Nano image.
  • Download just a VHD and import that into Hyper-V (user: administrator/pass: Pass@word1). While it successfully installs on VirtualBox, you will not be able to successfully to remote to the box. 
  • Are you a Vagrant user? Feel free to "vagrant up" my mwrock/WindowsNano Nano eval box (user: vagrant/pass: vagrant). It has a Hyper-V and VirtualBox provider. The Hyper-V provider may "blue screen" on first boot but should operate fine after rebooting a couple times.
  • Don't want to use some random dude's box but want to make your own with Packer? Check out my post where I walk through how to do that.

Depending on wheather you are a Vagrant user or VirtualBox user or not, options 2 and 3 are by far the easiest and should get you up and running in minutes. The longest part is just downloading the image but even that is MUCH faster than a standard windows ISO. About 6 minutes compared to 45 minutes on my FIOS home connection.

Preparing your environment

Basically we just need to make sure we can remote to the nano server and also copy files to it.

Setup a Host-Only network on VirtualBox

You will need to be able to access the box with a non localhost IP in order to transfer files over SMB. This only applies if you are using a NAT VirtualBox network (the default). Hyper-V Switches should require no additional configuration.

First create a Host-Only network if your VirtualBox host does not already have one:

VBoxManage hostonlyif create

Now add a NIC on your Nano guest that connects over the Host-Only network:

VBoxManage modifyvm "VM name" --nic2 hostonly

Now log on to the box to discover its IP address. Immediately after login, you should see its two IPs. One starts with 10. and the other starts with 172. You will want to use the 172 address.

Make sure you can remote to the Nano box

We'll establish a remote powershell connection to the box. If you can't do this now, we won't get far with Chef. So run the following:

$ip = "<ip address of Nano Server>"
Set-Item WSMan:\localhost\Client\TrustedHosts $ip
Enter-PSSession -ComputerName $ip -Credential $ip\Administrator

This should result in dropping you into an interactive remote powershell session on the nano box:

[172.28.128.3]: PS C:\Users\vagrant\Documents>

Open File and Print sharing Firewall Rule

If you are using my vagrant box or packer templates, this should already be setup. Otherwise, you will need to open the File and Print sharing rule so we can mount a drive to the Nano box. Don't you worry, we won't be doing any printing.

PS C:\dev\nano> Enter-PSSession -ComputerName 172.28.128.3 -Credential vagrant
[172.28.128.3]: PS C:\Users\vagrant\Documents> netsh advfirewall firewall set rule group="File and Printer Sharing" new enable=yes

Updated 16 rule(s).
Ok.

Mount a network drive from the host to the Nano server

We'll just create a drive that connects to the root of the C: drive on the Nano box:

PS C:\dev\nano> net use z: \\172.28.128.3\c$
Enter the user name for '172.28.128.3': vagrant
Enter the password for 172.28.128.3:
The command completed successfully.

PS C:\dev\nano> dir z:\


    Directory: z:\


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       11/21/2015   2:47 AM                Program Files
d-----       11/21/2015   2:47 AM                Program Files (x86)
d-----       11/21/2015   2:47 AM                ProgramData
d-----       11/21/2015   2:47 AM                Users
d-----       11/21/2015  10:50 AM                Windows

Ok. If you have gotten this far, you are ready to install Chef.

Installing Chef

Remember earlier in this post when I mentioned that there is no msiexec on Nano? We can't just download the MSI on Nano and install it like we normally would. There are some pre steps we will need to follow and we also need to make sure to use just the right Chef version.

Downloading the right version of Chef

If you remember, not only does Nano lack the ability to install MSIs, it also lacks a 32 bit subsystem. Lucky for us, Chef now ships a 64 bit Windows Chef client. On downloads.chef.io, the last few versions of the chef client provide a 64bit Windows install. However, don't get the latest one. The last couple versions have a specially compiled version of ruby that is not compatible with Nano server. This will be remedied in time, but for now, download the 64 bit install of Chef 12.7.2.

PS C:\dev\nano> Invoke-WebRequest https://packages.chef.io/stable/windows/2008r2/chef-client-12.7.2-1-x64.msi -OutFile Chef-Client.msi

Extract the Chef client files from the MSI

There are a couple ways to do this. The most straight forward is to simply install the MSI locally on your host. That will install the 64 bit Chef client, and in the process, extract its files locally.

Another option is to use a tool like LessMSI which can extract the contents without actually running the installer. This can be beneficial because it won't run installation code that adjusts your registry, changes environment variables or changes any other state on your local host that you would rather leave in tact.

You can grab lessmsi from chocolatey:

choco install lessmsi

Now extract the Chef MSI contents:

lessmsi x .\chef-client.msi c:\chef-client\

This extracts the contents of the MSI to c:\chef-client. What we are interested in now is the extracted C:\chef-client\SourceDir\opscode\chef.zip. That has the actual files we will copy to the Nano server. Lets expand that zip:

Expand-Archive -Path C:\chef-client\SourceDir\opscode\chef.zip -DestinationPath c:\chef-client\chef

Note I am on Windows 10. The Expand-Archive command is only available in powershell version 5. However if you are on a previous version of powershell, I am confident you can expand the zip using other methods.

Once this zip file is extracted by whatever means, you should now have a full Chef install layout at c:\chef-client\chef.

Editing win32-process gem before copying to Nano

Here's another area on "undelight." The latest version of the win32-process gem introduced coverage of some native kernel API functions that were not ported to Nano. So attempts to attach to these C functions results in an error when the gem is loaded. Since win32-process is a dependency of the chef client, loading the chef client will blow up.

Not to worry, we can perform some evasive maneuvers to work around this. Its gonna get dirty here, but I promise no one is looking. There are basically four lines in the win32-process source that we simply need to remove:

PS C:\dev\nano> (Get-content -Path C:\chef-client\chef\embedded\lib\ruby\gems\2.0.0\gems\win32-process-0.8.3\lib\win32\proces
s\functions.rb) | % { if(!$_.Contains(":Heap32")) { $_ }} | Set-Content C:\chef-client\chef\embedded\lib\ruby\gems\2.0.0\gems
\win32-process-0.8.3\lib\win32\process\functions.rb

Copy the Chef files to the Nano server

Now we are ready to copy over all the chef files to the Nano Server:

 Copy-Item C:\chef-client\chef\ z:\chef -Recurse

Validate the chef install

Lets set the path to include the chef ruby and bin files and then call chef-client -v:

PS C:\dev\nano> Enter-PSSession -ComputerName 172.28.128.3 -Credential vagrant
[172.28.128.3]: PS C:\Users\vagrant\Documents> $env:path += ";c:\chef\bin;c:\chef\embedded\bin"
[172.28.128.3]: PS C:\Users\vagrant\Documents> chef-client -v
Chef: 12.7.2

Hopefully you see similar output including the version of the chef-client.

Converging on Nano

Now we are actually ready to "do stuff" with Chef on Nano. Lets start simple. We will run chef-apply on a trivial recipe:

file "c:/blah.txt" do
  content 'blah'
end

Lets copy the recipe and converge:

[172.28.128.3]: PS C:\Users\vagrant\Documents> chef-apply c:/chef/blah.rb
[2016-03-20T07:49:38+00:00] WARN: unable to detect ipaddress
[2016-03-20T07:49:45+00:00] INFO: Run List is []
[2016-03-20T07:49:45+00:00] INFO: Run List expands to []
[2016-03-20T07:49:45+00:00] INFO: Processing file[c:/blah.txt] action create ((chef-apply cookbook)::(chef-apply recipe) line
 1)
[2016-03-20T07:49:45+00:00] INFO: file[c:/blah.txt] created file c:/blah.txt
[2016-03-20T07:49:45+00:00] INFO: file[c:/blah.txt] updated file contents c:/blah.txt
[172.28.128.3]: PS C:\Users\vagrant\Documents> cat C:\blah.txt
blah

That worked! Our file is created and we can see it has the intended content.

Bootstrapping a Nano node on a Chef server

Maybe you want to join a fleet of nano servers to a chef server. Well at the moment knife bootstrap windows throws encoding errors. This could be related to the fact that the codepage used by a Nano shell is UTF-8 unlike the "MS-DOS"(437) codepage used on all previous versions of windows.

Lets just manually bootstrap for now:

knife node create -d nano
# We use ascii encoding to avoid a UTF-8 BOM
knife client create -d nano | Out-File -FilePath z:\chef\nano.pem -Encoding ascii
knife acl add client nano nodes nano update

Note that the knife acl command requires the knife-acl gem that is not shipped with chef-dk but you can simply chef gem install it.

Now lets create a basic client.rb file in z:\chef\client.rb:

log_level        :info
log_location     STDOUT
chef_server_url  'https://api.opscode.com/organizations/ment'
client_key 'c:/chef/nano.pem'
node_name  'nano'

Now lets converge against our server:

[172.28.128.3]: PS C:\Users\vagrant\Documents> chef-client
[2016-03-20T18:28:07+00:00] INFO: *** Chef 12.7.2 ***
[2016-03-20T18:28:07+00:00] INFO: Chef-client pid: 488
[2016-03-20T18:28:08+00:00] WARN: unable to detect ipaddress
[2016-03-20T18:28:15+00:00] INFO: Run List is []
[2016-03-20T18:28:15+00:00] INFO: Run List expands to []
[2016-03-20T18:28:15+00:00] INFO: Starting Chef Run for nano
[2016-03-20T18:28:15+00:00] INFO: Running start handlers
[2016-03-20T18:28:15+00:00] INFO: Start handlers complete.
[2016-03-20T18:28:16+00:00] INFO: Loading cookbooks []
[2016-03-20T18:28:16+00:00] WARN: Node nano has an empty run list.
[2016-03-20T18:28:16+00:00] INFO: Chef Run complete in 1.671495 seconds
[2016-03-20T18:28:16+00:00] INFO: Running report handlers
[2016-03-20T18:28:16+00:00] INFO: Report handlers complete
[2016-03-20T18:28:16+00:00] INFO: Sending resource update report (run-id: 1fb8ae2a-8455-4b94-b956-99c5a34863ea)
[172.28.128.3]: PS C:\Users\vagrant\Documents>

We can use chef-client -r recipe[windows] if you have the windows cookbook on your server to converge its default recipe.

What else is missing?

The other gaping hole here is Test-Kitchen cannot converge a Nano test instance. Two main reasons prevent this:

  1. There is no provisioner that can get around the MSI install issue
  2. Powershell calls from the winrm gem will fail because Nano's Powershell.exe does not accept the -EncodedCommand argument.

One might think that the first issue could probably be worked around by creating a custom provisioner to extract the chef files on the host and copy them to the instance. However, because winrm-fs, used by Test-Kitchen to copy files to a windows test instance, makes ample use of the -EncodedCommand argument. Shawn Neal and I are currently working on a v2 of the winrm gem that will use a more modern protocol for powershell calls to get around the -EncodedCommand limitation and provide many other benefits as well.

Try it out!

Regardless of the missing pieces, as you can see, one can at least play with Chef and Nano today and get a glimpse of what life might be like in the future. Its no flying car but it beats our multi gigabyte Windows images of today.

Run Kitchen tests in Travis and Appveyor by Matt Wrock

There are a few cookbook repositories I help to maintain and services like Travis and Appveyor provide a great enhancement to the pull request workflow both for contributors and maintainers. As a contributor I get fast feedback without bothering any human on whether my PR meets a minimum bar of quality and complies with the basic code style standards of the maintainers. As a maintainer, I can more easily triage incoming contributions and simply not waste time reviewing failed commits.

Its common to use travis and appveyor for unit tests and even longer running functional tests as long as they take a reasonable amount of time to run. However, in the Chef cookbook world, you usually do not see them include Test-Kitchen tests which are the cookbook equivalent of end to end functional or integration tests. I have found myself wishing for an easy way to kick off cloud based Test-Kitchen runs triggered by pull request pushes. Often there just is no way to know for sure if a commit breaks a cookbook or if it "works" unless I manually pull down the PR and invoke the kitchen test (usually via vagrant). This takes time and I wish I could just see a green check mark or red 'X' automatically without doing anything.

This week while reviewing PRs for the Chocolatey cookbook, I set out to run the kitchen tests in appveyor and did not see any readily available way to do so until Stuart Preston steered me to the proxy driver included with test-kitchen.

In this post I'll talk about some other patterns folks have used to run kitchen tests in travis, why I went with the proxy driver,  and how to go about using it in your cookbook repository.

Using Docker

One approach is to install and start docker inside the travis VM and then use kitchen-docker or kitchen-dokken to run them in the travis run. This is a very viable approach. You can fire up a docker container very quickly and most cookbooks will run fine containerized. There are a few downsides:

  • Installing and configuring docker while fairly straight forward may require several lines of setup script.
  • Some cookbooks may not run well in containers.
  • Not a viable approach for windows based cookbooks. Looking forward to when they are.

Leveraging AWS or other cloud alternatives

Another popular approach is to run the kitchen tests from travis but reach out to another cloud provider to host the actual test instances. This also has its drawbacks:

  • Unless using the aws free tier, its not free and if you are, its not fast.
  • You have to stash keys in your travis.yml

I have an ephemeral machine, why cant I use it?

Both of the above solutions don't sit well for me because it just seems sub optimal to have to bring up another "instance" whether it be container or cloud based when travis or appveyor just did that for me. The beauty of these services is that they bring up an isolated and full fledged test VM so why do I need to bring up another one?

Most kitchen drivers are built to go "somewhere else"

This usually makes perfect sense. I don't want to run kitchen tests and have the test instance BE my machine because the state of that machine will be changed and I want to remain isolated from those changes and easily undo them. However in travis or appveyor, I AM somewhere else.

Yet having looked, I found no kitchen driver that would converge and test locally. The closest thing was kitchen-ssh which simply communicates with a test instance over SSH and does not try to spin up a vm. Surely one can easily just SSH to localhost. However, its just SSH and does not also leverage WinRM when talking to windows.

Enter the built in proxy driver

I wanted a driver that could run commands using whatever kitchen transport was configured (SSH or WinRM) but would not try to interact with a hypervisor or cloud to start a separate instance. If I just point the transport to localhost, that should succeed in running locally. Of course a "local" transport that would run native shell commands locally and use the native filesystem for file operations would be one step better. However, pointing SSH and WinRM to localhost seems to work just fine and requires no additional work.

Using the proxy driver

The chocolatey-cookbook repository includes a "real world" example that uses appveyor. I'll also briefly walk through both appveyor and travis samples here.

The .kitchen.yml or .kitchen.{travis or appveyor}.yml file

First we'll look at the .kitchen.yml file that may be the same for either travis or appveyor. Most actual cookbook repositories will likely use .kitchen.travis.yml or .kitchen.appveyor.yml since they will want to use .kitchen.yml for locally run tests.

---
driver:
  name: proxy
  host: localhost
  reset_command: "exit 0"
  port: <%= ENV["machine_port"] %>
  username: <%= ENV["machine_user"] %>
  password: <%= ENV["machine_pass"] %>

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-14.04
  - name: windows-2012R2

verifier:
  name: inspec

suites:
  - name: default
    run_list:
      - recipe[machine_test]

The driver section uses the proxy driver and configures a host, port username and password for the transport. The host setting is required and we will use "localhost". It also uses a reset_command which can be used for running a command on the instance during the "create" phase but we don't need it to do anything on a fresh appveyor or travis instance. The reset_command is required so we just specify "exit 0" which should work cross platform to do nothing.

The transport being used is determined by the same kitchen logic, either the transport is explicitly declared in the YAML or kitchen will default to SSH unless the platform name starts with "win." This example configures the credential and port from environment variables. That will be more clear when we look at the .travis.yml and appveyor.yml files.

.travis.yml

language: ruby

env:
  global:
    - machine_user=travis
    - machine_pass=travis
    - machine_port=22
    - KITCHEN_YAML=.kitchen.travis.yml

rvm:
  - 2.1.7

sudo: required
dist: trusty

before_install:
  - sudo usermod -p "`openssl passwd -1 'travis'`" travis

script:
  - bundle exec rake
  - bundle exec kitchen verify ubuntu

branches:
  only:
  - master

Here we set the port, user name and password environment variables and we set the KITCHEN_YAML variable to our special travis targeted kitchen.yml. We also give the travis user (what travis runs under) a password. Our test "script" runs kitchen verify against the ubuntu platform.

appveyor.yml

version: "master-{build}"

os: Windows Server 2012 R2
platform:
  - x64

environment:
  machine_user: test_user
  machine_pass: Pass@word1
  machine_port: 5985
  KITCHEN_YAML: .kitchen.appveyor.yml
  SSL_CERT_FILE: c:\projects\kitchen-machine\certs.pem

  matrix:
    - ruby_version: "21"

clone_folder: c:\projects\kitchen-machine
clone_depth: 1
branches:
  only:
    - master

install:
  - ps: net user /add $env:machine_user $env:machine_pass
  - ps: net localgroup administrators $env:machine_user /add
  - ps: $env:PATH="C:\Ruby$env:ruby_version\bin;$env:PATH"
  - ps: gem install bundler --quiet --no-ri --no-rdoc
  - ps: Invoke-WebRequest -Uri http://curl.haxx.se/ca/cacert.pem -OutFile c:\projects\kitchen-machine\certs.pem

build_script:
  - bundle install || bundle install || bundle install

test_script:
  - bundle exec kitchen verify windows

This one is a bit more involved but still not too bad. Like we did in the travis.yml, we assign our port, username and password environment variables and also set the .kitchen.yml file override. Here we actually go ahead and create a new user and make it an administrator. This is because its tough to get at the appveyor user's password and there may be issues changing its password in the middle of an active session.

Another thing to note for windows is we need to do some special SSL cert setup. Ruby and openssl do not use the native windows certificate store to manage certificate authorities; so we go ahead and download them and point openssl at them with the SSL_CERT_FILE variable. Those who use the Chef-dk can be thankful that hides this detail from you, but there is no chef-dk in this environment.

Finally our test_script invokes all tests in the windows platform.

The Results

Now I can see a complete Test-Kitchen log of converged resources and verifier test results right in my travis and appveyor build output.

Here is the end of the travis log:

And here is appveyor:


Need an SSH client on Windows? Don't use Putty or CygWin...use Git by Matt Wrock

SSH in a native powershell window. I prefer to use console2 and enjoy judging others who don't (conemu is good too).

SSH in a native powershell window. I prefer to use console2 and enjoy judging others who don't (conemu is good too).

Every once in a while I hear of windows users trying to find a good SSH client for Windows to connect to their Linux boxes. For the longest time, a couple of the more popular choices have been Cygwin and Putty.

These still work today but I personally find the experience of both to be sub-optimal. There are lots of annoyances I find in each but the main thing they both lack is an integrated SSH experience in the shell console I use for everything else (mainly powershell) day in/day out. Cygwin and Putty run in separate console experiences. I just want to type 'ssh mwrock@blahblah' in my console of choice and have it work.

You already have the software

Ok, maybe not...but its very likely that if you are reading this and find yourself needing to SSH here and there, you also use GIT. Well many are unaware that git for windows bundles several Linux familiar tools. Many might use these in the git bash shell.

Friends don't let friends use the git bash shell on windows

Don't get me wrong here - I'm not anti bash when I am on Linux. Its great. But I find tools like bash and cygwin offer a "worst of both worlds" experience on Windows. You don't need to run in the bash window to access SSH. You just need to make a small modification to your path. Assuming git was installed to C:/Program Files/Git (the default location), just add C:/Program Files/Git/usr/bin to your path:

$new_path = "$env:PATH;C:/Program Files/Git/usr/bin"
$env:PATH=$new_path
[Environment]::SetEnvironmentVariable("path", $new_path, "Machine")

Bam! Now type 'ssh':

C:\dev\WinRM [psrp +1 ~1 -0 !]> ssh
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
       [-D [bind_address:]port] [-E log_file] [-e escape_char]
       [-F configfile] [-I pkcs11] [-i identity_file]
       [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec]
       [-O ctl_cmd] [-o option] [-p port]
       [-Q cipher | cipher-auth | mac | kex | key]
       [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
       [-w local_tun[:remote_tun]] [user@]hostname [command]
C:\dev\WinRM [psrp +1 ~1 -0 !]>

Don't have git?

Well that's a problem easily solved. Just grab chocolatey if you don't have it already and install git:

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
choco install git -params "/GitAndUnixToolsOnPath"

Note that the "GitAndUnixToolsOnPath " param sets the environment variable for you. You will need to open a new shell for git and ssh to be available in your console and then you are ready to SSH to your heart's content!

Sane authentication/encryption arrives to ruby based cross platform WinRM remote execution by Matt Wrock

Dan Wanek's notes when first developing the ruby implementation of NTLM.

Dan Wanek's notes when first developing the ruby implementation of NTLM.

This week the WinRM ruby gem version 1.6.0 was released and there is really just one new feature it delivers but it is significant: NTLM/Negotiate authentication. Simply stated this provides a safe, low friction authentication and encryption mechanism long available in native Windows remote management tooling but absent in cross platform tools that implement the WinRM protocol.

In this post I'll highlight why I think this is a big deal, share a little history behind the feature release and discuss why authentication/encryption over WinRM has historically been a problem in the cross platform ecosystem.

Tell me again? Why is this awesome?

This is awesome because it means that assuming you are connecting to a Windows Server OS of Windows 2012 R2 or later, you can now securely connect from either a Windows or Linux application leveraging the ruby WinRM gem without any preconfiguration of the target machine. Client OS SKUs (like Windows 7, 8 and 10) and server versions 2008 R2 and prior still need to have WinRM explicitly enabled.

No SSL setup and no WinRM configuration that compromises its security. Just to be clear SSL is still a good idea and encouraged for production nodes, but if you are just trying to get your feet wet with Windows remote execution using tools like Chef or Vagrant, it is now much easier to accomplish and keep the security bar at a sane level.

How did this come to be?

I am a developer at Chef and we use the WinRM gem inside of Knife-Windows in order to execute remote commands on a Windows node from either Windows or Linux. We use a monkey patch gem called winrm-s to provide Negotiate authentication and encryption BUT it only works from windows workstations because it leverages the native Win32 APIs available only on Windows. Also its just monkey patches on top of the winrm and httpclient gems and therefore quite fragile.

My initial objective was to simply port the patches in winrm-s to winnrm and possibly the httpclient gem so that we could provide the same functionality but the implementation would live downstream where it belongs.

Well I remembered I had seen a PR in the WinRM repo that mentioned providing Negotiate auth implementation. It was submited by Dan Wanek, the original author of the WinRM gem. The PR was from late 2014 and I never really looked at it but filed it away in my mind as something worth looking at when the time came to drop winrm-s. So the time came and I quickly noticed that it did not seem to use any native Win32 APIs but leveraged another gem, rubyntlm (also authored by Dan Wanek), which appeared to be a pure ruby implementation of NTLM/Negotiate. That means it would work on Linux in addition to Windows which would be really really great.

It was pretty straight forward to get working and needed just a small amount of tweaking to be production ready - impressive since it had been dormant for over a year. Once it was up and running I verified that it indeed worked both from Windows and Linux. Nice work Dan!

The cross platform Windows remote execution landscape

I've written several posts covering different aspects of WinRM. Like this one and this one and also this one. Its a protocol that many native windows users likely take for granted and perhaps even forget they are using it when they are using its more full featured cousin: Powershell Remoting. However those who dont have access to direct Powershell because they either run on other operating systems that need to talk to Windows machines or are on Windows but use tools that are portable to non windows platforms are likely using a library that implements the WinRM protocol.

There are many such libraries:

This is just a list of the most popular libraries but there are many more. These are used by well known "DevOps" automation tools such as Chef, Ansible, Packer, Vagrant and others.

WinRM is a simple SOAP based client/server protocol. So the above libraries merely implement a web service client that issues requests to a WinRM Windows service and interprets the responses. Basically these exchanges result in:

  • Creating a Shell
  • Creating a Command
  • Requesting Command Output and Exit Code

The output can include both Standard Output and Standard Error streams.

Kind of like SSH but not.

What about Powershell Remoting

I'll save the details for another post but be aware there is a more modern protocol called the Powershell Remoting Protocol (PSRP). There is no cross platform implementation that I am aware of. It uses the same SOAP based wire protocol - MS-WSMV (Web Services Management Protocol Extensions for Windows Vista). I just love the "shout out" to Vista here.

PSRP is more feature rich but more difficult to implement. I've been playing with a ruby based partial implementation and will blog more details soon. You can run powershell commands with WinRM but you are shelling out to Powershell.exe from the traditional "fisher-price" Windows command shell.

Securing the transport

If you are using native WinRM on Windows (likely via Powershell Remoting) the most popular methods of authentication and encryption are (but are not limited to):

Negotiate

To quote the Windows Remote Management Glossary, Negotiate Authentication is defined:

A negotiated, single sign on type of authentication that is the Windows implementation of Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO). SPNEGO negotiation determines whether authentication is handled by Kerberos or NTLM. Kerberos is the preferred mechanism. Negotiate authentication on Windows-based systems is also called Windows Integrated Authentication.

Mmmmmmm. Lets all take a minute or two and think about just what this means and how it might guide our relationships and shape our perspectives on global social injustice.

Yeah its complex right? Anyways its also secure. Its much better than emailing your password to the contacts in your address book.

Kerberos

You know this Windows Remote Management Glossary is pretty darn smart. Lets just see what it has to say about Kerberos:

A method of mutual authentication between the client and server that uses encrypted keys. For computers running on a Windows-based operating system, the client account must be a domain account in the same domain as the server. When a client uses default credentials, Kerberos is the authentication method if the connection string is not one of the following: localhost, 127.0.0.1, or [::1].

Takeaways here are its "mutual", "encrypted" and importantly "for computers." Again, better than emailing your password to the contacts in your address book.

Basic Authentication

Secrets are hard. Not only technically but emotionally as well. There is no small effort involved hiding the truth. Well thank goodness Basic Authentication makes it easy to share our secrets easily. Ah sweet freedom.  Here credentials are transmitted in plain text. I like to think of it as a "giving" protocol. Sure we all think our secrets are worth being kept but are they? Really? Just use Basic Authentication and you might find out.

SSL

This really isn't so much an authentication mechanism but it is a familiar means of securely transporting data from one point to another where the contents are only to be accessible to the sender and receiver. WinRM communication can use either HTTP or HTTPS (SSL). HTTP is the default but HTTPS provides an added layer of encryption.

One of the critical keys to securely using SSL is having a valid certificate issued by a reputable certification authority that serves to ensure that those on either side of the communication are who they say they are. Without this, for example using a non validated self signed certificate, you run the risk of a "Man in the Middle Attack". Not a nice man who offers to fix your tire, provide financial assistance or offer grievance counseling after the loss of a loved family member or pet. Rather a mean man who wants to take what you have and not give it back. He can do that because the authenticity of your certificate cannot be validated and therefore he can stand in the middle between you and the remote windows machine pretending to be that machine.

The dilemma of sane encryption using cross platform libraries

Some of the dilemmas I'll mention are shared in both native windows and cross platform libraries. For example, the friction of getting SSL up and running is pretty much the same on both sides. The key difference is that cross platform libraries may not (usually don't in fact) have access to all the authentication mechanisms listed above or getting them installed and configured is far less than clear.

Good news: SSL is Everywhere, Bad News: SSL is a pain to setup everywhere

Of course this statement is somewhat relative. There are those that are familiar with the basic rules of computer security and work with the knobs and levers of these mechanisms frequently enough that it is straight forward for them to setup. Even among the technically savvy, this is NOT the majority and its especially true in the world of WinRM vs. SSH. Not because SSH users are smarter, but because in Linux land, SSH is just so ubiquitous and pervasive its hard not to deal with regularly enough for it to be unfamiliar.

Here are some points to highlight the friction and pitfalls of SSL over WinRM:

  • Its not on by default and there are several steps to set it up
  • Without a "valid" certificate its still not secure (but better than Basic Authentication over HTTP) and valid certificates take effort to obtain.
  • Bootstrapping problem: How do I get the valid certificate onto the remote machine before establishing a secure connection? There are several ways to do this depending on your cloud provider or image prep system but that assumes you have a cloud provider or an image prep system.

Kerberos: I cant tell you in a paragraph how to set it up and get it working

First just getting the right library can be the worst part - one that's compatible with your OS, architecture and the language runtime of your WinRM library. Most cross platform libraries support it but its less than trivial to get working.

Negotiate is likely not implemented

Well on ruby it is now, however, its the only non native Windows implementation I am aware of (which does not mean that there are not others). If it is implemented, and again - it is on Ruby, its secure and it "Just Works."

Basic auth is available everywhere, horrible everywhere and slightly painful to setup but easier than SSL

If you dig into almost all of the Readme files of the cross platform WinRM libraries, they will all tell you how to run horrible commands on your computer that put out the Welcome Mat for the bad guys. I've listed these in a few posts and for once I will not do so here.

Not only are running these commands a bad idea in general (certainly on production nodes) but again they represent a bootstrapping problem. Before you can successfully talk to the machine they have to be run. There are ways to accomplish this but again rely on cloud APIs or pre-baking images.

Next steps, caveats and how can we make this even better

Its totally awesome that NTLM/Negotiate authentication is now available as a cross platform option but it only lays down the foundation.

Its not the default and consuming applications need to "turn it on"

When using the WinRM gem, consumers must specify which authentication transport they want to use. There is no default. So today applications that specify ":plaintext, basic_auth: true" will continue to use basic authentication.

Chef and Test-Kitchen support coming very soon, and Vagrant support on the way

I am a developer at Chef and my PR for porting this into Knife-Windows is imminent. I will also be porting to Test-Kitchen's winrm transport and I plan to submit a PR to do the same for Vagrant.

Update: PRs to knife-windows, test-kitchen and vagrant are all submitted.

Still no support outside of Ruby

For instance Ansible (Python) and Packer (GO) - both very popular tools that help manage a data center have yet to have working Negotiate Auth implementations available.

The fact is its hard and tedious to implement something like an encryption specification solely from following a specification PDF. Further, there are other ways to get secure so its not a "show stopper." However as more become aware of the rubyntlm library, referencing such a library makes it much easier to implement.