Windows Updates and Hypervisor Management with Ansible

06:41 reading time


After some failed attempts at propping up a satisfactory WSUS server to manage patching our Windows hosts, we finally achieved a non-right-clicky Windows Update solution using Ansible and Powershell. Like any marriage of Linux and Windows, it wasn’t without its frustrations. Thankfully we continue to make use of the painfully achieved scaffolding by ansible-izing tasks such as SQL Server ChatOps and Hyper-V VM pause and resume.

Ansible + Windows: much pain, great reward

A while back Ansible announced support for Windows and provided some example scripts to do things like install software and run Windows Updates. We were already using Ansible to script maintenance operations on our Linux servers so we were excited to use the same tool in our Windows environment. Unfortunately these example scripts did not fully work as advertised. Despite great documentation from Ansible and plenty of blogposts about Powershell and Windows Updates, we still spent a fair amount of time gluing all the bits together. Perhaps our pain will be your gain?

Much Pain

Microsoft doesn’t want to make it easy to run OS updates remotely (I think for ‘security’ reasons, naturally). This made it hard to use any tool to remote in and run Windows updates, even with AD credentials that can elevate to Administrator. A sysinternals tool PSExec.exe can be called remotely, and makes Windows think the command is running locally, so that got us around the first issue.

The next problem was that different versions of Windows (2008R2) were stubborn about elevating the shell of PSExec (even with the PSExec -h switch intended for this very purpose) so we had to fall back to using the Windows Local Administrator account. Ansible wants to use its fancy kerberos integration to log you in with your locally cached credentials, so you will need to update the inventory file with the local administrator password as well as supply psexec with the administrator password (you will be prompted for this one when you run the script). Ansible does not allow sharing variables between the playbook and the inventory so adding the password to the inventory and being prompted additionally was required.

The final authentication issue was that updating domain controllers required logging in as the domain administrator account, so we had to update the Ansible playbook to check for our “-dc” host naming convention and switch the login to DOMAIN\Administrator for those servers. This naming scheme adjustment likely won’t apply to your environment but doesn’t hurt and may provide an example for handling ansible playbook variables based on hostname.

After these issues were resolved, the remaining challenges were tweaking the PSWindowsUpdate cmdlet to download and install all updates without halting when it found updates that required reboots, and reporting the status to Slack.

A turducken wrapped in an enigma

Our solution is not the most elegant, but it gets the job done. In a nutshell: Ansible uses pywinrm to log in to a Windows host as the local Administrator then run a 'raw’ command:

'"C:\Program Files (x86)\Ansible\PsExec.exe" -u {{ administrator_login }} -p {{ administrator_password }} -accepteula powershell get-wuinstallslack -WindowsUpdate -DownloadOnly -IgnoreReboot -AcceptAll -slackToken ***SLACK_TOKEN_HERE*** -slackUsername ***SLACK_USERNAME_HERE*** -slackChannel ***SLACK_CHANNEL_HERE***'

This raw command might benefit from some parsing. Psexec.exe is taking the first -u and -p switches to log in as Administrator. The -accepteula is needed so to avoid hanging at the eula prompt. Next, psexec is using its Admin-elevated authentication to run the get-wuinstall Powershell cmdlet. The options for get-wuinstall are documented here. The Slack parameters are required for the Slack API to post output to a specific channel. Sound fun? Here’s how you can do it too:

Requirements:
Control Machine (Linux / Mac)
☑ Ansible 1.9.1 (we had difficulties with later versions so use this release for this solution)
☑ pywinrm
☑ Windows Local Administrator Password
Playbooks/Inventory

Remote Machine (Windows)
☑ Powershell 3.0 and depending on Windows version, a hotfix to prevent out of memory errors
☑ Within Powershell 3.0 as administrator run: set-executionpolicy remotesigned and the Ansible provided ConfigureRemotingForAnsible.ps1
PSExec.exe placed in C:\Program Files (x86)\Ansible (our script calls out this path; feel free to change it)
☑ Slack customized PSWindowsUpdate cmdlet get-wuinstallslack copied to C:\Windows\System32\WindowsPowershell\v1.0\Modules. Here is the original PSWindowsUpdates cmdlet. Again, feel free to change the playbook and cmdlet if you don’t need the slack integrations or have a different set of customizations to implement
☑ For Hyper-V host management you will require the HyperV and WVMHost cmdlets. The HyperV cmdlet was slightly modified from this codeplex entry, and is only required if you are running a lower version than Server 2012.

Example inventory file with required password bits:

[windows:vars]
ansible_ssh_user=Administrator  
ansible_ssh_pass=****YOUR_ADMINISTRATOR_PASSWORD_HERE****
ansible_ssh_port=5986
ansible_connection=winrm

[windows:children]
bi-dev-servers
domain-controllers

[domain-controllers]
domaincontroller-01
domaincontroller-02

[bi-dev]
bi-dev-database
bi-dev-etl
bi-dev-olap

You can see the potential convenience of grouping hosts and limiting runs to groups or individual machines to handle dependencies and specific maintenance windows.

Example playbook:

- hosts: windows
  vars_prompt:
    - name: "administrator_password"
      prompt: "Enter Local Windows Administrator Password"
      private: yes
    - name: "windows_restart"
      prompt: "Do you want Windows to restart automatically after installing updates? y/n"
      default: "y"
      private: no
  vars:
    Reboot: ' '
    administrator_login: 'Administrator'
  tasks:
    - set_fact:
        Reboot: '-IgnoreReboot'
      when: windows_restart == 'n'
    - set_fact:
        Reboot: '-AutoReboot'
      when: windows_restart == 'y'
    - set_fact:
        administrator_login: 'LEAPFROGONLINE\Administrator'
      when: "'-dc' in '{{ inventory_hostname }}'"
    - debug: msg="Reboot Behavior set to {{ Reboot }} "
    - raw: '"C:\Program Files (x86)\Ansible\PsExec.exe" -u {{ administrator_login }} -p {{ administrator_password }} -accepteula C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe get-wuinstallslack -WindowsUpdate -AcceptAll {{ Reboot }} -slackToken ***SLACK_TOKEN_HERE*** -slackUsername ***SLACK_USERNAME_HERE*** -slackChannel ***SLACK_CHANNEL_HERE***'

You can see the other playbooks here.

Great Reward

Now, one person can issue:

ansible-playbook -i inventory/ops/windows playbooks/windows_updates_install.yml

…enter the administrator password when prompted, and monitor slack for updates status!

And now that all the basic authentication and administrative elevation is sorted, we can ansible-ize other tasks that used to require logging in and right-clicking, such as pausing and resuming Hyper-V VMs during actions like SAN maintenance.

I have included modified HyperV cmdlet (required for pre Windows Server 2008R2) and the Windows VM Host cmdlet we are using to pause and resume Hyper-V VMs here and the playbooks here.

Of course, every environment is a snowflake and our solution will need some massaging to work elsewhere. At the very least you should have a better starting point than we did. If you discover more elegant solutions to these problems please let us know!


5ab62922837c6a0926d5c9a53f4bfd25

Mike Broers
Sr. Manager Database Administrator