Windowless VirtualBox VMs (Windows Host)

Since I haven’t gotten around to playing with VMware or xen yet – VirtualBox is what I am using for virtualization in my test environment.

Under Windows I like to use PuTTy to connect to my VMs – even those running on the same machine. Other than providing a consistent interface, it has a few features that make life easier (cut & paste, scrollback, and my desired colour scheme being the notable ones). Given this, I really don’t need the VirtualBox GUI to launch and sit around doing nothing – I have enough necessary windows open at any time to not want an extra unnecessary one.

So, the starting point, of course, is to get a list of your VMs:

c:\Program Files\Oracle\VirtualBox> vboxmanage list vms

"CentOS" {1aa2fa18-fc25-4610-91ca-6e984b33edd2}

I’ve shown one VM above as an example – the output includes both a name and a UUID. If you look at the other programs in that directory, you will find ‘VBoxHeadless‘ – as the name suggests, it will run a VM without a GUI – sounds like what we want. We can pass the VM to it using the name, for example:

c:\Program Files\Oracle\VirtualBox\VBoxHeadless -s CentOS

Or using the UUID, for example:

c:\Program Files\Oracle\VirtualBox\VBoxHeadless -s 1aa2fa18-fc25-4610-91ca-6e984b33edd2

The ‘-s‘ parameter is short for ‘--start-vms‘. Giving it a try, you will find that it starts the VM and doesn’t show the GUI – however, the command prompt sticks around, which rather defeats the point of trying to do away with that extra window lying around (and closing out that command prompt aborts the VM).

A bit of research shows some interesting options, which I will briefly outline below – I’ll leave it up to you to decide which one is the best.

Method 1 – VBScript

Write a VBScript to launch VBoxHeadless so that you can alter the window settings – it works, but seems overly complex for the task, without being particularly interesting. Easy enough to come across elsewhere, so I won’t make any further mention of it here.

Method 2 – Modify the VBoxHeadless binary

This involves a hex editor and actually changing the bytes of a program – definitely more ‘hackish’ than using VBScript – but also more interesting.

Most Windows applications usually retain the (DOS) MZ header – literally, they start with MZ (ASCII) (4D 5A in hex) – so they can display the “This program cannot be run in DOS mode” error. Such programs define the offset to the PE header at byte 60 (in decimal). Looking at position 60 of VBoxHeadless we get the hex value F8 (which is 248 in decimal). As you might guess the PE header starts with… ‘PE‘ (in ASCII) – which is 50 54 in hex. Sure enough, going to position 248 of our binary shows us the expected start of the PE header.

The structure of the PE header is described in this MSDN article (it does make for an interesting read if you are into that sort of thing). The particular header field we are interested in – ‘Subsystem’ – begins on the 93rd byte of the header (add up all the BYTE, WORD (2 bytes each), and DWORD (4 bytes each) values before it to get 88, plus 4 bytes for the PE header, giving a total of 92). Starting from position 248, this takes us to 340. This field is 2 bytes in length (WORD), and is set to 03 00 (0x0003) in the original VBoxHeadless. It defines the subsystem that the application will run on – changing this from ‘03 00‘ (character subsystem) to ‘02 00‘ (GUI subsystem), allows the program to run without the console window. (So, just to restate that – change byte 340 of VBoxHeadless from ’03’ to ’02’). (I am using VBoxHeadless v4.1.8r75467 at the time of writing)

This method does work quite well – (obviously, you should backup your original executable before trying it) – and while easy enough, seems both overly complex for the purpose and will probably break with each upgrade.

Method 3 – The start command

The ‘start‘ command will launch a program, and it offers the /B parameter to do so without creating a new window. This one is rather interesting, in that by itself, the start command seems to still leave a command prompt window lying around. At first glance, it seemed pretty much identical to using VBoxHeadless directly – however, I was able to exit the running program (using Ctrl+C) without it terminating (on Windows 7 SP1). On the other hand, I have read quite a few comments suggesting that trying to exit it will terminate the program – so your mileage may vary. Obviously, if the program terminates it defeats the point – and at the same time, keeping a useless command prompt around isn’t ideal. Redirecting the output of start to nul got me back to the command prompt (trying to redirect the output of VBoxHeadless directly to nul, on the other hand, didn’t return me to the command prompt – just left me with a blinking cursor).

start /D "C:\program files\oracle\virtualbox" /B VBoxHeadless -s UUID > nul 2&>1

For example:

start /D "C:\program files\oracle\virtualbox" /B VBoxHeadless -s 1aa2fa18-fc25-4610-91ca-6e984b33edd2 > nul 2>&1

Sadly though, while you are returned to the command prompt, and everything seems great – exiting the command prompt aborts the VM – so this one doesn’t actually work.

Method 4 – PowerShell’s start-process

The final approach which I rather liked was using PowerShell. Now, I am not someone who has any interest in Windows sysadmin stuff – so using PowerShell isn’t exactly part of my skill set, but if something performs the task I need in a reasonble manner, I am open to using it. Powershell has a ‘start-process‘ command to which you can directly pass a WindowStyle parameter. In many ways this is very similar to what is being done in most VBScript approaches to this problem, I just find this to be a more elegant solution. Running it from the command prompt, you get:

powershell start-process 'C:\program files\oracle\virtualbox\vboxheadless' '-s 1aa2fa18-fc25-4610-91ca-6e984b33edd2' -WindowStyle Hidden

Of particular note here, is that:

1. You must provide the parameters to VBoxHeadless as a separate parameter to start-process and
2. Using double quotes around the parameters passed to start-process does not work (when done in this manner – it works just fine if you first launch PowerShell – but that is an extra step).

With that annoyance out of the way, perhaps I can now actually use my VM for what it was intended…