xhyve – Lightweight Virtualization on OS X Based on bhyve

The Hypervisor.framework user mode virtualization API introduced in Mac OS X 10.10 (Yosemite) cannot only be used for toy projects like the hvdos DOS Emulator, but is full-featured enough to support a full virtualization solution that can for example run Linux.

xhyve is a lightweight virtualization solution for OS X that is capable of running Linux. It is a port of FreeBSD’s bhyve, a KVM+QEMU alternative written by Peter Grehan and Neel Natu.

  • super lightweight, only 230 KB in size
  • completely standalone, no dependencies
  • the only BSD-licensed virtualizer on OS X
  • does not require a kernel extension (bhyve’s kernel code was ported to user mode code calling into Hypervisor.framework)
  • multi-CPU support
  • networking support
  • can run off-the-shelf Linux distributions (and could be extended to run other operating systems)

xhyve may make a good solution for running Docker on your Mac, for instance.

Running Tiny Core Linux on xhyve

The xhyve repository already contains a small Linux system for testing, so you can try out xhyve by just typing these few lines:

$ git clone https://github.com/mist64/xhyve
$ cd xhyve
$ make
$ ./xhyverun.sh

And you will see Tiny Core Linux booting in your terminal window:

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 3.16.6-tinycore64 (tc@box) (gcc version 4.9.1 (GCC) ) #777 SMP Thu Oct 16 10:21:00 UTC 2014
Command line: earlyprintk=serial console=ttyS0 acpi=off
e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.6 present.
AGP: No AGP bridge found
e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
CPU MTRRs all blank - virtualized system.

[...]

 (?-
 //\   Core is distributed with ABSOLUTELY NO WARRANTY.
 v_/_           www.tinycorelinux.com

tc@box:~$

To shut down the VM and exit to the Mac’s command line, enter:

$ sudo halt

Running Ubuntu on xhyve

You can also install a more complete Linux distribution on xhyve. The tricky bit is that xhyve doesn’t come with a BIOS or EFI booter, so it is necessary to extract the kernel and initrd from the Linux image and pass them to xhyve manually.

First download Ubuntu Server (the desktop version doesn’t support the text mode installer) into the directory “ubuntu” inside the “xhyve” directory:

$ ls -l
total 1218560
-rw-r--r--@ 1 mist  staff  623902720  6 Jun 22:14 ubuntu-14.04.2-server-amd64.iso

We need to extract the kernel and initrd, which is a little tricky, because OS X doesn’t recognize the hybrid file system on the image without a little hack:

$ dd if=/dev/zero bs=2k count=1 of=/tmp/tmp.iso
$ dd if=ubuntu-14.04.2-server-amd64.iso bs=2k skip=1 >> /tmp/tmp.iso
$ hdiutil attach /tmp/tmp.iso
$ cp /Volumes/Ubuntu-Server\ 14/install/vmlinuz .
$ cp /Volumes/Ubuntu-Server\ 14/install/initrd.gz .

Create a virtual hard disk image (8 GB in the example):

$ dd if=/dev/zero of=hdd.img bs=1g count=8

Then create a script to run xhyve with the correct arguments for the installer:

#!/bin/sh

KERNEL="ubuntu/vmlinuz"
INITRD="ubuntu/initrd.gz"
CMDLINE="earlyprintk=serial console=ttyS0 acpi=off"

MEM="-m 1G"
#SMP="-c 2"
NET="-s 2:0,virtio-net"
IMG_CD="-s 3,ahci-cd,ubuntu/ubuntu-14.04.2-server-amd64.iso"
IMG_HDD="-s 4,virtio-blk,ubuntu/hdd.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"

build/xhyve $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD -f kexec,$KERNEL,$INITRD,"$CMDLINE"

You will want networking enabled, so it’s easiest to run the script as root (this requirement is lifted if you codesign the binary):

$ sudo ./xhyverun_ubuntu_install.sh

You will see the Ubuntu text mode installer:

  ┌───────────────────────┤ [!!] Select a language ├────────────────────────┐
  │                                                                         │
  │ Choose the language to be used for the installation process. The        │
  │ selected language will also be the default language for the installed   │
  │ system.                                                                 │
  │                                                                         │
  │ Language:                                                               │
  │                                                                         │
  │                               C                                         │
  │                               English                                   │
  │                                                                         │
  │     <Go Back>                                                           │
  │                                                                         │
  └─────────────────────────────────────────────────────────────────────────┘

<Tab> moves; <Space> selects; <Enter> activates buttons

All answers should be straightforward, and the defaults are usually fine. Make sure to select “Yes” when asked “Install the GRUB boot loader to the master boot record”.

At the very end, on the “Installation complete” screen, select “Go back” and “Execute a shell”, so you can copy the installed kernel and initrd to the Mac side. In the VM, type this:

# cd /target
# sbin/ifconfig
# tar c boot | nc -l -p 1234

On the Mac, type this, replacing the IP with the output from ifconfig before:

$ cd ubuntu
$ nc 192.168.64.7 1234 | tar x

In the VM, exit the shell:

# exit

Then select “Finish the installation”.

To run the Ubuntu installation from the virtual hard disk, create the following script, fixing up the kernel and initrd version numbers:

#!/bin/sh

KERNEL="ubuntu/boot/vmlinuz-3.16.0-30-generic"
INITRD="ubuntu/boot/initrd.img-3.16.0-30-generic"
CMDLINE="earlyprintk=serial console=ttyS0 acpi=off root=/dev/vda1 ro"

MEM="-m 1G"
#SMP="-c 2"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4,virtio-blk,ubuntu/hdd.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"

build/xhyve $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD -f kexec,$KERNEL,$INITRD,"$CMDLINE"

Then run the script:

$ sudo ./xhyverun_ubuntu.sh

To make your Linux installation useful, you may want to install an SSH server:

$ sudo apt-get install openssh-server

Or install a full UI that you can access using VNC:

$ sudo apt-get install xubuntu-desktop vnc4server

Then run the VNC server:

$ vnc4server :0 -geometry 1024x768

And conntect to it by pasting this into Finder’s Cmd+K “Connect to Server” dialog:

vnc://ubuntu.local

If you also follow skerit’s Ubuntu VNC tutorial, you’ll get to a desktop like this.

Next Steps

xhyve is very basic and lightweight, but it has a lot of potential. If you are a developer, you are welcome to contribute to it. A list of current TODOs and ideas is part of the README file in the repository.

33 thoughts on “xhyve – Lightweight Virtualization on OS X Based on bhyve”

  1. This is fantastic!

    Any idea if it is possible to run FreeBSD in xhyve?

    Or when it may be possible to do so?

    Reply
    • Re FreeBSD guest: All that is needed is a small FreeBSD bootloader integrated into xhyve, like the Linux bootloader that is integrated today.

      Reply
  2. Well, running ” ./xhyverun.sh” crashed my Macbook (OS X 10.10.3)
    I will create an issue on Github to post more details.

    Reply
    • Had the same thing happen on a seed of 10.10.4, but after the (forced) restart it worked fine. Could be because I was also running a VirtualBox instance via Vagrant at the same time.

      Reply
    • Running OS X 10.10.3 with a couple of vagrant machines running. First time I ran ./xhyverun.sh crashed the mac and rebooted, just like the guys are saying above.

      Reply
  3. I’d add a comment to your “sudo halt” section saying “make sure you type this on your VM vs your Mac”. 🙂

    Reply
  4. Great work, thx.

    How can i reach the internet from the guest? Can i add a new network card, to tunneling the host machine internet?

    Reply
    • For what it’s worth: I was able to sign the binary, but when attempting to run said signed binary (with the entitlements), I get an error:

      ./xhyverun.sh: line 15: 7688 Illegal instruction: 4 build/xhyve $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD -f kexec,$KERNEL,$INITRD,”$CMDLINE”

      This is with networking enabled, and the number before Illegal instruction varies.

      Reply
  5. Hi, thx for your reply. I alraedy uncommented this section (i followed the ubuntu example)

    When the wm boot, i have eth0, and is up, i can reach from the mac (ping is ok from mac to vm), but in the virtual machine i have no internet. Maybe i have to add a new network card, and use it as a bridge?

    Reply
  6. Great Job, I successully installed debian 7.8, but not 8.1(kernel oops)

    is bridged networking supported with virtio-net?

    Reply
  7. Excellent work! Just tested this on our Flockbox boot2lxc vm based on Alpine Linux and it booted perfectly. Networking works, LXC works. With the -A option and removing acpi=off, xhyve cpu usage hovers around 0.2-0.5% at idle and around 58MB RAM, which is great.

    This makes it a more than decent option to try the the Flockport App store on OSX, apart from VirtualBox. can’t wait to see this improve and become an efficient high performance alternative.

    For Alpine Linux users, only had to make a change to enable ttys0 on /etc/inittab in Alpine so you can have login. Is there any plan to support sparse images? Also Is there any way to control the networking in terms of IP addresses across boots?

    Reply
  8. I have created a kernel extension to backport virtualization support from newer xnu to <= 10.9.5 kernels. Currently, it lets AppleHV.kext successfully start, but there are still some issues left. I would not like to make source code public (yet). If you are interested in helping, please contact me at dozorniyrezerv at (DOMAIN FOR GOOGLE MAIL).

    Reply
  9. Hi,

    I have a question. Could you elaborate a little bit more on how “things” are displayed from the VM to OS X? I mean, does Xhyve use X11 or perhaps more high level mechanisms such as printing directly to the OS X terminal?

    Reply
    • I can’t find an answer to this anywhere. I just need to access the Mac OS X11 server (Xquartz) from a container. I should be able to just set DISPLAY=:0

      That doesn’t just work, so there might be some system settings, etc., but this should be simpler than with VirtualBox. I don’t see why this would need that socat command to bridge between Mac OS and xhyve, but who knows, there is no information out there about this.

      Reply
  10. Impressive, most impressive. I’ve been doing arcane things with this for years and just wanted to say thank-you.

    Reply

Leave a Comment