Chapter 24. Virtualization

The first step in installing FreeBSD on Parallels is to create a new virtual machine for installing FreeBSD.

Choose Install Windows or another OS from a DVD or image file and proceed.

Select the FreeBSD image file.

Choose Other as operating system.

Name the virtual machine and check Customize settings before installation

When the configuration window pops up, go to Hardware tab, choose Boot order, and click Advanced. Then, choose EFI 64-bit as BIOS.

Click OK, close the configuration window, and click Continue.

The virtual machine will automatically boot. Install FreeBSD following the general steps.

After FreeBSD has been successfully installed on macOS® X with Parallels, there are a number of configuration steps that can be taken to optimize the system for virtualized operation.

The first step is to start VMware Fusion which will load the Virtual Machine Library. Click +→New to create the virtual machine:

This will load the New Virtual Machine Assistant. Choose Create a custom virtual machine and click Continue to proceed:

Select Other as the Operating System and either FreeBSD X or FreeBSD X 64-bit, as the Version when prompted:

Choose the firmware(UEFI is recommended):

Choose Create a new virtual disk and click Continue:

Check the configuration and click Finish:

Choose the name of the virtual machine and the directory where it should be saved:

Press command+E to open virtual machine settings and click CD/DVD:

Choose FreeBSD ISO image or from a CD/DVD:

Start the virtual machine:

Install FreeBSD as usual:

Once the install is complete, the settings of the virtual machine can be modified, such as memory usage and the number of CPUs the virtual machine will have access to:

The status of the CD-ROM device. Normally the CD/DVD/ISO is disconnected from the virtual machine when it is no longer needed.

The last thing to change is how the virtual machine will connect to the network. To allow connections to the virtual machine from other machines besides the host, choose Connect directly to the physical network (Bridged). Otherwise, Share the host’s internet connection (NAT) is preferred so that the virtual machine can have access to the Internet, but the network cannot access the virtual machine.

After modifying the settings, boot the newly installed FreeBSD virtual machine.

After FreeBSD has been successfully installed on macOS® X with VMware Fusion, there are a number of configuration steps that can be taken to optimize the system for virtualized operation.

VirtualBox™ is available as a FreeBSD package or port in emulators/virtualbox-ose. The port can be installed using these commands:

One useful option in the port’s configuration menu is the GuestAdditions suite of programs. These provide a number of useful features in guest operating systems, like mouse pointer integration (allowing the mouse to be shared between host and guest without the need to press a special keyboard shortcut to switch) and faster video rendering, especially in Windows® guests. The guest additions are available in the Devices menu, after the installation of the guest is finished.

A few configuration changes are needed before VirtualBox™ is started for the first time. The port installs a kernel module in /boot/modules which must be loaded into the running kernel:

To ensure the module is always loaded after a reboot, add this line to /boot/loader.conf:

To use the kernel modules that allow bridged or host-only networking, add this line to /etc/rc.conf and reboot the computer:

The vboxusers group is created during installation of VirtualBox™. All users that need access to VirtualBox™ will have to be added as members of this group. pw can be used to add new members:

The default permissions for /dev/vboxnetctl are restrictive and need to be changed for bridged networking:

To make this permissions change permanent, add these lines to /etc/devfs.conf:

To launch VirtualBox™, type from an Xorg session:

For more information on configuring and using VirtualBox™, refer to the official website. For FreeBSD-specific information and troubleshooting instructions, refer to the relevant page in the FreeBSD wiki.

VirtualBox™ can be configured to pass USB devices through to the guest operating system. The host controller of the OSE version is limited to emulating USB 1.1 devices until the extension pack supporting USB 2.0 and 3.0 devices becomes available on FreeBSD.

For VirtualBox™ to be aware of USB devices attached to the machine, the user needs to be a member of the operator group.

Then, add the following to /etc/devfs.rules, or create this file if it does not exist yet:

To load these new rules, add the following to /etc/rc.conf:

Then, restart devfs:

Restart the login session and VirtualBox™ for these changes to take effect, and create USB filters as necessary.

Access to the host DVD/CD drives from guests is achieved through the sharing of the physical drives. Within VirtualBox™, this is set up from the Storage window in the Settings of the virtual machine. If needed, create an empty IDECD/DVD device first. Then choose the Host Drive from the popup menu for the virtual CD/DVD drive selection. A checkbox labeled Passthrough will appear. This allows the virtual machine to use the hardware directly. For example, audio CDs or the burner will only function if this option is selected.

In order for users to be able to use VirtualBox™DVD/CD functions, they need access to /dev/xpt0, /dev/cdN, and /dev/passN. This is usually achieved by making the user a member of operator. Permissions to these devices have to be corrected by adding these lines to /etc/devfs.conf:

QEMU is available as a FreeBSD package or as a port in emulators/qemu. The package build includes reasonable options and defaults for most users and is the recommended method of installation.

The package installation includes several dependencies. Once the installation is complete, create a link to the host version of QEMU that will be used most often. If the host is an Intel™ or AMD™ 64 bit system that will be:

Test the installation by running the following command as a non-root user:

This brings up a window with QEMU actively trying to boot from hard disk, floppy disk, DVD/CD, and PXE. Nothing has been set up yet, so the command will produce several errors and end with "No bootable device" as shown in Figure 1. However, it does show that the QEMU software has been installed correctly.

Follow the steps below to create two virtual machines named "left", and "right". Most, but not all commands can be performed without root privileges.

QEMU will start up a virtual machine in a separate window and boot the FreeBSD iso as shown in Figure 2. All command options such as -cpu and -boot are fully described in the QEMU man page qemu(1).

During the installation there are several points to note:

Once the installation completes, the virtual machine reboots into the newly installed FreeBSD image.

Login as root and update the system as follows:

Login as root again and add any packages desired. To utilize the X Window system in the guest, see the section "Using the X Window System" below.

This completes the setup of the "left" VM.

To install the "right" VM, run the following script: This script has the modifications needed for tap1, format=qcow2, the image filename, the MAC address, and the terminal window name. If desired, include the "-runas" parameter as described in the above note.

Once the installation is complete, the "left" and "right" machines can communicate with each other and with the host. If there are strict firewall rules on the host, consider adding or modifying rules to allow the bridge and tap devices to communicate with each other.

To implement a serial console, a guest VM running FreeBSD needs to insert

in /boot/loader.conf to allow the use of the FreeBSD serial console.

The updated configuration below shows how to implement the serial console on the guest VM. Run the script to start the VM.

In Figure 7, the serial port is redirected to a TCP port on the host system at VM startup and the QEMU monitor waits (wait=on) to activate the guest VM until a telnet(1) connection occurs on the indicated localhost port. After receiving a connection from a separate session, the FreeBSD system starts booting and looks for a console directive in /boot/loader.conf. With the directive "console=comconsole", FreeBSD starts up a console session on a serial port. The QEMU monitor detects this and directs the necessary character I/O on that serial port to the telnet session on the host. The system boots and once finished, login prompts are enabled on the serial port (ttyu0) and on the console (ttyv0).

It is important to note that the this serial redirect over TCP takes place outside the virtual machine. There is no interaction with any network on the virtual machine and therefore it is not subject to any firewall rules. Think of it like a dumb terminal sitting on an RS-232 or USB port on a real machine.

QEMU also supports running applications that are precompiled on an architecture different from the host CPU. For example, it is possible to run a Sparc64 architecture operating system on an x86_64 host. This is demonstrated in the next section.

The QEMU monitor controls a running QEMU emulator (guest VM).

Using the monitor, it is possible to:

As well as many other operations.

The most common uses of the monitor are to examine the state of the VM, and to add, delete, or change devices. Some operations such as migrations are only available under hypervisor accelerators such as KVM, Xen, etc. and are not supported on FreeBSD hosts.

When using a graphical desktop environment, the simplest way to use the QEMU monitor is the -monitor stdio option when launching QEMU from a terminal session.

This results in a new prompt (qemu) in the terminal window as shown in Figure 10.

The image also shows the stop command freezing the system during the FreeBSD boot sequence. The system will remain frozen until the cont command is entered in the monitor.

QEMU supports the creation of virtual USB devices that are backed by an image file. These are virtual USB devices that can be partitioned, formatted, mounted, and used just like a real USB device.

This configuration includes a -drive specification with the id=usbstick, raw format, and an image file (must be created by qemu-img(1)). The next line contains the -device usb-ehci specification for a USB EHCI controller, with id=ehci. Finally, a -device usb-storage specification ties the above drive to the EHCI USB bus.

When the system is booted, FreeBSD will recognize a USB hub, add the attached USB device, and assign it to da0 as shown in Figure 15.

The device is ready to be partitioned with gpart(8), and formatted with newfs(8). Because the USB device is backed by a qemu-img(1) created file, data written to the device will persist across reboots.

QEMU USB passthrough support is listed as experimental in version 9.0.1 (August, 2024). However, the following steps show how a USB stick mounted on the host can be used by the guest VM.

For more information and examples, see:

The upper part of Figure 16 shows the QEMU monitor commands:

As before, once device_add completes, the FreeBSD kernel recognizes a new USB device, as shown in the lower half of the Figure.

Using the new device is shown in Figure 17.

If the USB device is formatted as a FAT16 or FAT32 filesystem it can be mounted as an MS-DOS™ filesystem with mount_msdosfs(8) as in the example shown. The /etc/hosts file is copied to the newly mounted drive and checksums are taken to verify the integrity of the file on the USB device. The device is then unmounted with umount(8).

If the USB device is formatted with NTFS it is necessary to install the fusefs-ntfs package and use ntfs-3g(8) to access the device:

Change the above device identifiers to match the installed hardware. Consult ntfs-3g(8) for additional information on working with NTFS filesystems.

As noted above, QEMU works with several different hypervisor accelerators.

The list of Virtualization Accelerators supported by QEMU includes:

All the examples in this section used the Tiny Code Generator (tcg) accelerator as that is the only supported accelerator on FreeBSD at present.

The first step to creating a virtual machine in bhyve is configuring the host system. First, load the bhyve kernel module:

There are several ways to connect a virtual machine guest to a host’s network; one straightforward way to accomplish this is to create a tap interface for the network device in the virtual machine to attach to. For the network device to participate in the network, also create a bridge interface containing the tap interface and the physical interface as members. In this example, the physical interface is igb0:

Create a file to use as the virtual disk for the guest machine. Specify the size and name of the virtual disk:

Download an installation image of FreeBSD to install:

FreeBSD comes with an example script vmrun.sh for running a virtual machine in bhyve. It will start the virtual machine and run it in a loop, so it will automatically restart if it crashes. vmrun.sh takes several options to control the configuration of the machine, including:

The last parameter is the name of the virtual machine and is used to track the running machines. The following command lists all available program argument options:

This example starts the virtual machine in installation mode:

The virtual machine will boot and start the installer. After installing a system in the virtual machine, when the system asks about dropping into a shell at the end of the installation, choose Yes.

Reboot the virtual machine. While rebooting the virtual machine causes bhyve to exit, the vmrun.sh script runs bhyve in a loop and will automatically restart it. When this happens, choose the reboot option from the boot loader menu to escape the loop. Now the guest can be started from the virtual disk:

Linux guests can be booted either like any other regular UEFI-based guest virtual machine, or alternatively, you can make use of the sysutils/grub2-bhyve port.

To do this, first ensure that the port is installed, then create a file to use as the virtual disk for the guest machine:

Starting a Linux virtual machine with grub2-bhyve is a two-step process.

Create a device.map that grub will use to map the virtual devices to the files on the host system:

Use sysutils/grub2-bhyve to load the Linux® kernel from the ISO image:

This will start grub. If the installation CD contains a grub.cfg, a menu will be displayed. If not, the vmlinuz and initrd files must be located and loaded manually:

Now that the Linux® kernel is loaded, the guest can be started:

The system will boot and start the installer. After installing a system in the virtual machine, reboot the virtual machine. This will cause bhyve to exit. The instance of the virtual machine needs to be destroyed before it can be started again:

Now the guest can be started directly from the virtual disk. Load the kernel:

Boot the virtual machine:

Linux® will now boot in the virtual machine and eventually present you with the login prompt. Login and use the virtual machine. When you are finished, reboot the virtual machine to exit bhyve. Destroy the virtual machine instance:

In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI firmware. This option may support guest operating systems that are not supported by the other loaders.

To make use of the UEFI support in bhyve, first obtain the UEFI firmware images. This can be done by installing sysutils/bhyve-firmware port or package.

With the firmware in place, add the flags -l bootrom,/path/to/firmware to your bhyve command line. The actual bhyve command may look like this:

To allow a guest to store UEFI variables, you can use a variables file appended to the -l flag. Note that bhyve will write guest modifications to the given variables file. Therefore, be sure to first create a per-guest-copy of the variables template file:

Then, add that variables file into your bhyve arguments:

To view or modify the variables file contents, use efivar(8) from the host.

sysutils/bhyve-firmware also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:

The UEFI firmware support is particularly useful with predominantly graphical guest operating systems such as Microsoft Windows®.

Support for the UEFI-GOP framebuffer may also be enabled with the -s 29,fbuf,tcp=0.0.0.0:5900 flags. The framebuffer resolution may be configured with w=800 and h=600, and bhyve can be instructed to wait for a VNC connection before booting the guest by adding wait. The framebuffer may be accessed from the host or over the network via the VNC protocol. Additionally, -s 30,xhci,tablet can be added to achieve precise mouse cursor synchronization with the host.

The resulting bhyve command would look like this:

Note, in BIOS emulation mode, the framebuffer will cease receiving updates once control is passed from firmware to guest operating system.

Setting up a guest for Windows versions 10 or earlier can be done directly from the original installation media and is a relatively straightforward process. Aside from minimum resource requirements, running Windows as guest requires

An example for booting a virtual machine guest with a Windows installation ISO:

Only one or two VCPUs should be used during installation but this number can be increased once Windows is installed.

VirtIO drivers must be installed to use the defined virtio-net network interface. An alternative is to switch to E1000 (Intel E82545) emulation by changing virtio-net to e1000 in the above command line. However, performance will be impacted.

If ZFS is available on the host machine, using ZFS volumes instead of disk image files can provide significant performance benefits for the guest VMs. A ZFS volume can be created by:

When starting the VM, specify the ZFS volume as the disk drive:

If you are using ZFS for the host as well as inside a guest, keep in mind the competing memory pressure of both systems caching the virtual machine’s contents. To alleviate this, consider setting the host’s ZFS filesystems to use metadata-only cache. To do this, apply the following settings to ZFS filesystems on the host, replacing <name> with the name of the specific zvol dataset name of the virtual machine.

Modern hypervisors allow their users to create "snapshots" of their state; such a snapshot includes a guest’s disk, CPU, and memory contents. A snapshot can usually be taken independent of whether the guest is running or shut down. One can then reset and return the virtual machine to the precise state when the snapshot was taken.

For improved security and separation of virtual machines from the host operating system, it is possible to run bhyve in a jail. See Jails for an in-depth description of jails and their security benefits.

It is advantageous to wrap the bhyve console in a session management tool such as sysutils/tmux or sysutils/screen in order to detach and reattach to the console. It is also possible to have the console of bhyve be a null modem device that can be accessed with cu. To do this, load the nmdm kernel module and replace -l com1,stdio with -l com1,/dev/nmdm0A. The /dev/nmdm devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable (/dev/nmdm0A and /dev/nmdm0B). See nmdm(4) for more information.

To disconnect from a console, enter a newline (i.e. press RETURN) followed by tilde (~), and finally dot (.). Keep in mind that only the connection is dropped while the login session remains active. Another user connecting to the same console could therefore make use of any active sessions without having to first authenticate. For security reasons, it’s therefore recommended to logout before disconnecting.

The number in the nmdm device path must be unique for each virtual machine and must not be used by any other processes before bhyve starts. The number can be chosen arbitrarily and does not need to be taken from a consecutive sequence of numbers. The device node pair (i.e. /dev/nmdm0a and /dev/nmdm0b) are created dynamically when bhyve connects its console and destroyed when it shuts down. Keep this in mind when creating scripts to start your virtual machines: you need to make sure that all virtual machines are assigned unique nmdm devices.

A device node is created in /dev/vmm for each virtual machine. This allows the administrator to easily see a list of the running virtual machines:

A specified virtual machine can be destroyed using bhyvectl:

Destroying a virtual machine this way means killing it immediately. Any unsaved data will be lost, open files and filesystems may get corrupted. To gracefully shut down a virtual machine, send a TERM signal to its bhyve process instead. This triggers an ACPI shutdown event for the guest:

There are numerous utilities and applications available in ports to help simplify setting up and managing bhyve virtual machines:

In order to configure the system to start bhyve guests at boot time, some configuration file changes are required.

To run the Xen™ hypervisor on a host, certain hardware functionality is required. Hardware virtualized domains require Extended Page Table (EPT) and Input/Output Memory Management Unit (IOMMU) support in the host processor.

Users should install the emulators/xen-kernel and sysutils/xen-tools packages, based on Xen™ 4.18.

Configuration files must be edited to prepare the host for the Dom0 integration after the Xen packages are installed. An entry to /etc/sysctl.conf disables the limit on how many pages of memory are allowed to be wired. Otherwise, DomU VMs with higher memory requirements will not run.

Another memory-related setting involves changing /etc/login.conf, setting the memorylocked option to unlimited. Otherwise, creating DomU domains may fail with Cannot allocate memory errors. After making the change to /etc/login.conf, run cap_mkdb to update the capability database. See Resource Limits for details.

Add an entry for the Xen™ console to /etc/ttys:

Selecting a Xen™ kernel in /boot/loader.conf activates the Dom0. Xen™ also requires resources like CPU and memory from the host machine for itself and other DomU domains. How much CPU and memory depends on the individual requirements and hardware capabilities. In this example, 8 GB of memory and 4 virtual CPUs are made available for the Dom0. The serial console is also activated, and logging options are defined.

The following command is used for Xen 4.7 packages:

For Xen versions 4.11 and higher, the following command should be used instead:

Activate the xencommons service during system startup:

These settings are enough to start a Dom0-enabled system. However, it lacks network functionality for the DomU machines. To fix that, define a bridged interface with the main NIC of the system which the DomU VMs can use to connect to the network. Replace em0 with the host network interface name.

Restart the host to load the Xen™ kernel and start the Dom0.

After successfully booting the Xen™ kernel and logging into the system again, the Xen™ management tool xl is used to show information about the domains.

The output confirms that the Dom0 (called Domain-0) has the ID 0 and is running. It also has the memory and virtual CPUs that were defined in /boot/loader.conf earlier. More information can be found in the Xen™ Documentation. DomU guest VMs can now be created.

Unprivileged domains consist of a configuration file and virtual or physical hard disks. Virtual disk storage for the DomU can be files created by truncate(1) or ZFS volumes as described in “Creating and Destroying Volumes”. In this example, a 20 GB volume is used. A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs. The ISO installation file is retrieved with fetch(1) and saved locally in a file called freebsd.iso.

A ZFS volume of 20 GB called xendisk0 is created to serve as the disk space for the VM.

The new DomU guest VM is defined in a file. Some specific definitions like name, keymap, and VNC connection details are also defined. The following freebsd.cfg contains a minimum DomU configuration for this example:

These lines are explained in more detail:

After the file has been created with all the necessary options, the DomU is created by passing it to xl create as a parameter.

The output of xl list confirms that the DomU has been created.

To begin the installation of the base operating system, start the VNC client, directing it to the main network address of the host or to the IP address defined on the vnclisten line of freebsd.cfg. After the operating system has been installed, shut down the DomU and disconnect the VNC viewer. Edit freebsd.cfg, removing the line with the cdrom definition or commenting it out by inserting a # character at the beginning of the line. To load this new configuration, it is necessary to remove the old DomU with xl destroy, passing either the name or the id as the parameter. Afterwards, recreate it using the modified freebsd.cfg.

The machine can then be accessed again using the VNC viewer. This time, it will boot from the virtual disk where the operating system has been installed and can be used as a virtual machine.

This section contains basic information in order to help troubleshoot issues found when using FreeBSD as a Xen™ host or guest.