Tapping into Qemu

March 25, 2023

Qemu - the virtualization software and system emulator by Fabrice Bellard and the fine folks at www.qemu.org is capable of emulating a wide variety of systems. When you pair it up with FreeBSD, and the if_bridge(4) and tap(4) interfaces you can create a complete virtualized network inside your host machine.

Figure 1: Qemu Setup on FreeBSD Host

Figure 1: Qemu Setup on FreeBSD Host

This techbits article will show you how to set up and run the virtual machines in the above image. They were developed with Qemu version 7.2.0 along with a number of custom scripts that allow easy bridge and tap setup and virtual machine setup.

If you want, you can download the lab layout and scripts here and follow along.

Directory Structure

The host machine file directory layout for the exercises is:

/home/foo
         /LAB
             /SCRIPTS
                _create2bsds.sh  (script to create two Qemu disks images)
                left.sh          (script for running 'left' VM host')
                right.sh         (script for running 'right' VM host')
                mkbr.sh          (script to create bridge and tap devices)
                   /BMP
                      FreeBSD_splash_640x480.bmp  (splash screen)
             /ISO
                fbsd.iso         (latest FreeBSD install iso)
             /VM
                left.img         (Qemu disk image for 'left' hosts)
                right.img        (Qemu disk image for 'right' host)

You will need:

  • A laptop or desktop machine, preferably running a 64 bit AMD or Intel CPU with at least 8 GB of memory. More memory will give you better performance when running the host and four or more QEMU virtual machines simultaneously. This will be the host machine, noted as such throughout this article. A large, fast disk (SSD preferred) with at least 256GB is recommended.
  • A decent sized monitor to be able to view several (possibly up to 8) terminal windows at once. A screen capable of 1920 x 1080 works nicely.
  • A decent Internet connection, preferably wired (gigabit speed recommended)
  • A recently downloaded ISO of FreeBSD to install on the QEMU virtual machines. It is, of course, best if you can use the same version on the host and all virtual machines.
  • The host machine will need a graphical environment such as X Windows and a decent window manager such as XFCE or similar.

Step by step instructions follow.

Host Setup

If you already have FreeBSD installed on your host machine, skip to the next section.

To download the FreeBSD installation .iso visit: https://www.freebsd.org/releases/ Help on the installtion of FreeBSD is provided in the FreeBSD Handbook.

  1. Install FreeBSD on the host machine on your local network, and update it to the latest patch level with freebsd-update(8) fetch install.
    Note that the default user should be a member of the wheel group.

  2. Install a graphical desktop and window manager on the host machine. The following packages should be sufficient. Installation, even over a fast Internet connection will take some time.

    • xorg – Graphical environment, will bring in a lot of dependencies
    • xfce4 (or use a different window manager. Use the latest version.)
    • qemu – software for virtualization. With Qemu, you get very granular control over all aspects of the virtual machine. The KVM Qemu version is not required (or desired).
    • nmap – required for including the ncat(1) program, the version used in this article.
    • sudo – required to run virtual machines as root (configure sudo as needed)
    • shell_of_your_choice – or just use the Bourne shell if you are old-school.

    $ sudo pkg install xorg xfce qemu nmap sudo

Once the host software is installed, you should confgure the graphical environment and window manager to your liking, and double check all network settings.

One note about the xfce4 window manager - its xfce-terminal graphical terminal has features that make using it for these exercises very handy - scriptable tabs and resizing, command execution, and window placement. Some examples are given below.

Installation of the virtual machines and the required packages are described below.

Virtual Machine Install

You’ll need to move the downloaded .iso to the ~/LAB/ISO directory. Also, the scripts below assume the name to be fbsd.iso so either rename the downloaded file, or better, make a link:

  $ ln -s FreeBSD-13.1-RELEASE-amd64-dvd1.iso  fbsd.iso

  $ ls -al
  total 3755526
  drwxr-xr-x  2 jpb  jpb           4 Mar 26 17:20 .
  drwxr-xr-x  5 jpb  jpb           5 Mar 26 17:33 ..
  -rw-r--r--  1 jpb  jpb  4621281280 May 16  2022 FreeBSD-13.1-RELEASE-amd64-dvd1.iso
  lrwxr-xr-x  1 jpb  jpb          35 May 16  2022 fbsd.iso -> FreeBSD-13.1-RELEASE-amd64-dvd1.iso

The next step is to set up the two virtual machines. This involves creating a Qemu file-based disk image for each and installing FreeBSD on the images from the same ISO you used for the host install. Follow these steps for installing the virtual machines:

  1. Change directory to the location for virtual machines ( ~/LAB/VM). Run the qemu-img command to create two images – one for the “left” VM and the other for the “right” VM:
$ qemu-img create -f raw -o size=15G left.img
$ qemu-img create -f raw -o size=15G right.img

Note that Qemu uses a technique called “copy on write” for managing disk space. This technique does not require a complete 15G disk, just a stub version that is managed directly by the VM. The disk grows dynamically as the VM writes to it.

The naming conventions above are used throughout this article, but you can name them whatever you want.

Virtual Machine Network

Before you start the Qemu software, you will need to set up one software if_bridge(4) devices and two tap(4) devices. The tap devices are used by Qemu as part of the network em(4) devices.

A script, mkbr.sh (included in the above zip file), is the easiest way to accomplish correctly setting up the bridge and tap devices. You can specify any number of bridge and tap devices. The only requirement is that the bridge come before the tap devices that are connected to it. Here’s an example:

$ sudo /bin/sh mkbr.sh reset bridge0 tap0 tap1  _hostintf_

Substitute your host ethernet interface (igb0, em0, etc.) for “hostintf” in the above command. This script will create bridge0 and attach two taps and the ethernet interface. Run ifconfig -a to see all the devices in their new configuration. It also sets the required sysctls to work with tap(4) devices.

Install the Virtual Machines

Note that this install creates a FreeBSD system with a single interface - em0. The interface is bound to a tap(4) device defined above.

We will assume that the directory layout is as shown above and the FreeBSD iso file is in /home/foo/LAB/ISO/fbsd.iso.

The below script will start Qemu set to boot the fbsd.iso image and configure the virtual machine with 4GB memory, a vga card, hard disk, and one ethernet port.

Put the below text into a script file (e.g. left.sh) execute this script under sudo(8). Note the use of relative file paths.

#!/bin/sh

# Basic Qemu vm startup script for booting FreeBSD

/usr/local/bin/qemu-system-x86_64 -monitor none \
  -cpu qemu64 \
  -vga cirrus \
  -m 4096    \
  -cdrom ../ISO/fbsd.iso \
  -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \
  -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/left.img \
  -device virtio-blk,drive=drive0  \
  -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \
  -name \"left\"

Qemu will start up a virtual machine and boot the FreeBSD iso.

TIP: If you click into the Qemu console window, it will “grab” the mouse. To get your mouse back, type “Ctl-Alt-G”.

During the installation there are several points to note:

  • Select to use UFS as the filesystem. ZFS does not perform well with small memory sizes.
  • Do not select to use ntpd or ntpdate.
  • For em0, use 203.0.113.10 , netmask 255.255.255.0, with default gateway 203.0.113.100.
  • Do not fill in any DNS nameservers or search path.
  • When adding the default user, ensure they are a member of the wheel group.

Once the installation completes, the virtual machine should reboot into the newly installed FreeBSD image. Several additional steps are required to configure the virtual machine:

Login as root. To update the system, you will first have to override the network settings to get a working DHCP address for the VM. (Note that these instructions only work on a wired system and will not work on a wireless only system. Ensure that the host ethernet interface is connected to the same bridge as em0.)

The examples here are shown for the left VM. When making changes to the right VM, use 203.0.113.50 .

   # ifconfig em0 203.0.113.10 delete	(this only changes the settings for this session)
   # dhclient em0			(this will load a valid IP on the local LAN)
   # freebsd-update fetch install	(download all OS patches since the release)
   # reboot				(reboot into latest patch level)

   Note: you may need to run the `left.sh` or `right.sh` script again to
   restart the Qemu virtual machine.

Login as root again and perform the same network override as above. You can now add any packages you want for your system:

   # ifconfig em0 203.0.113.10 delete
   # dhclinet em0
   # pkg install cmdwatch nmap lynx nginx bash 
   # reboot

That’s all you need for the left VM.

Put the below text into a script file (e.g. right.sh) and execute this script under sudo(8) to create the right VM:

#!/bin/sh

# Basic Qemu vm startup script for booting FreeBSD

/usr/local/bin/qemu-system-x86_64 -monitor none \
  -cpu qemu64 \
  -vga cirrus \
  -m 4096    \
  -cdrom ../ISO/fbsd.iso \
  -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \
  -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/right.img \
  -device virtio-blk,drive=drive0  \
  -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:72:69:67:68:74 \
  -name \"right\"

Once the second VM is installed, update it as shown above.

You may find it desireable to disable sendmail. Put the following statements at the end of /etc/rc.conf:

  sendmail_enable="NONE"
  sendmail_submit_enable="NO"
  sendmail_outbound_enable="NO"
  sendmail_msp_queue_enable="NO"

On the next reboot, sendmail and its associated components will not start.

Adding the Serial Port Interface to the VMs

You may have noticed that we have not added the serial interfaces yet.

Each Qemu virtual machine (VM) running FreeBSD needs ‘console=“comconsole” ' in /boot/loader.conf to allow use of the Qemu serial port. The serial port is redirected to a TCP port on the host system at VM startup and the Qemu monitor waits to activate the guest VM until a telnet(1) connection occurs on that port. Once the FreeBSD system starts booting and recognizes the console directive in /boot/loader.conf it starts up a serial console. The Qemu monitor detects this and directs the necessary character I/O on that serial port to the TCP port on the host.

It is important to note that the this serial redirect over TCP takes place outside the virtual machine. There is no interaction with any network on the virtual machine and thus it is not subject to any firewall rules. Think of it just like a dumb terminal sitting on an RS-232 port on a real machine.

Also required are changes to the startup scripts for left.sh and right.sh.

Consider copying them as left+serial.sh and right+serial.sh and making the below changes.

left+serial.sh:

#!/bin/sh

# Basic Qemu vm startup script for booting FreeBSD


echo
echo "NOTE!!! telnet server running!"
echo "To start QEMU, start another session and telnet to localhost 4410 "
echo


/usr/local/bin/qemu-system-x86_64 -monitor none \
  -serial telnet:localhost:4410,server=on,wait=on \
  -display gtk \
  -cpu qemu64 \
  -vga cirrus \
  -m 4096    \
  -cdrom ../ISO/fbsd.iso \
  -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \
  -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/left.img \
  -device virtio-blk,drive=drive0  \
  -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \
  -name \"left\"

and right+serial.sh:

#!/bin/sh

# Basic Qemu vm startup script for booting FreeBSD

echo
echo "NOTE!!! telnet server running!"
echo "To start QEMU, start another session and telnet to localhost 4450 "
echo

/usr/local/bin/qemu-system-x86_64 -monitor none \
  -serial telnet:localhost:4450,server=on,wait=on \
  -display gtk \
  -cpu qemu64 \
  -vga cirrus \
  -m 4096    \
  -cdrom ../ISO/fbsd.iso \
  -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \
  -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/right.img \
  -device virtio-blk,drive=drive0  \
  -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:72:69:67:68:74 \
  -name \"right\"

Running these scripts is a bit different. Here’s an example running right+serial.sh:

[jpb@jpbsd:~/LAB/SCRIPTS]$ sudo /bin/sh right+serial.sh 

NOTE!!! telnet server running!
To start QEMU, start another session and telnet to localhost 4450 

qemu-system-x86_64: -serial telnet:localhost:4450,server=on,wait=on: info: QEMU waiting for connection on: disconnected:telnet:::1:4450,server=on

Start up another terminal session and telnet to port 4450:

[jpb@jpbsd:~]$ telnet localhost 4450

The new terminal session starts up the Qemu VGA console and starts up the boot process in the session as if on a real serial port.

Figure 2: Qemu Serial Port Operation

Figure 2: Qemu Serial Port Operation

Eventually you end up with a console session, and a serial session as shown in Figure 3:

Figure 3: Qemu With Console and Serial Sessions

Figure 3: Qemu With Console and Serial Sessions

You now have two working virtual machines with a console and a serial port session for each one as shown in Figure 4:

Figure 3: Two Qemu VMs With Console and Serial Sessions

Figure 3: Two Qemu VMs With Console and Serial Sessions

Notes on Using the Serial Console

In the VMs, use “vidcontrol black lightwhite” to reverse the video if you want. It has no effect on the serial console. Also, on the serial console if you resize the window, execute the resizewin(1) command to update the terminal size.

Disabling syslog messages to the console in the virtual machines

You may find it desirable (even necessary) to stop syslog message from being sent to the console (either the Qemu console, or the serial port).

To configure syslog to stop logging to the console perform thes steps as root:

# # Configure a file to receive console messages:
#
# touch /var/log/console.log
#
# chmod 0600 /var/log/console.log
#

Then, modify the line in /etc/syslog.conf to appear (instead of /dev/console):

*.err;kern.warning;auth.notice;mail.crit        /var/log/console.log

And, if necessary:

# service syslogd restart

All messages previously bound for the console, will be directed to /var/log/console.log instead.

Enjoy!