Appendix A. QEMU Setup

Table of Contents

It is highly advised to read through the entire installation procedure, including all notes and tips, before proceeding.

This appendix contains helpful information for getting QEMU installed and running on FreeBSD, and instructions for common use. The installation presumes a graphical desktop running on top of X Windows or Wayland. Because of this, the user environment, either X Window or Wayland, must allow for use of the DISPLAY variable in the local environment.

The examples in this book utilize virtual machines (VMs) using an SDL-based vt(4) console and also include a using a FreeBSD serial console. Instructions for setting up the VMs are below. Instructions for setting up and managing the serial consoles are found in Adding and Managing Serial Console Access to the VMs below.

Additional resources for understanding and using QEMU include:

Installation

QEMU is available as a package or a port. There are a large number of build options on the port, so in most cases it is best to install the package. sudo(8) will also need to be installed as well.

Altogether, there are nine virtual machines used in this book. The following procedure will make ready all nine VMs. However, all nine are not needed immediately. Most of the first half of this book can be done with just the firewall, internal, external1, and external2 VMs.

Setting Up the Initial Virtual Machines. Refer to paragraphs below.
Figure 1. Setting Up the Initial Virtual Machines

The initial setup is that shown in Figure 1.

A.1. QEMU and VM Installation Process

Follow the steps below to install and configure the QEMU virtual machines needed for this book. Various scripts are used to create virtual machines and set up bridge and tap devices. If desired, examine all scripts before use.

  1. On the FreeBSD host, install the necessary packages - qemu(1), sudo(8) (or doas(1)). Sudo, (or doas) is necessary for running the virtual machines as these QEMU configurations open a separate console window through SDL. The examples in this book use sudo.

    # pkg install qemu sudo

    Configure sudo as desired.

  2. Clone the scripts for this book from the ipfw-primer project on GitHub. Then fetch the current FreeBSD ISO.

    Note: The initial path can be any directory.
All scripts use relative directory addressing.
Adjust the paths below as necessary.
    % cd $HOME
% git clone https://github.com/jimmyb-gh/ipfw-primer.git
% cd ~/ipfw-primer/ipfw/ISO
% fetch https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/<latest version>/FreeBSD-<latest-version>-RELEASE-amd64-dvd1.iso

The example for FreeBSD 14.2 would be:
% fetch https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/14.2/FreeBSD-14.2-RELEASE-amd64-dvd1.iso

% # Link a shorter name to the ISO image.
% ln -s FreeBSD-<latest-version>-RELEASE-amd64-dvd1.iso fbsd.iso

  3. Using sudo, create the bridge and tap devices for the virtual machines to use. See the description of the mkbr.sh script in Using mkbr.sh for Bridge and Tap Setup.

    % cd ../HOST_SCRIPTS
% sudo /bin/sh mkbr.sh reset bridge0 tap0 tap1 tap2 tap3 tap5 tap6 tap7 tap9 tap12  hostintf <--- replace hostintf with host network interface (em0, bge0, etc.)

  4. As a normal user, create all VM image files. Each VM is 4GB except for the jail1 VM which is 12GB.

    % /bin/sh _CreateAllVMs.sh

The next command starts the installation for the firewall VM.
Once finished, return to this point and repeat for the following VMs:
external1, external2, external3, firewall2, internal, dnshost, v6only.

% sudo /bin/sh firewall.sh

Ignore the "NOTE!!! telnet server running..." message for now.
Instructions for setting up a serial console are found later in this setup guide.

    The FreeBSD installer should boot. Perform a standard installation of FreeBSD.

    During the installation note the following:

    • On all VMs except the jail1 VM, select to use UFS as the filesystem. ZFS does not perform well with small memory sizes. The jail1 VM has more memory and ZFS will be used to create jails on the jail1 VM. It is the only VM that requires ZFS.

    • For these installations, use DHCP for networking. If desired, configure IPv6 if supported by the local LAN.

    • When adding the default user, ensure they are a member of the wheel group.

      Once the installation completes, the virtual machine reboots into the newly installed FreeBSD image.

  5. Login as root, update the system, and reboot.

    # freebsd-update fetch install
# reboot

  6. On all virtual machines, install the packages listed below The nmap package brings in the version of ncat(1) used by scripts on the firewall and external VMs. nginx, lynx, cmdwatch, hping3, tsctp, and iperf3 will be used in later chapters.

    # pkg install nmap nginx lynx cmdwatch hping3 tsctp iperf3

  7. Finally, download ~/ipfw-primer/ipfw/VM_SCRIPTS/IPFW_root_bin.tgz file to all VMs. This tar file has a number of scripts needed for the virtual machines.

    Move the tarzip file into /root and extract the contents:

    On each VM, login as root, copy and untar the following file:

# scp user@hostip:~/ipfw-primer/ipfw/VM_SCRIPTS/IPFW_root_bin.tgz .
#
# mv IPFW_root_bin.tgz /root
#
# cd /root
#
# tar xvzf IPFW_root_bin.tgz
... files are extracted into /root/bin
#
# chmod +x /root/bin/*.sh

  8. Repeat the installation procedure for each virtual machine.

Some additional configurations are required for examples later in the book:

On the firewall and firewall2 VMs:

  • Add net.inet.ip.fowarding=1 and net.inet6.ip6.fowarding=1 to /etc/sysctl.conf.

On the external1 and internal VMs install these extra packages:

  • pkg install git

  • pkg install cmake

On each VM, navigate to /usr/local/www/nginx. Download the modified index.html file from the host system to replace the original:

  • scp user@host:~/ipfw-primer/ipfw/VM_SCRIPTS/VM_name/index.html .

On the firewall VM, download the bsdclat464.sh script

  • cd /root/bin

  • scp user@host:~/ipfw-primer/ipfw/VM_SCRIPTS/firewall/bsdclat464.sh .

On the *firewall2 VM, download the bsdplat464.sh script

  • cd /root/bin

  • scp user@host:~/ipfw-primer/ipfw/VM_SCRIPTS/firewall/bsdplat464.sh .

(End installation procedure.)

For this Quick Start, it is Ok to use DHCP for both VMs. In later examples there will be multiple external VMs using the 203.0.113.0/24 network and other private networks, all set up the same way and attached via tap(4) interfaces to one or more if_bridge(4) interfaces on the FreeBSD host.

To ensure the first two VMs are set up correctly, ping the firewall VM from the external1 VM and vice-versa. Communications should be successful. If not, check the above installation details and troubleshoot any network issues. It should be possible ping in both directions, and even ssh(1) from one VM to the other.

For additional helpful information on getting QEMU set up correctly, check the QEMU virtualization section in the FreeBSD Handbook.

If the mouse is clicked in the QEMU console window, QEMU will “grab” the mouse. If this happens,type, Ctl+Alt+G to release the mouse.

If suddenly, the QEMU console window is full screen, you may have accidentally typed Ctl+Alt+F. If this happens, retype Ctl+Alt+F to restore the desktop screen.

A.1.1. Disabling Syslog Messages to the Console in the Virtual Machines

It may be advantageous (even necessary) to stop syslog messages from being sent to the console (either the QEMU console, or the serial port).

To configure syslog to stop logging to the console, configure a file to receive console messages:

# touch /var/log/console.log
#
# chmod 0600 /var/log/console.log

Then, as root, modify the line in /etc/syslog.conf to read (instead of /dev/console):

*.err;kern.warning;auth.notice;mail.crit        /var/log/console.log

And, if necessary:

# service syslogd restart

All messages previously bound for the console, will be directed to /var/log/console.log instead.

Before continuing, there is one more piece to add to each VM - a serial console. A serial console permits examination of the state of each VM, independent of the main console.

A.1.2. Adding and Managing Serial Console Access to the VMs

Adding a serial console the FreeBSD VM

To add a serial console to each FreeBSD VM, start up the VM and edit the file /boot/loader.conf and add the line console=“comconsole” to allow use of the serial console. Reboot the VM to begin using the serial console. Note that FreeBSD diverts boot I/O to the serial console, so until the FreeBSD operating system is completely ready, output to the QEMU window will be limited.

The startup scripts for each VM are already configured to use a serial console. A single configuration line was added to the QEMU configuration to provide a serial console. The serial console is actually accessed over a telnet(1) session. The QEMU manual page, qemu(1), describes how the -serial keyword works in detail.

QEMU redirects the serial port I/O to a TCP port on the host system at VM startup, and allows a telnet(1) connection on the configured port on the host. Once the FreeBSD system starts booting and recognizes the console directive in /boot/loader.conf it redirects I/O to the serial console. QEMU detects this and manages the necessary character I/O on that serial port to the TCP port on the host.

It is important to note that the this serial redirect over TCP takes place outside the virtual machine. There is no interaction with any network on the virtual machine and thus it is not subject to any firewall rules. Think of it just like a "dumb terminal" sitting on an RS-232 serial port on a real machine.

Management of serial console windows on the FreeBSD host

Each QEMU VM generates a console window, and each serial device also needs its own window, potentially doubling the number of windows used.

Possible solutions are:

  • Separate windows for each QEMU VM (doubles the number of windows)

  • Use tabbed windows (available on XFCE and some other desktops)

  • Use a terminal multiplexer such as tmux(1) or screen(1)

The selected solution uses the multiplexer approach with the tmux(1) program for window management. Appendix D provides details on using both tmux(1) and screen(1). The following figures and descriptions use tmux(1).

Install tmux on the FreeBSD host with:

# pkg install tmux

and if necessary, copy the file swim.sh (or scim.sh for using screen(1)) from Appendix B into the HOST_SCRIPTS directory. If using scim.sh, follow the instructions in the script to set up a .screenrc file.

The figure below shows the use of the swim.sh tmux session manager. Run sh swim.sh in the HOST_SCRIPTS directory to start up the session manager.

Starting Up tmux(1) Session Manager. Refer to paragraphs below.
Figure 2. Starting Up tmux(1) Session Manager

The figure shows five named windows in one session (session [0]) with the tmux status line in green at the bottom:

  • 0:bash - a terminal window of the user running swim.sh

  • 1:firewall - a terminal window to access the firewall VM

  • 2:external1 - a terminal window to access the external1 VM

  • 3:external2 - a terminal window to access the external2 VM

  • 4:external3 - a terminal window to access the external3 VM

The current window is marked with the '*' character on the status bar.

Simplified tmux(1) Usage

tmux uses Ctl+b as its control key. To move from window to window use Ctl+b n to move to the next window or Ctl+b p to move to the previous window. Use Ctl+b ? for a list of all key bindings.

Type tmux kill-server in any host session shell window (not a VM window) to completely leave tmux.

Consult the tmux manual page tmux(1) for more usage details.

Accessing the QEMU Serial Consoles

To access the VM serial consoles, move to the indicated window and telnet to the port on the local host for that VM:

Move to the external1 window in tmux, then

% telnet localhost 4410
Trying ::1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/amd64 (external1) (ttyu0)

login:

To exit out of the telnet session, press Ctl+] then press q like this:

  login:  (type Ctl+])
  telnet> q
  Connection closed.
  %

There should now be two QEMU VMs (firewall and external1) started, with serial console sessions available through the tmux sessions as shown below.

Configure the FreeBSD host, firewall VM, and external1 VM with DHCP addressing as shown in the figure at the beginning of this Quick Start session. There should be full connectivity between the FreeBSD host, the firewall VM and the external1 VM.

Firewall and External1 VMs Startup with Serial Console. Refer to paragraphs below.
Figure 3. External1 and Firewall VMs Startup with Serial Console

Using ipfw to control traffic between these two QEMU VMs is discussed in the next chapter.

A.2. Using mkbr.sh for Bridge and Tap Setup

Included in Appendix B, the mkbr.sh script is used to set up if_bridge(4) and tap(4) devices on the FreeBSD host. These interfaces are required to allow the virtual machines to communicate with each other and, when suitably configured, to communicate with the outside world.

Many examples in this book include a statement such as:

# /bin/sh mkbr.sh reset bridge0 tap0 tap1 em0

or something similar.

The above invocation resets the kernel modules necessary for bridge and tap operation, then it creates one bridge, bridge0, and connects tap0, tap1, and em0. Here, "em0" refers to a host ethernet interface, but it can be any ethernet interface on the local machine.

The script can be used to create any number of bridges and taps for complex network designs. For example, the following invocation creates three bridges - bridge0 with tap0 and tap1 connected, bridge1 with tap2, tap3, and tap4 connected, and bridge2 with tap5 and host interface igb0 connnected.

%  sudo /bin/sh mkbr.sh reset bridge0 tap0 tap1 bridge1 tap2 tap3 tap4 bridge2 tap5 igb0

To add other taps to existing bridges, do not specify the "reset" parameter:

#  /bin/sh mkbr.sh bridge0 tap10 tap11  bridge1 tap12 tap13  ... etc.

To delete all bridge and tap devices:

#  /bin/sh mkbr.sh reset

Usually, the examples include an architecture diagram, like that shown earlier, and an invocation of mkbr.sh that will create the architecture shown.

Below is a handy chart to show the relationship of virtual machines and tap devices. Note that some virtual machines have more than one interface. The chart also shows how all virtual machines could be attached to the same bridge if needed for administrative purposes.

TAPLIST.TXT

This file contains just the taps that are needed to
connect all VMs to one bridge for admin purposes:

  +----------------------------------------------+
  |                                              |
  |                  bridge0                     |
  |                                              |
  +-+----+----+----+---+----+---+----+----+----+-+
   /    /    /    /    |    |    \    \    \    \
  tap0 tap1 tap2 tap3 tap5 tap6 tap7 tap9 tap12  host_interface
   |    |    |    |    |    |    |    |    |
   |    |    |    |    |    |    |    |    +- jail1:em0
   |    |    |    |    |    |    |    +- firewall2:em0
   |    |    |    |    |    |    +- dnshost:em0
   |    |    |    |    |    +- v6only:em0
   |    |    |    |    +- internal:em0
   |    |    |    +- external3:em0
   |    |    +- external2:em0
   |    +- external1:em0
   +- firewall:em0


The remaining taps are located on the VM listed:

tap4  - firewall:em1
tap8  - dnshost:em1
tap10 - firewall2:em1
tap11 - dnshost:em2

Home