Static site for www.danielmoch.com
git clone git://git.danielmoch.com/danielmoch.com.git
Log | Files | Refs | LICENSE

using-qemu-without-pulling-your-hair-out.md (7437B)

      1 title: Using QEMU Without Pulling Your Hair Out
      2 author: Daniel Moch
      3 copyright: 2018, Daniel Moch
      4 date: 2019-07-15 21:25:02 UTC-04:00
      5 category: technology
      6 description: Tips for keeping your sanity with a very powerful tool
      8 I make it a rule to choose my tools carefully and to invest the time
      9 to learn them deeply. QEMU has been one of those tools that I've
     10 wanted to learn how to use for a long time, but was always a bit
     11 intimidated. I actually had been able to use it indirectly via
     12 libvirt, but it felt like I was cheating my rule by using one tool to
     13 manage another. Despite my vague sense of guilt, things continued this
     14 way until I read a recent(ish) [introductory post on
     15 QEMU](https://drewdevault.com/2018/09/10/Getting-started-with-qemu.html)
     16 by Drew DeVault. The article is well written (as per usual for
     17 DeVault), and you'd do well to read it before continuing here. The
     18 point is that it was the kick in the pants I needed to finally roll up
     19 my sleeves and learn some QEMU.
     21 The process of gaining some level of mastery over QEMU ended up being
     22 a fair bit more painful than I had anticipated, and so I wanted to
     23 capture some of my lessons learned over and above the
     24 introductory-level topics. The hard lessons were, by and large, not
     25 related directly to QEMU per-se, but more how to manage QEMU VM's. I
     26 use my virtual machines to isolate environments for various reasons,
     27 and so I need ways to automate their management. In particular, I had
     28 two needs that took some time to work out satisfactorily.
     30 1. Starting VM's automatically on system startup (and cleanly shutting
     31 them down
     32    with the host system).
     34 2. Securely and remotely accessing VM consoles.
     36 Let's take each of those two issues in turn.
     38 Automatic VM Management
     39 -----------------------
     41 For our purposes, what I mean by automatic management of VM's is just
     42 what I said above. If I need to restart the host server, I want the
     43 VM's to cleanly shut down with the host system, and come back up
     44 automatically after the host restarts. Since this is the kind of thing
     45 init systems are designed to do, it's only natural that we start there
     46 as a place to design our VM management infrastructure.
     48 So we just tell our init system to signal QEMU to shut down any
     49 running VM's and we're good right? In theory yes, but in reality
     50 QEMU's management interface is a bit tricky to script interactions
     51 with. There is a `-monitor` switch that allows you to configure a
     52 very powerful management interface, and you'll need to use that
     53 because the default is to attach that interface to a virtual console
     54 in the VM itself (or stdio, if you're not running a graphical
     55 interface locally). There are several options for configuring the
     56 monitor and the device it's connected to, but the best compromise I
     57 found between convenience and security was to make it available via a
     58 UNIX socket.
     60 If you've read DeVault's entry already, then you know that QEMU allows
     61 you to configure anything you could want via the command line. After
     62 deciding how to expose the monitor to the init system (systemd in my
     63 case), the rest came together pretty quickly. Here's what my service
     64 file looks like:
     66 		[Unit]
     67 		Description=QEMU virtual machine
     68 		Wants=network-online.target
     69 		After=network-online.target
     71 		[Service]
     72 		User=qemu
     73 		Group=qemu
     74 		UMask=0007
     75 		Environment=SMP=1
     76 		EnvironmentFile=/etc/conf.d/qemu.d/%I
     77 		ExecStart=/bin/sh -c "/usr/bin/qemu-${TYPE} -enable-kvm -smp ${SMP} -spice unix,disable-ticketing,addr=/run/qemu/%I.sock -m ${MEMORY} -nic bridge,br=${BRIDGE},mac=${MAC_ADDR},model=virtio -kernel /var/lib/qemu/%I/vmlinuz-linux -initrd /var/lib/qemu/%I/initramfs-linux-fallback.img -drive file=/var/lib/qemu/%I/%I.qcow2,media=disk,if=virtio -append 'root=/dev/vda rw' -monitor unix:/run/qemu/%I-mon.sock,server,nowait -name %I"
     78 		ExecStop=/bin/sh -c "echo system_powerdown | nc -U /run/qemu/%I-mon.sock"
     80 		[Install]
     81 		WantedBy=multi-user.target
     83 The `%I` should clue you in that this is a [service
     84 template](https://www.freedesktop.org/software/systemd/man/systemd.service.html#Service%20Templates),
     85 which is a nice convenience if you plan to run more than one VM as a
     86 service. This allows multiple VM's to use the same service file via a
     87 symlink. For example, a symlink from qemu@webhost.service to
     88 qemu@.service would cause systemd to replace `%I` with `webhost`.
     89 In-depth description of service templates is beyond the scope of this
     90 post, but the link above should be sufficient to answer additional
     91 questions. The last point I'll make here is that the netcat (nc)
     92 implementation used in `ExecStop` must be OpenBSD netcat, otherwise
     93 the service will not shut down cleanly. Other implementations will
     94 disconnect immediately after sending the `system_powerdown` message,
     95 while OpenBSD netcat waits for the socket to close.
     97 It's also worth taking a moment to stress how important the `UMask`
     98 directive in the above service template is. QEMU uses this to set
     99 permissions for the files it creates (including sockets), so we use
    100 this to secure our monitor and console sockets. A umask of 0007
    101 directs QEMU to create any files with full permissions for the qemu
    102 user and group, while giving no global permissions.
    104 All that's missing then is the environment file, and that looks like
    105 the following:
    107 		TYPE=system-x86_64
    108 		SMP=cores=1,threads=2
    109 		MEMORY=4G
    110 		MAC_ADDR=52:54:BE:EF:00:01
    111 		BRIDGE=br0
    113 The point of the environment file is to be tailored to your needs, so
    114 don't just blindly copy this. In particular, the `BRIDGE` device
    115 will need to exist, otherwise the service will fail. It bears
    116 mentioning that we use a bridge device so that the VM can appear like
    117 it's own machine to the outside world (and thus we can route traffic
    118 to it).
    120 So much for automating VM startup/shutdown, let's talk about how to
    121 access the console.
    123 Accessing Your VM Console
    124 -------------------------
    126 Again, QEMU has a plethora of options for accessing the VM console,
    127 both local and remote. Since I run my VM's on a server, I wanted
    128 something that would allow remote access, but I also wanted something
    129 reasonably secure. UNIX sockets end up being a good,
    130 middle-of-the-road solution again. They're treated like files, with
    131 all of the standard UNIX permissions, but it's also easy to route
    132 traffic from a remove machine to a UNIX socket via SSH.
    134 The applicable switch to achieve this configuration is `-spice`. In
    135 the above service template, you see the full switch reads::
    137 		-spice unix,disable-ticketing,addr=/run/qemu/%I.sock
    139 `unix` configures QEMU to use a UNIX socket (as opposed to, say, a
    140 TCP port), `disable-ticketing` configures the console without an
    141 additional password (this is okay since we're relying on UNIX file
    142 permissions), and `addr` gives the socket path.
    144 Now if you want to access the console remotely, it's as simple as
    145 setting up a forwarding socket via SSH and connecting your local SPICE
    146 client to it. Here's a shell script I whipped up to wrap that
    147 behavior:
    149 		#!/bin/sh
    150 		uid=`id -u`
    151 		path=/run/user/$uid/qemu
    153 		if [ ! -d $path ]
    154 		then
    155 		        mkdir -p $path
    156 		        chmod 700 $path
    157 		fi
    159 		ssh -NL $path/$1.sock:/run/qemu/$1.sock urbock.djmoch.org &
    160 		pid=$!
    162 		while [ ! -S $path/$1.sock ]
    163 		do
    164 		        sleep 1
    165 		done
    167 		spicy --uri="spice+unix://$path/$1.sock"
    168 		kill $!
    169 		rm $path/$1.sock
    171 And that's how I learned to use QEMU without pulling my hair out. It's
    172 a great tool, and I'm glad I took the time to learn how to use it. I
    173 suggest you do the same!