Howto do QEMU full virtualization with MacVTap networking

To use full virtualization under linux you can use QEMU and no other library or manager like virt-manager. QEMU is simple enough and with couple of parameters to it you can start KVM virtual machines with near native performance. To use KVM you must enable it in the BIOS of your server (or desktop machine).

Here a several simple step to start a KVM virtual server:

STEP 1) Enable KVM in the BIOS

• For Intel machine you must find option Intel Virtualization Technology (or Intel VT-x) probably in BIOS menu of Chipset, Advanced CPU Configuration or other.
• For AMD machine the virtualization cannot be disabled so it is enabled by default, but you can check for additional virtualization features to enable like Virtualization Extensions, Vanderpool and other.
• Enable also additional features – Intel VT-d or AMD IOMMU, if they are available.

Reboot your machine and check if the KVM is supported:

srv@local ~$ cat /proc/cpuinfo |grep -E "vmx|svm"
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap xsaveopt dtherm ida arat pln pts
...

STEP 2) Install QEMU

Under CentOS 7 you can just install couple of packets – that’s all you need:

yum install -y qemu qemu-common qemu-img qemu-kvm-common qemu-system-x86 qemu-user bridge-utils

Or under Ubuntu

apt-get install qemu-kvm bridge-utils

STEP 3) Prepare the network

srv@local ~$ ip link add link enp8s0f1 name macvtap0 type macvtap mode bridge
srv@local ~$ ip link set macvtap0 up

Here we create a macvtap0 device in bridge mode, these commands will create a tap device bridged to the network interface “enp8s0f1” (in our case, you must replace this device name with the device name you want to bridge your virtual machine network, probably the main interface of your server/desktop machine?). Only these two commands are needed, no other devices or network reload is needed.
The device will show in “ip addr”

7: macvtap0@enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 2e:51:7e:bb:44:ee brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2c51:7eff:febb:44ee/64 scope link 
       valid_lft forever preferred_lft forever

This setup could expose the MAC address of the macvtap device to the router port connected

STEP 4) Create a QEMU hard drive

Create a 100G file

srv@local ~$ cd /mnt/storage1/disks/
srv@local ~$ qemu-img create -f qcow2 vm_harddisk.qcow2 100G
Formatting 'vm_harddisk.qcow2', fmt=qcow2 size=107374182400 encryption=off cluster_size=65536 lazy_refcounts=off 

or you can enable encryption (but on every start of your virtual machine you must set the key through the qemu console to start the virtual machine):

srv@local ~$ qemu-img create -f qcow2 vm_harddisk_e.bin -o encryption 100G
Formatting 'vm_harddisk_e.bin', fmt=qcow2 size=107374182400 encryption=on cluster_size=65536 lazy_refcounts=off

STEP 5) Boot up the QEMU KVM virtual server

srv@local ~$ qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -runas qemu -daemonize -vnc 127.0.0.1:1 \
-drive file=/mnt/storage1/disks/vm_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio \
-boot d -net nic,model=virtio,macaddr=$(cat /sys/class/net/macvtap0/address) \
-net tap,fd=3 3<>/dev/tap$(cat /sys/class/net/macvtap0/ifindex) \
-balloon virtio -m 2048 -monitor telnet:127.0.0.1:5801,server,nowait

The command above will :

• “-enable-kvm” – enable the KVM – full virtualization with near native performance
• “-cpu host” – will expose all supported host CPU features (only supported in KVM mode)
• “-smp 4” – sets 4 processors to the virtual machine
• “-daemonize” – start the command in daemon mode
• “-runas qemu” – run under user, you can run thwo whole virtual machine from a user created especially for it, no need to run it with root, even it is recommended to run it under unprivileged user
• “-vnc 192.168.1.10:1” – start a VNC server on this IP:PORT = 192.168.1.10:5901, the IP must present on the server or you can use 0.0.0.0:1 for 0.0.0.0:5901, but in every situation limit the access by a firewall
• “-drive file=/mnt/storage1/disks/vm_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio” – set the main hard drive of the system
• “-boot d” – boot from the first hard drive
• “-net nic,model=virtio,macaddr=$(cat /sys/class/net/macvtap0/address) -net tap,fd=3 3<>/dev/tap$(cat /sys/class/net/macvtap0/ifindex)” – set the network interface using the tap device created by macvtap0 device (STEP 3)
• “-balloon virtio” – use balloon driver to be able to hot add or hot remove RAM
• “-m 2048” – set virtual RAM size to megs
• “-monitor telnet:127.0.0.1:5801,server,nowait” – set the management console for the this virtual server, you can connect with:

  srv@local ~$ telnet 127.0.0.1 5801
  Trying 127.0.0.1...
  Connected to 127.0.0.1.
  Escape character is '^]'.
  QEMU 2.0.0 monitor - type 'help' for more information
  (qemu) 
  <Press "CTRL+]">
  telnet> Connection closed.
  

  When quitting the management console you must NOT exit the console with quite/exit or CTRL+d, becuause it will terminate the virtual server, you must disconnect from the console with “CTRL+]” and then quit the telnet shell. With the console you can hot add/remove CPU, RAM, network cards, pci devices, harddrives, start/stop/shutdown/reset the virtual machine and a lot more.

Boot the virtual machine from the hard drive given by “-drive” with network “-net” (couple of options), the RAM uses baloon memory and could be adjusted on-the-fly and sets the vncserver to listen for connection on port IP:port = 192.168.1.1:5901 (probably you’ll want to change this with a the real IP of your server, but be careful to set up a firewall rule for 5901 – the vnc port) and a management console listening on IP:port 127.0.0.1:5801.

* Boot the virtual server from a virtual CD/DVD

Probably the first time booting you might need to boot from an installation disk, this could be done by the following command:

srv@local ~$ qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -runas qemu -daemonize -vnc 127.0.0.1:1 -cdrom /mnt/storage1/disks/isos/CentOS-7-x86_64-NetInstall-1708.iso -boot c -drive file=/mnt/storage1/disks/vm_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio -net nic,model=virtio,macaddr=$(cat /sys/class/net/macvtap0/address) -net tap,fd=3 3<>/dev/tap$(cat /sys/class/net/macvtap0/ifindex) -balloon virtio -m 2048 -monitor telnet:127.0.0.1:5801,server,nowait

The changes:

1. “-boot c” – First boot device is now CD/DVD. “c” is for CD, “d” is for disk
2. “-cdrom /mnt/storage1/disks/isos/CentOS-7-x86_64-NetInstall-1708.iso” – added the installation disk to the virtual machine