How to run QEMU full virtualization with bridged networking using NetworkManager under CentOS 8

In addition to the previously presented article on the subject Howto do QEMU full virtualization with bridged networking this one shows how to run a QEMU virtual machine with a bridge networking on the host server configured only by using the NetworkManager cli – nmcli.

It is worth mentioning the bridge interface presented in this article is a local bridge device for the server and no Internet addresses or real (or main or Internet-connected) network cards are bound to it. So no MAC addresses of slaved bridged devices will leave the server.
If a network bridge, which includes the Internet (main) server network device is needed, for example, to set real IPs in a virtual machine, there is another article on the bridge networking subject – Replace current interface configuration with a bridge device using nmcli (NetworkManager)

Summary

  1. Add bridge and TUN/TAP device.
  2. Install QEMU.
  3. Create QEMU local disk.
  4. Run a QEMU virtual server.

STEP 1) Add bridge and TUN/TAP device.

[root@srv ~]# nmcli connection add type bridge ifname br0 con-name br0 ipv4.method manual ipv4.addresses "192.168.0.1/24"
Connection 'br0' (ad6878c8-1e06-4af8-a81f-1eb39e761df8) successfully added.
[root@srv ~]# nmcli connection up br0
Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
[root@srv ~]# nmcli connection add type tun ifname tap0 con-name tap0 mode tap owner 0 ip4 0.0.0.0/24
Connection 'tap0' (dacee2be-a14b-4cf5-83d4-96d072a96725) successfully added.
[root@srv ~]# nmcli con add type bridge-slave ifname tap0 master br0
Connection 'bridge-slave-tap0' (66490382-b239-4eb2-ae1d-ee811e39596c) successfully added.
[root@srv ~]# nmcli con
NAME               UUID                                  TYPE      DEVICE 
System eno1        abf4c85b-57cc-4484-4fa9-b4a71689c359  ethernet  eno1   
br0                ad6878c8-1e06-4af8-a81f-1eb39e761df8  bridge    br0    
tap0               dacee2be-a14b-4cf5-83d4-96d072a96725  tun       tap0   
bridge-slave-tap0  66490382-b239-4eb2-ae1d-ee811e39596c  ethernet  -- 

First, a bridge device is added with manual IP. If the IP is skipped the bridge interface br0 would have DHCP enabled by default, which may not be the desired.
More detailed information on how to create and add TUN/TAP device with the NetworkManager here – Create bridge and add TUN/TAP device using NetworkManager nmcli under CentOS 8

STEP 2) Install QEMU.

Install the QEMU virtual tools under CentOS 8 Stream. At present, the QEMU version is 6.2, which is pretty new.

[root@srv ~]# dnf install -y qemu-img qemu-kvm-common qemu-kvm-core
Last metadata expiration check: 1:03:59 ago on Thu Apr 14 14:18:51 2022.
Dependencies resolved.
==============================================================================================================
 Package                        Arch     Version                                            Repository   Size
==============================================================================================================
Installing:
 qemu-img                       x86_64   15:6.2.0-5.module_el8.6.0+1087+b42c8331            appstream   2.1 M
 qemu-kvm-common                x86_64   15:6.2.0-5.module_el8.6.0+1087+b42c8331            appstream   1.0 M
 qemu-kvm-core                  x86_64   15:6.2.0-5.module_el8.6.0+1087+b42c8331            appstream   3.4 M
Installing dependencies:
 device-mapper-multipath-libs   x86_64   0.8.4-22.el8                                       baseos      325 k
 edk2-ovmf                      noarch   20220126gitbb1bba3d77-2.el8                        appstream   3.6 M
 ipxe-roms-qemu                 noarch   20181214-9.git133f4c47.el8                         appstream   1.2 M
 libfdt                         x86_64   1.6.0-1.el8                                        appstream    32 k
 libpmem                        x86_64   1.6.1-1.el8                                        appstream    79 k
 librdmacm                      x86_64   37.2-1.el8                                         baseos       78 k
 seabios-bin                    noarch   1.15.0-1.module_el8.6.0+1087+b42c8331              appstream   136 k
 seavgabios-bin                 noarch   1.15.0-1.module_el8.6.0+1087+b42c8331              appstream    43 k
 sgabios-bin                    noarch   1:0.20170427git-3.module_el8.6.0+983+a7505f3f      appstream    13 k
 userspace-rcu                  x86_64   0.10.1-4.el8                                       baseos      101 k

Transaction Summary
==============================================================================================================
Install  13 Packages

Total download size: 12 M
Installed size: 44 M
Downloading Packages:
(1/13): libfdt-1.6.0-1.el8.x86_64.rpm                                         163 kB/s |  32 kB     00:00    
(2/13): libpmem-1.6.1-1.el8.x86_64.rpm                                        606 kB/s |  79 kB     00:00    
(3/13): ipxe-roms-qemu-20181214-9.git133f4c47.el8.noarch.rpm                  2.8 MB/s | 1.2 MB     00:00    
(4/13): edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch.rpm                      7.0 MB/s | 3.6 MB     00:00    
(5/13): qemu-kvm-common-6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64.rpm       4.2 MB/s | 1.0 MB     00:00    
(6/13): qemu-kvm-core-6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64.rpm          14 MB/s | 3.4 MB     00:00    
(7/13): seabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch.rpm          1.2 MB/s | 136 kB     00:00    
(8/13): seavgabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch.rpm       741 kB/s |  43 kB     00:00    
(9/13): sgabios-bin-0.20170427git-3.module_el8.6.0+983+a7505f3f.noarch.rpm    179 kB/s |  13 kB     00:00    
(10/13): qemu-img-6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64.rpm             2.0 MB/s | 2.1 MB     00:01    
(11/13): librdmacm-37.2-1.el8.x86_64.rpm                                       39 kB/s |  78 kB     00:02    
(12/13): device-mapper-multipath-libs-0.8.4-22.el8.x86_64.rpm                 152 kB/s | 325 kB     00:02    
(13/13): userspace-rcu-0.10.1-4.el8.x86_64.rpm                                 64 kB/s | 101 kB     00:01    
--------------------------------------------------------------------------------------------------------------
Total                                                                         3.8 MB/s |  12 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1 
  Installing       : userspace-rcu-0.10.1-4.el8.x86_64                                                   1/13 
  Running scriptlet: userspace-rcu-0.10.1-4.el8.x86_64                                                   1/13 
  Installing       : device-mapper-multipath-libs-0.8.4-22.el8.x86_64                                    2/13 
  Running scriptlet: device-mapper-multipath-libs-0.8.4-22.el8.x86_64                                    2/13 
  Installing       : librdmacm-37.2-1.el8.x86_64                                                         3/13 
  Running scriptlet: librdmacm-37.2-1.el8.x86_64                                                         3/13 
  Installing       : sgabios-bin-1:0.20170427git-3.module_el8.6.0+983+a7505f3f.noarch                    4/13 
  Installing       : seavgabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                         5/13 
  Installing       : seabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                            6/13 
  Installing       : qemu-img-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                             7/13 
  Installing       : libpmem-1.6.1-1.el8.x86_64                                                          8/13 
  Running scriptlet: libpmem-1.6.1-1.el8.x86_64                                                          8/13 
  Installing       : libfdt-1.6.0-1.el8.x86_64                                                           9/13 
  Running scriptlet: libfdt-1.6.0-1.el8.x86_64                                                           9/13 
  Installing       : ipxe-roms-qemu-20181214-9.git133f4c47.el8.noarch                                   10/13 
  Installing       : qemu-kvm-common-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                     11/13 
  Running scriptlet: qemu-kvm-common-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                     11/13 
  Installing       : edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch                                       12/13 
  Installing       : qemu-kvm-core-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                       13/13 
  Running scriptlet: qemu-kvm-core-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                       13/13 
  Verifying        : edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch                                        1/13 
  Verifying        : ipxe-roms-qemu-20181214-9.git133f4c47.el8.noarch                                    2/13 
  Verifying        : libfdt-1.6.0-1.el8.x86_64                                                           3/13 
  Verifying        : libpmem-1.6.1-1.el8.x86_64                                                          4/13 
  Verifying        : qemu-img-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                             5/13 
  Verifying        : qemu-kvm-common-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                      6/13 
  Verifying        : qemu-kvm-core-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                        7/13 
  Verifying        : seabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                            8/13 
  Verifying        : seavgabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                         9/13 
  Verifying        : sgabios-bin-1:0.20170427git-3.module_el8.6.0+983+a7505f3f.noarch                   10/13 
  Verifying        : device-mapper-multipath-libs-0.8.4-22.el8.x86_64                                   11/13 
  Verifying        : librdmacm-37.2-1.el8.x86_64                                                        12/13 
  Verifying        : userspace-rcu-0.10.1-4.el8.x86_64                                                  13/13 

Installed:
  device-mapper-multipath-libs-0.8.4-22.el8.x86_64                                                            
  edk2-ovmf-20220126gitbb1bba3d77-2.el8.noarch                                                                
  ipxe-roms-qemu-20181214-9.git133f4c47.el8.noarch                                                            
  libfdt-1.6.0-1.el8.x86_64                                                                                   
  libpmem-1.6.1-1.el8.x86_64                                                                                  
  librdmacm-37.2-1.el8.x86_64                                                                                 
  qemu-img-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                                                     
  qemu-kvm-common-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                                              
  qemu-kvm-core-15:6.2.0-5.module_el8.6.0+1087+b42c8331.x86_64                                                
  seabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                                                    
  seavgabios-bin-1.15.0-1.module_el8.6.0+1087+b42c8331.noarch                                                 
  sgabios-bin-1:0.20170427git-3.module_el8.6.0+983+a7505f3f.noarch                                            
  userspace-rcu-0.10.1-4.el8.x86_64                                                                           

Complete!

STEP 3) Create QEMU local disk.

The hard disk file for the QEMU virtual machine will be under path /srv/qemu/test. The last argument is the size of the file.

[root@srv ~]# mkdir -p /srv/qemu/test
[root@srv ~]# qemu-img create -f qcow2 /srv/qemu/test/test_harddisk.qcow2 100G
Formatting '/srv/qemu/test/test_harddisk.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=107374182400 lazy_refcounts=off refcount_bits=16

The hard disk is of 100G size provided to the virtual machine, but the actual size varies because qcow2 grows when the data is written.

STEP 4) Run a QEMU virtual server.

Here is an example command to run a QEMU virtual server. First, add masquerade with firewall-cmd to enable NAT, i.e. Internet in the virtual machine:

[root@srv ~]# firewall-cmd --permanent --add-masquerade
success
[root@srv ~]# firewall-cmd --reload
success

Second, run the QEMU virtual machine. The QEMU KVM executable is /usr/libexec/qemu-kvm:

/usr/libexec/qemu-kvm -cpu host,-sse4.1,-sse4.2 -smp 2,maxcpus=8 -daemonize -vnc 192.168.1.10:1 \
-drive file=/srv/qemu/test/test_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio \
-cdrom /srv/qemu/test/install-amd64-minimal-20220412T191925Z.iso -boot d -m 8192 \
-net nic,model=virtio,macaddr=e6:2b:b3:40:81:0c -net tap,ifname=tap0,script=no,downscript=no \
-monitor telnet:127.0.0.1:5801,server,nowait -writeconfig /srv/qemu/test/test.qcow2.conf

The command above will :

  • “-cpu host” – will expose all supported host CPU features (only supported in KVM mode)
  • “-smp 4” – sets 4 processors to the virtual machine
  • “-daemonize” – start the command in daemon mode
  • “-vnc 192.168.1.10:1” – start a VNC server on this IP:PORT = 192.168.1.10:5901, the IP must present on the server or you can use 0.0.0.0:1 for 0.0.0.0:5901, but in every situation limit the access by a firewall
  • -drive file=/srv/qemu/test/test_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio – set the main hard drive of the system
  • “-boot d” – boot from the first hard drive
  • “-net nic,model=virtio,macaddr=e6:2b:b3:40:81:0c -net tap,ifname=tap0,script=no,downscript=no” – set the network interface using the tap device created by STEP 1).
  • “-m 8192” – set virtual RAM size to megs
  • “-monitor telnet:127.0.0.1:5801,server,nowait” – set the management console for the this virtual server, you can connect with:
    [root@srv ~]# telnet 127.0.0.1 5801
    Trying 127.0.0.1...
    Connected to 127.0.0.1.
    Escape character is '^]'.
    QEMU 6.2.0 monitor - type 'help' for more information
    (qemu) 
    telnet> quit
    Connection closed.
    

    When quitting the management console you must NOT exit the console with quite/exit or CTRL+d, becuause it will terminate the virtual server, you must disconnect from the console with “CTRL+]” and then quit the telnet shell. With the console you can hot add/remove CPU, RAM, network cards, pci devices, harddrives, start/stop/shutdown/reset the virtual machine and a lot more.

Interesting additional option to consider:

  • “-runas qemu” – run under user, you can run the whole virtual machine from a user created especially for it, no need to run it with root, even it is recommended to run it under unprivileged user

* Boot the virtual server from a virtual CD/DVD

Probably the first time booting you might need to boot from an installation disk, this could be done by the following command:

  1. “-boot c” – First boot device is now CD/DVD. “c” is for CD, “d” is for disk
  2. “-cdrom /mnt/storage1/disks/isos/CentOS-7-x86_64-NetInstall-1708.iso” – added the installation disk to the virtual machine

A newer QEMU version may need adding “script=no,downscript=no” to the tap0 interface!

/usr/libexec/qemu-kvm -cpu host,-sse4.1,-sse4.2 -smp 2,maxcpus=8 -daemonize -vnc 192.168.1.10:1 \
-drive file=/srv/qemu/test/test_harddisk.qcow2,index=0,cache=none,aio=threads,if=virtio \
-cdrom /srv/qemu/test/install-amd64-minimal-20220412T191925Z.iso -boot c -m 8192 \
-net nic,model=virtio,macaddr=e6:2b:b3:40:81:0c -net tap,ifname=tap0,script=no,downscript=no \
-monitor telnet:127.0.0.1:5801,server,nowait -writeconfig /srv/qemu/test/test.qcow2.conf

Leave a Reply

Your email address will not be published.