I manage virtualization servers, and we have full control over the hypervisors. The administration of the virtual machines is handled by the clients.

Ubuntu is the most commonly used operating system on our virtual machines. We deploy them using cloud images. The initial configuration is managed by cloud-init, which takes care of networking and SSH keys. It also automatically installs the qemu-guest-agent package.

Running a virt-inspector on the image we use provides us with details about this package.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

[fifi@kwaks:~]# virt-inspector noble-server-cloudimg-amd64.prep.img | xmllint --xpath '//application[name="qemu-guest-agent"]' -
<application>
        <name>qemu-guest-agent</name>
        <epoch>1</epoch>
        <version>8.2.2+ds</version>
        <release>0ubuntu1</release>
        <arch>amd64</arch>
        <url>http://www.qemu.org/</url>
        <source_package>qemu</source_package>
        <summary>Guest-side qemu-system agent</summary>
        <description>QEMU is a fast processor emulator: currently the package supports Alpha, ARM,
CRIS, i386, LoongArch, M68k (ColdFire), MicroBlaze, MIPS, PowerPC, RISC-V,
S390x, SH4, SPARC, x86-64, Xtensa and other emulations. By using dynamic
translation it achieves reasonable speed while being easy to port on new host
CPUs.
.
This package provides a daemon (agent) to run inside qemu-system
guests (full system emulation).  It communicates with the host using
a virtio-serial channel org.qemu.guest_agent.0, and allows one to perform
some functions in the guest from the host, including:
 - querying and setting guest system time
 - performing guest filesystem sync operation
 - initiating guest shutdown or suspend to ram
 - accessing guest files
 - freezing/thawing guest filesystem operations
 - others.
.
Install this package on a system which is running as guest inside
qemu virtual machine.  It is not used on the host.</description>
      </application>

This package installs a systemd unit that, in a virtualization context, allows the hypervisor to send commands such as fsfreeze and fsthaw. These commands enable clean snapshots of virtual machines. It can also be used to trigger TRIM operations on the virtual machines’ storage, speeding up live migrations with local storage.

Sometimes, our clients lose access to their virtual machines, often for the same reasons:

  • Firewall misconfiguration
  • Network misconfiguration
  • Deletion of SSH keys

Recovering a Shell

Overly aggressive firewall

1
2

[fifi@kwaks:~]# ssh vm1.internal.com
ssh: connect to host vm1.internal.com port 22: Connection refused

We connect to the hypervisor and examine the current nftables rules in place.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

[fifi@kwaks:~]# qm guest exec 100 -- /bin/bash -c "nft list ruleset" | jq -r '.["out-data"]' | echo -e "$(cat)"
table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
	}

	chain output {
		type filter hook output priority filter; policy accept;
	}
}

On the virtual machine side, we can see traces of our commands in the logs of the qemu-guest-agent unit.

1
2

root@vm1:/home/ubuntu# journalctl -u qemu-guest-agent -f
Jan 28 23:20:46 vm1 qemu-ga[632]: info: guest-exec called: "/bin/bash -c nft list ruleset"

In this case, we can either add an nftables rule to allow our administrative traffic or simply stop nftables.

1
2
3
4
5

[fifi@kwaks:~]# qm guest exec 100 -- /bin/bash -c "nft add rule inet filter input tcp dport 22 accept"
{
   "exitcode" : 0,
   "exited" : 1
}

or

1
2
3
4
5

[fifi@kwaks:~]# qm guest exec 100 -- /bin/bash -c "systemctl stop nftables"
{
   "exitcode" : 0,
   "exited" : 1
}

The VM is accessible again:

1
2

[fifi@kwaks:~]# ssh vm1.internal.com
Welcome to Ubuntu 24.04 LTS

Network Misconfiguration

Here, our VM no longer has an IP address:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[fifi@kwaks:~]# qm guest cmd 100 network-get-interfaces
[
   {
      "hardware-address" : "dc:00:b0:bd:d9:0a",
      "ip-addresses" : [
         {
            "ip-address" : "fe80::be24:11ff:febd:d90a",
            "ip-address-type" : "ipv6",
            "prefix" : 64
         }
      ],
      "name" : "eth0",
      "statistics" : {
         "rx-bytes" : 1114133363,
         "rx-dropped" : 1,
         "rx-errs" : 0,
         "rx-packets" : 10006089,
         "tx-bytes" : 2728706943,
         "tx-dropped" : 0,
         "tx-errs" : 0,
         "tx-packets" : 3165011
      }
   }
]

In such cases, we can use the qemu-agent to modify the network configuration or simply rely on link-local addresses.

1
2

[fifi@kwaks:~]# ssh fe80::be24:11ff:febd:d90a%br0.2
Welcome to Ubuntu 24.04 LTS

The percentage symbol in the ssh command allows the Linux system to specify which interface should send the traffic. This is because IPv6 link-local addresses are not globally unique but are unique only on a specific link. This behavior is defined in RFC 4007.

Deletion of SSH Keys

In the case of a client who wants to add a key after their authorized_keys file was erased or their SSH key was lost.

1
2
3
4
5

[fifi@kwaks:~]# qm guest exec 100 -- /bin/bash -c "echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBXz5q/J6LhP/N8uW9T5I8ENyGxnFMq3k+aUX/T31xFR riri@coin' >> /root/.ssh/authorized_keys"
{
   "exitcode" : 0,
   "exited" : 1
}

The VM is accessible again:

1
2

[fifi@kwaks:~]# ssh vm1.internal.com
Welcome to Ubuntu 24.04 LTS

Adding Restrictions to qemu-guest-agent Capabilities

Finally, here’s a tip to help protect against arbitrary code execution with the qemu-guest-agent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

root@vm1:~# cat /etc/systemd/system/qemu-guest-agent.service
[Unit]
Description=QEMU Guest Agent
BindsTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
After=dev-virtio\x2dports-org.qemu.guest_agent.0.device

[Service]
ExecStart=/usr/sbin/qemu-ga -b guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status
Restart=always
RestartSec=0

With this configuration, the administrator will receive an error if they attempt to call a guest exec.

1
2

[fifi@kwaks:~]# qm guest exec 100 -- /bin/bash -c "echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBXz5q/J6LhP/N8uW9T5I8ENyGxnFMq3k+aUX/T31xFR riri@coin' >> /root/.ssh/authorized_keys"
Agent error: Command guest-exec has been disabled