VirtualBox on Mac Host: High CPU utilization for idle VM

Many people in the IT industry do use VirtualBox as a Type-2 Hypervisor to test software in virtual machines. I’m using it with Vagrant on my Macbook Pro 2019 (Intel with Ventura 13.2. when writing this Blog) to setup testcases for Oracle Databases or use a Windows VM for tests of Windows programs. Since migrating to Virtualbox V7 it feels as if programs like VBoxHeadless (when starting the VM through Vagrant) or VirtualBoxVM (when starting my Windows VM) take lots of CPU resources on my Mac host even though the VM itself is idle.

Analysis

The issue specifically happens when running queries against the DB, which produce high load. After the load has finished, I still see my host CPU busy for minutes, but the virtual CPUs in the VM are idle.

To get more information about what threads actually take CPU-time on MacOS the tool spindump can be quite useful. Let’s start with an idle VM after booting it through Vagrant:

$ vagrant up
getting Proxy Configuration from Host...
Bringing machine 'oracle-19c6-vagrant' up with 'virtualbox' provider...
==> oracle-19c6-vagrant: Clearing any previously set forwarded ports...
==> oracle-19c6-vagrant: Clearing any previously set network interfaces...
==> oracle-19c6-vagrant: Preparing network interfaces based on configuration...
    oracle-19c6-vagrant: Adapter 1: nat
==> oracle-19c6-vagrant: Forwarding ports...
    oracle-19c6-vagrant: 1521 (guest) => 1521 (host) (adapter 1)
    oracle-19c6-vagrant: 22 (guest) => 2222 (host) (adapter 1)
==> oracle-19c6-vagrant: Running 'pre-boot' VM customizations...
==> oracle-19c6-vagrant: Booting VM...
==> oracle-19c6-vagrant: Waiting for machine to boot. This may take a few minutes...
    oracle-19c6-vagrant: SSH address: 127.0.0.1:2222
    oracle-19c6-vagrant: SSH username: vagrant
    oracle-19c6-vagrant: SSH auth method: private key
==> oracle-19c6-vagrant: Machine booted and ready!
==> oracle-19c6-vagrant: Configuring proxy environment variables...
==> oracle-19c6-vagrant: Configuring proxy for Yum...
[oracle-19c6-vagrant] GuestAdditions 7.0.6 running --- OK.
==> oracle-19c6-vagrant: Checking for guest additions in VM...
==> oracle-19c6-vagrant: Setting hostname...
==> oracle-19c6-vagrant: Mounting shared folders...
    oracle-19c6-vagrant: /vagrant => /Users/cbl/Vagrant/oracle/vagrant-boxes/OracleDatabase/19.x
==> oracle-19c6-vagrant: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> oracle-19c6-vagrant: flag to force provisioning. Provisioners marked to run always will still run.
$ 

So far everything looks good. The program VBoxHeadless does not take much CPU resources on my Mac. I can use spindump to see what my process is doing. It samples the program VBoxHeadless for 10 seconds:

$ sudo spindump -file VirtualBoxVM-spindump-idle.txt -onlytarget VBoxHeadless
$ grep "cpu time" VirtualBoxVM-spindump-idle.txt
...
  Thread 0x32d19    Thread name "EMT-0"    1001 samples (1-1001)    priority 31 (base 31)    cpu time 0.188s (398.4M cycles, 107.5M instructions, 3.71c/i)
  Thread 0x32d1a    Thread name "EMT-1"    1001 samples (1-1001)    priority 31 (base 31)    cpu time 0.674s (1302.9M cycles, 312.2M instructions, 4.17c/i)
...

The threads “EMT-0” and “EMT-1” are my virtual CPUs in the VM.

After starting an Oracle DB in mv VM the VM is idle:

But on my Mac I do see between 30% and 40% utilization of a Core:

My spindump shows the following:

$ grep "cpu time" VirtualBoxVM-spindump-DB-idle-40percent.txt
...
  Thread 0x32d19    Thread name "EMT-0"    1001 samples (1-1001)    priority 31 (base 31)    cpu time 2.878s (6.1G cycles, 1727.6M instructions, 3.51c/i)
  Thread 0x32d1a    Thread name "EMT-1"    1001 samples (1-1001)    priority 31 (base 31)    cpu time 0.870s (1795.8M cycles, 538.1M instructions, 3.34c/i)
...

To see more information I sampled the process VBoxHeadless in Activity Monitor:

Unfortunately that does not help much. Running dtruss on the process VBoxHeadless produces many “invalid user access” errors:

$ ps -ef | grep VBoxHeadless | grep -v grep
272972148 34870  1838   0  9:52PM ??        33:19.36 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment oracle-19c6-vagrant --startvm 084f3493-f531-4492-b8dc-09e3bf232948 --vrde config
$ sudo dtruss -d -e -f -p 34870
Password:
dtrace: system integrity protection is on, some features will not be available
...
dtrace: error on enabled probe ID 1713 (ID 173: syscall::read:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 1712 (ID 961: syscall::write_nocancel:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 1712 (ID 961: syscall::write_nocancel:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 1712 (ID 961: syscall::write_nocancel:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 1744 (ID 353: syscall::select:return): invalid user access in action #5 at DIF offset 0
...

As the first line indicates: “system integrity protection is on” and hence I do not get the data I need. It would require a reboot of the Mac into recovery mode to change that and disable SIP for dtrace.

Unfortunately the VBox.log also does not have more info.

After producing load on my VM (the 2 Cores 100% utilized) and then the VM has become idle again I still see the process VBoxHeadless utilizing 2 Cores on the Host. I.e. my VM is idle:

But my host CPU still utilizes 2 Cores for almost 1 minute:

After googling of such a behavior I found some remarks about enabling the High Precision Event Timer, which improved the situation for other users. So I stopped my VM and changed it to use the High Precision Event Timer:

$ VBoxManage showvminfo "oracle-19c6-vagrant" | grep -i hpet
HPET:                        disabled
$ VBoxManage modifyvm "oracle-19c6-vagrant" --hpet on
$ VBoxManage showvminfo "oracle-19c6-vagrant" | grep -i hpet
HPET:                        enabled

After booting the VM and running my load test again, the Host CPUs became idle (i.e. back to 30-40% for VBoxHeadless) after around 10 secs. So the process recovers much faster compared to the default setting:

The Virtualbox High Precision Event Timer setting is documented as follows:

 --hpet=on | off

    Enables or disables a High Precision Event Timer (HPET) that can replace a legacy system timer. This feature is disabled by default. Note HPET is supported on Windows versions starting with Vista. 

Enabling hpet for my Windows VM also had a very positive effect. The VM runs more stable and produces less load on the Host as well.

REMARK: With hpet=on I do see the Host process VBoxHeadless for an idle VM being around 10% more utilized compared to hpet=off. It does not matter in my case, but I could reduce this for an Oracle DB-server by e.g. following the steps here.

Summary

Running Virtualbox-VMs on a Mac-host may gain from enabling the High Precision Event Timer. The VMs recover faster from high load and run more stable. Unfortunately tracing issues of high CPU-load of VirtualBox-processes is difficult because of the enabled system integrity protection (SIP).