vmm: Disable WAITPKG

If UMWEAIT/TPAUSE instructions are executed within a guest, a physical
processor is put into an implementation-depedent optimized state. It may
lead to an overhead of waking it up when scheduling another guest on it.
See comment in code for more details.

Signed-off-by: Takahiro Itazuri <[email protected]>

Author: zulinx86

CHANGELOG.md

@@ -22,6 +22,11 @@ and this project adheres to
 
 ### Changed
 
+- [#5118](https://github.com/firecracker-microvm/firecracker/pull/5118): Cleared
+  WAITPKG CPUID bit in CPUID normalization since it may lead to an overhead to
+  guest scheduling in a serverless computing environment with CPU
+  oversubsription.
+
 ### Deprecated
 
 - [#4948](https://github.com/firecracker-microvm/firecracker/pull/4948):

docs/cpu_templates/cpuid-normalization.md

@@ -35,6 +35,7 @@ See also: [boot protocol settings](boot-protocol.md)
 | Disable frequency selection                                    |                0x6                 |    -    |   ECX    |   3   |
 | Set FDP_EXCPTN_ONLY bit                                        |                0x7                 |   0x0   |   EBX    |   6   |
 | Set "Deprecates FPU CS and FPU DS values" bit                  |                0x7                 |   0x0   |   EBX    |  13   |
+| Disable WAITPKG (UMONITOR / UMWAIT / TPAUSE)                   |                0x7                 |   0x0   |   ECX    |   5   |
 | Disable performance monitoring                                 |                0xa                 |    -    |   all    |  all  |
 | Fill v2 extended topology enumeration leaf                     |                0x1f                |   all   |   all    |  all  |
 | Update brand string to use a default format and real frequency | 0x80000002, 0x80000003, 0x80000004 |    -    |   all    |  all  |

src/vmm/src/cpu_config/x86_64/cpuid/intel/normalize.rs

@@ -194,6 +194,46 @@ impl super::IntelCpuid {
         set_bit(&mut leaf_7_0.result.ebx, 6, true);
         set_bit(&mut leaf_7_0.result.ebx, 13, true);
 
+        // CPUID.(EAX=07H,ECX=0):ECX[5] (Mnemonic: WAITPKG)
+        //
+        // WAITPKG indicates support of user wait instructions (TPAUSE, UMONITOR and UMWAIT).
+        // - UMONITOR arms address monitoring hardware that checks for store operations on the
+        //   specified address range.
+        // - UMWAIT instructs the processor to enter an implementation-dependent optimized state
+        //   (either a light-weight power/performance optimized state (C0.1 idle state) or an
+        //   improved power/performance optimized state (C0.2 idle state)) while monitoring the
+        //   address range specified in UMONITOR. The instruction wakes up when the time-stamp
+        //   counter reaches or exceeds the implicit EDX:EAX 64-bit input value.
+        // - TPSUSE instructs the processor to enter an implementation-dependent optimized state.
+        //   The instruction wakes up when the time-stamp counter reaches or exceeds the implict
+        //   EDX:EAX 64-bit input value.
+        //
+        // These instructions may be executed at any privilege level. Even when UMWAIT/TPAUSE are
+        // executed within a guest, the *physical* processor enters the requested optimized state.
+        //
+        // MONITOR/MWAIT instructions are the privileged variant of UMONITOR/UMWAIT and are
+        // unconditionally emulated as NOP by KVM.
+        // https://github.com/torvalds/linux/commit/87c00572ba05aa8c9db118da75c608f47eb10b9e
+        //
+        // When UMONITOR/UMWAIT/TPAUSE were initially introduced, KVM clears the WAITPKG CPUID bit
+        // in KVM_GET_SUPPORTED_CPUID by default, and KVM exposes them to guest only when VMM
+        // explicitly sets the bit via KVM_SET_CPUID2 API.
+        // https://github.com/torvalds/linux/commit/e69e72faa3a0709dd23df6a4ca060a15e99168a1
+        // However, since v5.8, if the processor supports "enable user wait and pause" in Intel VMX,
+        // KVM_GET_SUPPORTEDC_CPUID sets the bit to 1. So if the return value is passed to
+        // KVM_SET_CPUID2 API as it is, guests are able to execute them.
+        // https://github.com/torvalds/linux/commit/0abcc8f65cc23b65bc8d1614cc64b02b1641ed7c
+        //
+        // Similar to MONITOR/MWAIT, we disbale the guest WAITPKG in order to prevent guests from
+        // executing those instructions and putting a physical processor to an idle state which may
+        // lead to an overhead of waking it up when scheduling another guest on it. By clearing the
+        // WAITPKG bit in KVM_SET_CPUID2 API, KVM does not set the "enable user wait and pause" bit
+        // (bit 26) of the secondary processor-based VM-execution control, which makes guests get
+        // #UD when attempting to executing those instructions.
+        //
+        // Note that the WAITPKG bit is reserved on AMD.
+        set_bit(&mut leaf_7_0.result.ecx, 5, false);
+
         Ok(())
     }
 
@@ -419,6 +459,7 @@ mod tests {
             .unwrap();
         assert!((leaf_7_0.result.ebx & (1 << 6)) > 0);
         assert!((leaf_7_0.result.ebx & (1 << 13)) > 0);
+        assert_eq!((leaf_7_0.result.ecx & (1 << 5)), 0);
     }
 
     #[test]