About exception handling in vcpu, IRQ injection and emulated device APIs.

[hky1999]

Problem1: How to set exception vector base (VBAR_EL2 in arm_vcpu) before entering vcpu.run()

The problem comes from our design that we want to converge all exception handling into xxx_vcpu implementation.

For x86_64, this is a reasonable and nature implementation, because each unexpected exception (not configured in VMCS) will cause a VM-Exit and jump to the VM-Exit entry defined in the RIP of Host-State Fields in VMCS.

However, for aarch64 and riscv64, this is a bit awkward since we can not determined our own "VM-Exit" entry, all VM-Exits will jump to the exception vector base (VBAR_EL2 in arm_vcpu).

In arm_vcpu, after hardware_enable() is called, we set the VBAR_EL2 as exception_vector_base_vcpu, which will jump to vmexit_trampoline to pop the host contexts from host sp, and resume workflow to Aarch64VCpu.run() eventually.

#[naked]
#[no_mangle]
unsafe extern "C" fn vmexit_trampoline() {
    // Note: Currently `sp` points to `&Aarch64VCpu.ctx`, which is just the base address of Aarch64VCpu struct.
    core::arch::asm!(
        "add sp, sp, 34 * 8",       // Skip the exception frame.
        "mov x9, sp", // Currently `sp` points to `&Aarch64VCpu.host_stack_top`, see `run_guest()` in vcpu.rs.
        "ldr x10, [x9]", // Get `host_stack_top` value from `&Aarch64VCpu.host_stack_top`.
        "mov sp, x10", // Set `sp` as the host stack top.
        restore_regs_from_stack!(), // Restore host function context frame.
        "ret", // Control flow is handed back to Aarch64VCpu.run(), simulating the normal return of the `run_guest` function.
        options(noreturn),
    )
}

This leads to another awkward phenomena: the VM-Exit handling routine only works when at least one vcpu is running, (that is, when we try to restore host context from host sp, someone should have store the host context into host sp before, which is done by run_guest called by Aarch64VCpu.run()). When there is no running vCPU on this physical core, a exception will lead to a undefined behavior.

To a certain extent, this is somehow reasonable, because a exception is meaningless when there is no active vCPU on this core, we can rudely disable all exceptions when no active vCPU is running, and only enable exception after run_guest is called. BUT, this is a really bad design.

I've come up with some idea to fix this, that is, after we catch exceptions in a exception handling routine defined in exception_vector_base_vcpu, and before the control flow jumps to the vmexit_trampoline, we need to use a global flag to check if there is a running vcpu when this exception happens.

• If so, we can jumps to the vmexit_trampoline safely, because the host context has been stored in host sp before this active vcpu is running.
• If not, no one has put anything to the host stack, we can use a callback here, try to dispatch this exception back to vmm-app, and then dispatch to arceos finally. Generally this is triggered by hypervisor reserved irqs (like timer irq for scheduling, or IPIs).

Problem2: How to get irq number in arch specfic vcpu implementation

In x86_64, irq number can be obtained from VMEXIT_INTERRUPTION_INFO field defined in VMCS, so we can return a AxVCpuExitReason of ExternalInterrupt with the irq no gracefully.

#[non_exhaustive]
#[derive(Debug)]
pub enum AxVCpuExitReason {
    /// An external interrupt happened.
    ///
    /// Note that fields may be added in the future, use `..` to handle them.
    ExternalInterrupt {
        /// The interrupt vector.
        vector: u8,
    },
}

However, in aach64 and riscv64, we can not get any irq info from any system registers or arch related structures directly.

• in aarch64, the irq number has to be read from gicc GICC.IAR.get()
• in riscv64, the irq number is defined by scause reg:
  • CAUSE_INTR_SOFT => IPI
  • CAUSE_INTR_TIMER => TIMER
  • CAUSE_INTR_EXTERNAL => get from PLIC

There is two solutions for this:

1. set irq_no as None in AxVCpuExitReason, until the control return to vmm-app, where wen can get gic/plic instance from arceos
2. reintroduce AxHal trait for vcpu? add fetch_irq() interface.

Problem3: How to inject a irq to VM's target vCPU, (generally called from emulated devices)

The irq injectection to a guest VM is quite complicated (this is also the reason why it has been delayed for so long).

First, let's see who needs to inject interrupts into the guest VM:

1. guest timer interrupt, as described in https://github.com/orgs/arceos-hypervisor/discussions/36#discussioncomment-11002988
2. passthrough device's irq, this type of irq may be intercepted by hypervisor in aarch64's gicv2 architecture (in x86_64, I believe it can be transprant to the hypervisor if we configure some irq as passthrough in VMCS, I kind of forget it, @aarkegz we can confirm it together)
3. notification triggered by emulated devices to inform that a emulated device req is done.
4. vIPI triggered by guest VM (I think 4. can be considered together with 3.)

For 3. notification triggered by emulated devices, there are several types of interrupts that may need to be handled: normal irq, msi and msix.

• for normal irq
  • in x86_64, a irq injection can be handled by x86_vcpu itself through VMCS::VMENTRY_INTERRUPTION_INFO_FIELD
  • in arm_vcpu or riscv_vcpu, a irq injection should be handled by emulated gic or plic.
• for msi and msix, there are revelant to the PCI device

In summary, I think it is more reasonable to encapsulate a vm.inject_irq(vcpu_id: usize) method for axvm.
As the same time, we need to inject this method into the specific implementation of each emulated device (including the emulated interrupt controller).