[Bug] Could not create the network device: IoVecBuffer(Mut) in 1.10.0

[gounselor]

Describe the bug

I have a working vmconfig.json which works with firecracker-v1.9.1-aarch64 and guest kernel 6.11.* and multiple root filesystems.

If I use the same vmconfig.json with firecracker-v1.10.0-aarch64 it fails with:

2024-11-08T19:44:55.998563652 [anonymous-instance:main] Running Firecracker v1.10.0
2024-11-08T19:44:55.999048225 [anonymous-instance:main] RunWithoutApiError error: Failed to build MicroVM from Json: Configuration for VMM from one single json failed: Network device error: Could not create the network device: IoVecBuffer(Mut) error: Error with underlying `IovDeque`: Error calling mmap: Invalid argument (os error 22)
Error: RunWithoutApiError(BuildMicroVMFromJson(ParseFromJson(NetDevice(CreateNetworkDevice(IoVecError(IovDeque(Mmap(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))))))))
2024-11-08T19:44:55.999238447 [anonymous-instance:main] Firecracker exiting with error. exit_code=1

To Reproduce

Start firecracker 1.10.0 with this command line:

sudo release-v1.10.0-aarch64/firecracker-v1.10.0-aarch64 --no-api --config-file vmconfig.json

Config file vmconfig.json used:

{
  "boot-source": {
    "kernel_image_path": "kernel",
    "boot_args": "ro console=ttyS0 noapic reboot=k panic=1 random.trust_cpu=on ip=192.168.179.2::192.168.179.1:255.255.255.0::eth0:off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/dev/nvme/alpine-minimal",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "network-interfaces": [
      {
          "iface_id": "eth0",
          "guest_mac": "02:FC:00:00:00:01",
          "host_dev_name": "fc-tap0"
      }
  ],
  "machine-config": {
    "vcpu_count": 1,
    "mem_size_mib": 256
  }
}

Expected behaviour

I expect the vm to start like v1.9.1 does.

Environment

• Firecracker version: firecracker-v1.10.0-aarch64
• Host and guest kernel versions:
  • Host: 6.6.51+rpt-rpi-2712
  • Guest: 6.11.x, Image-6.11.7: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
• Rootfs used: alpine, void, ubuntu2[24] etc. does not seem to matter as the vm creation fails
• Architecture: arm64/aarch64
• Any other relevant software versions: don't think so, works with firecracker 1.9.1

Additional context

Happy to help debugging.

Checks

• Have you searched the Firecracker Issues database for similar problems?
• Have you read the existing relevant Firecracker documentation?
• Are you certain the bug being reported is a Firecracker issue?

[roypat]

Hi @gounselor,

Thanks a lot for reaching out about this. Since you mention running on a raspberry pi, may I ask what page size your host is configured with? We suspect the issue you're seeing is due to an assumption on host page size being 4K based into the iov_deque.rs code. If your host is configured with 16K (or another non-4K page size), could you try if it still reproduces with 4K pages?

[gounselor]

Hi @roypat,

yes, good finding. Indeed the page size is 16k according to the supplied kernel config and dmesg.
kernel is 6.6.51+rpt-rpi-2712 which is configured with
CONFIG_ARM64_16K_PAGES=y

This is also reported by:
root@pi5:~# getconf PAGESIZE
16384

The kernel is supplied by Raspberry Pi OS (former raspbian?) which is based on Debian GNU/Linux 12 (bookworm).

I can try to get or compile a kernel with 4k pages, it's not possible to change via cmdline etc.

[ShadowCurse]

@gounselor You may be able to switch to 4k kernel by setting kernel=kernel8.img in the config.txt

[gounselor]

@ShadowCurse and @roypat

Not so easy on a headless pi5 but managed it. Actually moving kernel_2712.img "away" boots kernel8.img.

With this one (kernel8.img) and
`
ms@pi5:/mnt/firecracker $ getconf PAGESIZE

4096
`

it works:

2024-11-13T18:05:30.871718746 [anonymous-instance:main] Running Firecracker v1.10.0 2024-11-13T18:05:31.880249104 [anonymous-instance:main] Artificially kick devices. 2024-11-13T18:05:31.880349391 [anonymous-instance:fc_vcpu 0] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding 2024-11-13T18:05:31.880387726 [anonymous-instance:main] Successfully started microvm that was configured from one single json [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1] [ 0.000000] Linux version 6.11.7 (ms@pi5) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Fri Nov 8 18:35:52 CET 2024 ...

[hackeroneulevel]

The issue you're experiencing with Firecracker v1.10.0 seems to be related to a compatibility change or bug in the new version, particularly in how it handles network device configurations. The error message suggests a problem with the memory mapping (mmap) for the network device, which wasn't present in v1.9.1. Here's how to approach solving it:

---

Steps to Investigate and Resolve

1. Validate Changes in Firecracker v1.10.0:

   • Check the [Firecracker v1.10.0 release notes](https://github.com/firecracker-microvm/firecracker/releases) for any breaking changes or updates related to network devices, memory mapping, or IovDeque.
   • Look for deprecations or additional parameters introduced in v1.10.0.

2. Verify the Network Device Configuration:

   • Ensure the fc-tap0 network interface exists on the host.

     ip link show fc-tap0

     If it doesn't exist, recreate it:

     sudo ip tuntap add mode tap fc-tap0
     sudo ip link set fc-tap0 up

   • Check if the guest_mac address is valid. While the provided MAC address (02:FC:00:00:00:01) seems fine, ensure there are no conflicts.

3. Inspect mmap Parameters:

   • The Invalid argument error could be caused by how memory regions are allocated for network devices. Check your host kernel configuration to ensure it supports all required features for Firecracker v1.10.0.
   • Specifically, validate support for TAP devices and networking-related kernel modules (CONFIG_TUN, etc.).

4. Test with a Minimal vmconfig.json:

   • Simplify the vmconfig.json to isolate the issue. Start by removing optional components, such as the network interface, and attempt to boot:

     {
       "boot-source": {
         "kernel_image_path": "kernel",
         "boot_args": "ro console=ttyS0 noapic reboot=k panic=1 random.trust_cpu=on"
       },
       "drives": [
         {
           "drive_id": "rootfs",
           "path_on_host": "/dev/nvme/alpine-minimal",
           "is_root_device": true,
           "is_read_only": false
         }
       ],
       "machine-config": {
         "vcpu_count": 1,
         "mem_size_mib": 256
       }
     }

   • If this succeeds, incrementally add components back (e.g., the network interface) to identify the problematic section.

5. Enable Debug Logging:

   • Run Firecracker with debug logging enabled to get more detailed error messages:

     sudo release-v1.10.0-aarch64/firecracker-v1.10.0-aarch64 --no-api --config-file vmconfig.json --log-path /tmp/fc.log --level Debug

   • Review the log for additional details.

6. Compare Host Kernel Configuration:

   • Since the issue occurs on Firecracker v1.10.0 but not v1.9.1, compare the host kernel requirements for both versions. Confirm that the host kernel supports features required by Firecracker v1.10.0 (check the [documentation](https://github.com/firecracker-microvm/firecracker)).

7. Test with Alternative Kernel and Root Filesystem:

   • While you mentioned the issue isn't rootfs-specific, try using a minimal, well-tested guest kernel (e.g., Alpine's default kernel) and a smaller rootfs to rule out any kernel-specific issues.

---

If None of the Above Work

1. File an Issue:

   • If the issue persists, file a detailed bug report with the Firecracker team. Include:
     • Logs from debug mode.
     • A minimal reproduction case (simplified vmconfig.json and commands).
     • Any relevant host/kernel configuration details.

2. Temporary Workaround:

   • Use Firecracker v1.9.1 for your setup until the issue is resolved in v1.10.0. If you need v1.10.0 features, consider contributing a patch or engaging with the Firecracker community for assistance.

---

Next Steps

Run through these steps and identify where the issue lies. If debugging logs or specific configurations provide more details, feel free to share them for further analysis!

[ShadowCurse]

Hi @gounselor, the PR #4916 with a fix was merged. This should resolve the issue with 16K host kernel.
Closing the issue.