`limactl start template://docker` no longer boots (virtio-serial seems intermittently breaking virtio-net-pci)

[AkihiroSuda]

• host: macOS 13.5 (Intel)
• version: master (08bece2)
• QEMU: 8.0.3 (Homebrew)
• /var/log/cloud-init-output.log in the guest:

...
LIMA| Executing /mnt/lima-cidata/boot/30-install-packages.sh
+ INSTALL_IPTABLES=0
+ [  = 1 ]
+ [  = 1 ]
+ [ 58690 -ne 0 ]
+ INSTALL_IPTABLES=1
+ [ -d /mnt/lima-cidata/provision.dependency ]
+ [  = 1 ]
+ command -v apt-get
+ hexdump -C -n 4 /usr/bin/apt-get
+ grep -qF ELF
+ pkgs=
+ [ reverse-sshfs = reverse-sshfs ]
+ [ 2 -gt 0 ]
+ command -v sshfs
+ pkgs= sshfs
+ [ 1 = 1 ]
+ [ ! -e /usr/sbin/iptables ]
+ [  = 1 ]
+ [ -n  sshfs ]
+ DEBIAN_FRONTEND=noninteractive
+ export DEBIAN_FRONTEND
+ apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [634 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [149 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.0 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [656 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [104 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [764 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [137 kB]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.3 kB]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB]
Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7060 B]
Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [855 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [209 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [668 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [106 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [528 B]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [962 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [207 kB]
Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.4 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9768 B]
Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [476 B]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.9 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.2 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [22.2 kB]
Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [15.4 kB]
Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [580 B]
Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 26.4 MB in 8s (3454 kB/s)
Reading package lists...
+ apt-get install -y --no-upgrade --no-install-recommends -q sshfs
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  sshfs
0 upgraded, 1 newly installed, 0 to remove and 86 not upgraded.
Need to get 45.1 kB of archives.
After this operation, 126 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 sshfs amd64 3.7.1+repack-2 [45.1 kB]
Fetched 45.1 kB in 2s (22.2 kB/s)
Selecting previously unselected package sshfs.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64219 files and directories currently installed.)
Preparing to unpack .../sshfs_3.7.1+repack-2_amd64.deb ...
Unpacking sshfs (3.7.1+repack-2) ...
Setting up sshfs (3.7.1+repack-2) ...
Processing triggers for man-db (2.10.2-1) ...
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-72-generic
NEEDRESTART-KEXP: 5.15.0-72-generic
NEEDRESTART-KSTA: 1
LIMA| Executing /mnt/lima-cidata/boot/35-setup-packages.sh
+ SETUP_DNS=0
+ [ -n 58690 ]
+ [ 58690 -ne 0 ]
+ SETUP_DNS=1
+ [ -n 63183 ]
+ [ 63183 -ne 0 ]
+ SETUP_DNS=1
+ [ 1 = 1 ]
+ /mnt/lima-cidata/boot/09-host-dns-setup.sh
+ readonly chain=LIMADNS
+ command -v iptables
+ chain_exists
+ iptables --table nat -n --list LIMADNS
+ iptables --table nat --flush LIMADNS
+ iptables --table nat --append LIMADNS --destination 192.168.5.3 --protocol udp --dport 53 --jump DNAT --to-destination 192.168.5.2:58690
+ iptables --table nat --append LIMADNS --destination 192.168.5.3 --protocol tcp --dport 53 --jump DNAT --to-destination 192.168.5.2:63183
+ [ reverse-sshfs = reverse-sshfs ]
+ update_fuse_conf
+ [ 2 -gt 0 ]
+ fuse_conf=/etc/fuse.conf
+ [ -e /etc/fuse3.conf ]
+ grep -q ^user_allow_other /etc/fuse.conf
+ echo user_allow_other
LIMA| Executing /mnt/lima-cidata/boot/40-install-containerd.sh
+ '[' '' '!=' 1 ']'
+ '[' '' '!=' 1 ']'
+ exit 0
LIMA| Executing /mnt/lima-cidata/provision.system/00000000
LIMA| Executing /mnt/lima-cidata/provision.system/00000001
+ command -v docker
+ export DEBIAN_FRONTEND=noninteractive
+ DEBIAN_FRONTEND=noninteractive
+ curl -fsSL https://get.docker.com
+ sh
# Executing docker install script, commit: c2de0811708b6d9015ed1a2c80f02c9b70c8ce7b
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
+ sh -c chmod a+r /etc/apt/keyrings/docker.gpg
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease  Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.83). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.82). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.81). - connect (101: Network is unreachable)
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease  Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.91.38). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.91.39). - connect (101: Network is unreachable)
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/jammy/InRelease  Could not connect to download.docker.com:443 (18.172.31.124). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.28). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.47). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.22). - connect (101: Network is unreachable)
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
E: Package 'docker-ce' has no installation candidate
E: Unable to locate package docker-ce-cli
E: Unable to locate package containerd.io
E: Couldn't find any package by glob 'containerd.io'
E: Couldn't find any package by regex 'containerd.io'
E: Unable to locate package docker-compose-plugin
E: Unable to locate package docker-ce-rootless-extras
E: Unable to locate package docker-buildx-plugin
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000001
LIMA| Executing /mnt/lima-cidata/provision.user/00000002 (as user suda)
+ systemctl --user start dbus
+ dockerd-rootless-setuptool.sh install
/home/suda.linux/.lima-user-script: line 4: dockerd-rootless-setuptool.sh: command not found
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.user/00000002 (as user suda)
LIMA| Exiting with code 1
2023-07-28 13:55:40,403 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
2023-07-28 13:55:40,403 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
Cloud-init v. 23.1.2-0ubuntu0~22.04.1 finished at Fri, 28 Jul 2023 13:55:40 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 70.48 seconds

[AkihiroSuda]

The issue seems to happen with Lima v0.16, so this might not be a regression on Lima side.
(EDIT: my testing was wrong. Not reproducible with Lima v0.16)

Can anybody repro this?

[AkihiroSuda]

The issue doesn't seem to happen with --vm-type vz

[AkihiroSuda]

template://k3s boots but every pods fails with ImagePullBackOff: failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-metrics-server/manifests/v0.6.3": dial tcp: lookup registry-1.docker.io: Try again

[AkihiroSuda]

Seems a regression in

• qemu, vz: connect virtio serial to serialv.log #1674

[AkihiroSuda]

This seems to work, but not sure why

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index a120b6e..813a93d 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -830,7 +830,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        }
        const serialvChardev = "char-serial-virtio"
        args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server=on,wait=off,logfile=%s", serialvChardev, serialvSock, serialvLog))
-       args = append(args, "-device", "virtio-serial-pci,id=virtio-serial0")
+       args = append(args, "-device", "virtio-serial-pci,id=virtio-serial0,max_ports=2")
        args = append(args, "-device", fmt.Sprintf("virtconsole,chardev=%s,id=console0", serialvChardev))
 
        // We also want to enable vsock here, but QEMU does not support vsock for macOS hosts

Possibly relevant:

• https://bugzilla.redhat.com/show_bug.cgi?id=1370351 [virtio-win][vioser] serial port can not transfer data when nr>=31
• https://bugzilla.redhat.com/show_bug.cgi?id=1432297 virtio-serial port can not transfer data when boot >=33 ports for 1 pci