Error with quickstart and suggestion on docker dependency

[alexellis]

There are two things this issue is about:

1. There is an error with the Quickstart that I cannot get past yet, when using Ubuntu
2. The dependency on Docker is confusing since Docker and containerd ship very different (incompatible) versions of containerd

I was chatting to @estesp about trying to get faasd (based upon containerd) to work with firecracker, however I was surprised to see that Docker is required to do a build of the containerd binary.

Quite often Docker and containerd ship with different binaries that aren't compatible, so is the idea that I should build on my workstation and copy over to a server which has a stand-alone version of the tools on it?

I'd much prefer being able to build then deploy to the same Linux host where I'll be testing out firecracker. This indirection is causing some friction and confusion for me.

Hope this is useful feedback on some level.

[alexellis]

This is the last command that worked in the quickstart.

root@firecracker:/var/lib/firecracker-containerd/snapshotter/devmapper# if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
>     sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
> fi

device-mapper: reload ioctl on fc-dev-thinpool  failed: No such device or address
Command failed.

System details (on Packet/Equinix Metal)

root@firecracker:~# uname -a
Linux firecracker 5.8.0-26-generic #27-Ubuntu SMP Wed Oct 21 22:29:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@firecracker:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=groovy
UBUNTU_CODENAME=groovy
root@firecracker:~# 

I could run the rest of the commands, but then the ctr run command failed as follows:

ecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.166272676Z]                                               attempt=87 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.266184968Z]                                               attempt=88 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.366237569Z]                                               attempt=89 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.466279831Z]                                               attempt=90 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.566329880Z]                                               attempt=91 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.666165227Z]                                               attempt=92 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.766108613Z]                                               attempt=93 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.866233051Z]                                               attempt=94 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:17:52.966206940Z]                                               attempt=95 error="temporary vsock dial failure: vsock ack message failure: EOF" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7

recracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:18:03.092693638Z] sending signal 9 to 23926                     jailer=noop runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
ERRO[2021-02-09T20:18:03.092831511Z] failed to create VM                           error="failed to dial the VM over vsock: context deadline exceeded" runtime=aws.firecracker vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:18:03.093004894Z] stopVMM(): sending sigterm to firecracker     runtime=aws.firecracker
ERRO[2021-02-09T20:18:03.094887472Z] shim CreateVM returned error                  error="rpc error: code = DeadlineExceeded desc = VM \"3d28cc6e-5911-4c65-9fc6-bd025452e7e7\" didn't start within 20s: failed to dial the VM over vsock: context deadline exceeded"
DEBU[2021-02-09T20:18:03.098511483Z] shim has been terminated                      error="signal: killed" vmID=3d28cc6e-5911-4c65-9fc6-bd025452e7e7
DEBU[2021-02-09T20:18:03.099836751Z] remove snapshot                               key=test
DEBU[2021-02-09T20:18:03.100551896Z] event published                               ns=default topic=/snapshot/remove type=containerd.events.SnapshotRemove
DEBU[2021-02-09T20:18:03.101131712Z] event published                               ns=default topic=/containers/delete type=container

Running the commands again gave:

https://gist.github.com/alexellis/e9d0671e4c9602fca598d7cc9f4f58c7

Would someone from the team be able to make a suggestion?

[alexellis]

Here are the equivalent setup instructions for Ubuntu for the initial setup of Docker and Go 1.15.

curl -4 -o go.tgz -SL https://golang.org/dl/go1.15.8.linux-amd64.tar.gz

mkdir /usr/local/go

tar -xvf go.tgz --strip-components=1 -C /usr/local/go

sudo apt update && sudo apt install -qy   make   git   curl   e2fsprogs   util-linux   bc   gnupg

export PATH=$PATH:/usr/local/go/bin

sudo usermod -aG docker $(whoami)
curl -sLS https://get.docker.com | sh

sudo apt install build-essential -qy

The rest should be identical from # Check out firecracker-containerd and build it. onwards.

This quick start for firecracker itself did work, despite the issues I had above with firecracker-containerd https://github.com/firecracker-microvm/firecracker/blob/master/docs/getting-started.md#running-firecracker

[kzys]

Sorry for the confusion. If you just want to build firecracker-containerd, you can do that without Docker. make (or make all) build all binaries on the host itself.

However the root filesystem creation and automated test related targets depend on Docker right now.

[alexellis]

@kzys I installed Docker, but as you can see there are vsock errors after following the instructions with Ubuntu.

I appreciate you responding, but this doesn't unblock me. Can you verify that this project is functioning on Ubuntu Linux?

[samuelkarp]

Hey @alexellis!

Quite often Docker and containerd ship with different binaries that aren't compatible, so is the idea that I should build on my workstation and copy over to a server which has a stand-alone version of the tools on it?

I'd much prefer being able to build then deploy to the same Linux host where I'll be testing out firecracker. This indirection is causing some friction and confusion for me.

The quickstart is designed to let you build and run on the same Linux host where you're testing Firecracker. We use separate binaries to avoid conflicts with Docker.

I looked in the gist you provided and found this line, which looks to be the relevant culprit:

[    0.599754] agent[716]: /usr/local/bin/agent: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/local/bin/agent)

It looks like the agent binary was not built with the STATIC_AGENT environment variable and it's instead trying to link against the glibc present on your Ubuntu host rather than the glibc present in the Debian microVM image (even though the Makefile sets that variable). We usually build the agent as statically-linked to avoid this problem. Did you build the agent binary before installing Docker? I wonder if our Makefile isn't cleaning up and rebuilding when it should be.

Can you try rebuilding the image and reinstalling it to the correct location (should be sudo cp ~/firecracker-containerd/tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img if you followed the quickstart)? make image might do it alone, but if it doesn't you'll want to make clean image (which will take longer since it'll rebuild everything).

You might also find the getting started guide to be helpful as it steps through the requirements in more detail.

It's late for me here, but I'm happy to help more in the morning (my time).

Sam

[alexellis]

Thanks for the pointers. I just followed the quick-start verbatim, with the modification for a newer Docker CE version and a newer Golang version.

When running this step, I get the following error:

~/firecracker-containerd# sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

fatal: Not a git repository: ../../.git/modules/runc
fatal: Not a git repository: ../../.git/modules/runc

[alexellis]

Running this block gave me an error again, just like when I did this with Ubuntu Linux. I have a feeling that the instructions are not complete or are missing a step.

root@firecracker-debian10:/var/lib/firecracker-containerd/snapshotter/devmapper# 
root@firecracker-debian10:/var/lib/firecracker-containerd/snapshotter/devmapper# if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
>     sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
> fi
device-mapper: reload ioctl on fc-dev-thinpool  failed: No such device or address
Command failed.
root@firecracker-debian10:/var/lib/firecracker-containerd/snapshotter/devmapper# 
root@firecracker-debian10:/var/lib/firecracker-containerd/snapshotter/devmapper# 

[alexellis]

After re-running several of the commands I got a VM to launch through ctr run!

This message was printed out by containerd, but I don't know if it's an error

DEBU[2021-02-19T10:07:03.428545928Z] [    1.471698] systemd[756]: serial-getty@ttyS0.service: Executable /sbin/agetty missing, skipping: No such file or directory  jailer=noop runtime=aws.firecracker vmID=f7ac9a76-637a-4fb0-ac9c-8d933fabc4ab vmm_stream=stdout

[kzys]

That's great. I think the getty one is fine. Let me take a look.

Regarding your previous comments;

• Getting fatal: Not a git repository: ../../.git/modules/runc from apt-get is really odd. apt-get must not call Git.
• sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}" would fail if you don't have the thin-pool but then sudo dmsetup create "${POOL}" --table "${THINP_TABLE}" will create the thin-pool. Can you check dmsetup ls on your host?

[kzys]

FYI, the getty error is fixed in master. Thanks for reporting!

[alexellis]

🙏 thank you

I'm having issues with CNI, I had hoped to find a quickstart that I could drop in and use with the faasd code. I would appreciate some pointers or examples.

I tried to create a minimal example that someone on the team can run to see the issues with using CNI and the PID netns (the PID isn't found because it appears to be from the VM, and not readable from where containerd is running on the host)

Thanks to @samuelkarp for giving some pointers on Slack. I'm at a bit of a dead-end now though.

[alexellis]

@kzys

When you said:

Sorry for the confusion. If you just want to build firecracker-containerd, you can do that without Docker. make (or make all) build all binaries on the host itself.
However the root filesystem creation and automated test related targets depend on Docker right now.

Could you provide those (tested) instructions in the repository? The quick start mandates the use of Docker https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

As a side-note, can firecracker work with a newer version of Go than 1.13 (which is used in the quickstart)?

[sgaist]

Hitting the same issue, I found what the culprit might be.

The getting started guide gives commands that for the majority of the tasks executes on the host machine. If passed the -in-docker suffix (for example make all-in-docker), they will be executed through docker using the image defined by FIRECRACKER_CONTAINERD_BUILDER_IMAGE.

There lies the catch, for the build of agent, STATIC_AGENT is set to on when doing the build through docker but otherwise it's empty and thus triggers a dynamic build when doing that on the host.