Missing documentation for resolving update failure states

[joshqou]

Had to deal with an unexpected shutdown on a machine that was running podman update. Unsure exactly where it was in the update process but it was far enough to make the images of some containers unbootable due to missing files.

Found #24106 after a while of searching, which lead to me running podman system check -r, which fixed the effected images. File mangling is unlikely since the root filesystem is btrfs which provides CoW.

This solution works fine for root containers, but not for user containers, which have their own independent cache. For the user containers, I had to just rm -rf .local/share/containers, which is an acceptable solution for me as I use fuse mounts, but won't be if I had been using container volumes.

Given how podman auto-update is frequently recommended, it seems insane to me that this kind of failure condition isn't well documented.

Filesystem errors for searchability:

Error: faccessat /home/gitea/.local/share/containers/storage/overlay/b94fc4481c4bfeff247eafcf05afd8d7af520750591501e4f9a6fc09d74b7562: no such file or directory
Error: faccessat /var/lib/containers/storage/overlay/14949f4fd718fa37ffc23280c9c2b3164b48929a6323a9fc5b83fe7a292cf594: no such file or directory

[baude]

This solution works fine for root containers, but not for user containers, which have their own independent cache. For the user containers, I had to just rm -rf .local/share/containers, which is an acceptable solution for me as I use fuse mounts, but won't be if I had been using container volumes.

@joshqou I don't follow your logic here or perhaps I am misunderstanding what you mean by independent cache. Care to elaborate?

[joshqou]

@baude Running podman system check -r fixed the podman containers I was running rootfully, however it did not fix the containers I had running under user accounts. By independent cache, I'm talking about ~/.local/share/containers/ and the data stored in storage/overlay/. It's the error scoped under /home/gitea.