podman pull does not work

[Karotte128]

Issue Description

podman pull results in Error: initializing source docker://libary/ubuntu:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.178.1:53: dial udp 192.168.178.1:53: socket: permission denied

Steps to reproduce the issue

Steps to reproduce the issue

1. podman pull ubuntu:latest

Describe the results you received

Error: initializing source docker://libary/ubuntu:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.178.1:53: dial udp 192.168.178.1:53: socket: permission denied

Describe the results you expected

podman should pull the container image

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 98.95
    systemPercent: 0.52
    userPercent: 0.53
  cpus: 2
  distribution:
    codename: mantic
    distribution: ubuntu
    version: "23.10"
  eventLogger: journald
  hostname: CT109
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.8.4-2-pve
  linkmode: dynamic
  logDriver: journald
  memFree: 1639899136
  memTotal: 2147483648
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.5-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 31m 5.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 8350298112
  graphRootUsed: 1269452800
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

podman is running in proxmox lxc container

Additional information

this worked until now, after a reboot it was broken

[Luap99]

socket: permission denied

This is not a podman problem, your LXC or whatever likely blocks networking.

[Aster-the-Med-Stu]

Could confirm it's network issue. Nowadays Proxmox has apparmor, and the default config blocks podman network!

Taken from my dmesg:

[25690.731206] audit: type=1400 audit(1728949935.450:1904): apparmor="DENIED" operation="create" class="net" namespace="root//lxc-1002_<-var-lib-lxc>" profile="podman" pid=132774 comm="pasta.avx2" family="inet" sock_type="stream" protocol=6 requested="create" denied="create"

You might wish to run systemctl stop apparmor on your PVE host before editing apparmor audit.

[LoganTann]

Thanks a lot for hinting about apparmor. I spent a whole hour trying to debug why I couldn't pull images...

[sbrivio-rh]

But wait, pasta ships its own AppArmor profile (https://passt.top/passt/tree/contrib/apparmor), and now it should finally run with it as the AppArmor stub for Podman was dropped (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100135). Is this Debian? Ubuntu?

Don't just whack AppArmor please, report issues (with excerpts from audit.log) if you find them!

[LoganTann]

In my case, Podman was installed in an Ubuntu LXC container, which itself was running on Proxmox VE. The AppArmor on the Proxmox host was blocking my container from accessing the network configuration.

In any case, trying to install Podman inside an LXC container is a very bad idea, I should just have used a true VM; I never managed to get it working (which makes perfect sense), so I gave up.

[sbrivio-rh]

So I suppose it's Debian, but perhaps without https://salsa.debian.org/apparmor-team/apparmor/-/commit/afd2a9ade8bd32f050b8ebe3d4221730db75c2ab? If that's the case, then it's all fine. I'm just trying to make sure that it's not an issue we still have either in the upstream AppArmor profile for pasta or in the Debian package (I maintain both).

[almereyda]

Also seeing this here on Ubuntu 25.10 with an Incus Ubuntu 24.04 guest.

Is there a preferred way forward? It is certainly the host's AppArmor that defeats Podman in the guest.

$ incus storage set default volume.zfs.delegate true
$ incus launch images:ubuntu/noble u1 -c security.nesting=true -c security.syscalls.intercept.mknod=true -c security.syscalls.intercept.setxattr=true
$ incus exec u1 -- podman run --rm hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Error: initializing source docker://hello-world:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: permission denied

$ sudo dmesg | tail -n 1 
[207277.805695] audit: type=1400 audit(1761801363.391:1457): apparmor="DENIED" operation="create" class="net" info="failed af match" error=-13 namespace="root//incus-u1_<var-lib-incus>" profile="podman" pid=779629 comm="podman" family="inet" sock_type="dgram" protocol=0 requested="create" denied="create"

The same happens with using --network slirp4netns.

• Should this be eventually be downstreamed to the Podman packagers of Ubuntu and/or Debian?

[almereyda]

I'm only an Ubuntu user, not a developer of it. I'm not acquainted with their issue trackers. Probably somewhere on Launchpad?

Also I wouldn't know how to check if they pulled in this specific patch into their release. With instructions how to do so, I could.

Sould I try deleting /etc/apparmor.d/podman, rebooting and seeing, if it works then?

Canonical have never been a strong supporter of systems and services originating from RedHat. We're already lucky, that we get fairly recent (Podman) builds meanwhile, which wasn't always the case.

Further I have no interaction points with the Debian ecosystem whatsoever.

[sbrivio-rh]

I'm only an Ubuntu user, not a developer of it. I'm not acquainted with their issue trackers. Probably somewhere on Launchpad?

Not a search engine developer either, I reckon. What about a user? :)

Anyway, yeah, https://bugs.launchpad.net/ubuntu/+source/podman.

Also I wouldn't know how to check if they pulled in this specific patch into their release. With instructions how to do so, I could.

Just look at the content of that commit. Is /etc/apparmor.d/podman present? If yes, it's that former Debian bug that didn't get fixed on Ubuntu. If not, we need to look further.

Sould I try deleting /etc/apparmor.d/podman, rebooting and seeing, if it works then?

Maybe move it away (outside /etc/apparmor.d/) instead of deleting it but yes, that sounds like a plan, thanks.

Canonical have never been a strong supporter of systems and services originating from RedHat. We're already lucky, that we get fairly recent (Podman) builds meanwhile, which wasn't always the case.

The package (and its dependencies) is very well maintained on Debian (https://tracker.debian.org/pkg/podman), and I myself as a Red Hat employee maintain the Debian package providing the default rootless networking back-end (https://tracker.debian.org/pkg/passt), so, yeah, magic of the ecosystem.

[almereyda]

Thanks for the cross-pollination in the ecosystem. A few years ago we had to resort to some PPA for recent builds, which then ceased to publish releases and ultimately changed at both upstreams.

Nowadays it's even part of their commercial offering. As seen during update, eventually by the advantage programme:

Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  buildah podman

The file is present, can be moved aside, the next reboot is due and I will report back.

In their tracker this is https://bugs.launchpad.net/ubuntu/+source/podman/+bug/2118824

From there:

a workaround: I removed the apparmor package inside the container(s):

After a restart of the containers It's now working

Which I can confirm. Running mv /etc/apparmor.d/podman /root/ and restarting the container allowed to run Podman inside again. No reboot of the incus host needed.

[almereyda]

Now Incus doesn't start anymore, so I cannot validate the reproducer for the moment.

• /var/lib/incus dataset of an encrypted, delegated guest mounted on top of host mount lxc/incus#2605

Podman inside rootless Podman executes fine:

$ podman run --rm --privileged quay.io/podman/stable podman run --rm hello | head -n 1
Resolved "hello" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob sha256:81df7ff16254ed9756e27c8de9ceb02a9568228fccadbf080f41cc5eb5118a44
Copying config sha256:5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0
Writing manifest to image destination
!... Hello Podman World ...!

Slightly-OT: If there's a way around the --privileged flag here, although I don't fully understand its implications in rootless mode, I'm interested in knowing alternatives. What does it do in this context?

https://docs.podman.io/en/latest/markdown/podman-run.1.html#privileged says it's probably fine:

Containers running in a user namespace (e.g., rootless containers) cannot have more privileges than the user that launched them.

The two paragraphs of the "Note:" on https://www.redhat.com/en/blog/privileged-flag-container-engines are more verbose about the implications.

[almereyda]

Since the issue with the outer container host is resolved no removal of /etc/apparmor.d/podman is needed nor possible anymore:

$ incus launch images:ubuntu/noble u2 -c security.nesting=true -c security.syscalls.intercept.mknod=true -c security.syscalls.intercept.setxattr=true
Launching u2

$ incus exec u2 -- sh -c 'apt install -y podman > /dev/null 2>&1; rm /etc/apparmor.d/podman'
rm: cannot remove '/etc/apparmor.d/podman': No such file or directory

$ incus exec u2 -- sh -c 'podman run --rm quay.io/podman/hello 2>/dev/null | head -n 1'
!... Hello Podman World ...!

Can others confirm that this was resolved during the day and podman pull now works as expected per the OP, since the patch obviously arrived in the image builds?

[Karotte128]

removing the file /etc/apparmor.d/podman finally solved the issue for me.

[sbrivio-rh]

Meanwhile, we also had a report at https://bugs.passt.top/show_bug.cgi?id=169.

I think it would help if you recorded yourself as affected user of https://bugs.launchpad.net/ubuntu/+source/podman/+bug/2118824 (assuming it's the same issue, I didn't check).