support User= in systemd for running rootless services

[Gchbg]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

I want to have a systemd system service that runs a rootless container under an isolated user, but systemd rejects the sd_notify call and terminates the service.

Got notification message from PID 15150, but reception only permitted for main PID 14978

A similar problem was menitoned but not resolved in #5572, which seems to have been closed without a resolution.

Happy to help tracking this down.

Steps to reproduce the issue:

1. Start with a Debian testing system. Create a system user with an empty home dir, and enable lingering:

groupadd -g 200 nginx
useradd -r -s /usr/sbin/nologin -l -b /var/lib -M -g nginx -u 200 nginx
usermod -v 165536-231071 -w 165536-231071 nginx
mkdir -m 770 /var/lib/nginx
nginx:nginx /var/lib/nginx
loginctl enable-linger nginx

2. Use this unit file, adapted from podman generate systemd --new:

❯ cat /etc/systemd/system/nginx.service
[Unit]
Description=Nginx
Wants=network-online.target
After=network-online.target

[Service]
WorkingDirectory=/var/lib/nginx
User=nginx
Group=nginx
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=no
TimeoutStopSec=70
Type=notify
NotifyAccess=all
ExecStartPre=/bin/rm -f %T/%N.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%T/%N.ctr-id --replace --rm -d --sdnotify=conmon --cgroups=no-conmon --name nginx nginx:mainline
ExecStop=/usr/bin/podman stop --cidfile=%T/%N.ctr-id -i
ExecStopPost=/usr/bin/podman rm --cidfile=%T/%N.ctr-id -f -i
KillMode=none

[Install]
WantedBy=default.target

❯ sudo systemctl daemon-reload

3. Start the unit:

❯ sudo systemctl start nginx

Describe the results you received:

Jan 09 14:54:00 Cubert systemd[1]: /etc/systemd/system/nginx.service:24: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
Jan 09 14:54:00 Cubert systemd[1]: Starting Nginx...
Jan 09 14:54:00 Cubert systemd[14978]: Started podman-15150.scope.
Jan 09 14:54:00 Cubert podman[15150]: Resolving "nginx" using unqualified-search registries (/etc/containers/registries.conf)
Jan 09 14:54:00 Cubert podman[15150]: Trying to pull docker.io/library/nginx:mainline...
Jan 09 14:54:03 Cubert podman[15150]: Getting image source signatures
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a0bcbecc962ed2552e817f45127ffb3d14be31642ef3548997f58ae054deb5b2
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a2abf6c4d29d43a4bf9fbb769f524d0fb36a2edab49819c1bf3e76f409f953ea
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a9edb18cadd1336142d6567ebee31be2a03c0905eeefe26cb150de7b0fbc520b
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:589b7251471a3d5fe4daccdddfefa02bdc32ffcba0a6d6a2768bf2c401faf115
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:186b1aaa4aa6c480e92fbd982ee7c08037ef85114fbed73dbb62503f24c1dd7d
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:b4df32aa5a72e2a4316aad3414508ccd907d87b4ad177abd7cbd62fa4dab2a2f
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:589b7251471a3d5fe4daccdddfefa02bdc32ffcba0a6d6a2768bf2c401faf115
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a0bcbecc962ed2552e817f45127ffb3d14be31642ef3548997f58ae054deb5b2
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a9edb18cadd1336142d6567ebee31be2a03c0905eeefe26cb150de7b0fbc520b
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:b4df32aa5a72e2a4316aad3414508ccd907d87b4ad177abd7cbd62fa4dab2a2f
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:a2abf6c4d29d43a4bf9fbb769f524d0fb36a2edab49819c1bf3e76f409f953ea
Jan 09 14:54:03 Cubert podman[15150]: Copying blob sha256:186b1aaa4aa6c480e92fbd982ee7c08037ef85114fbed73dbb62503f24c1dd7d
Jan 09 14:54:12 Cubert podman[15150]: Copying config sha256:605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85
Jan 09 14:54:12 Cubert podman[15150]: Writing manifest to image destination
Jan 09 14:54:12 Cubert podman[15150]: Storing signatures
Jan 09 14:54:12 Cubert podman[15150]:
Jan 09 14:54:12 Cubert podman[15150]: 2022-01-09 14:54:12.101247642 +0200 EET m=+11.607938154 container create 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a (image=docker.io/library/nginx:mainline, name=nginx, maintainer=NGINX Docker Maintainers <docker-maint@nginx.com>, PODMAN_SYSTEMD_UNIT=nginx.service)
Jan 09 14:54:12 Cubert systemd[14978]: Started libcrun container.
Jan 09 14:54:12 Cubert podman[15150]: 2022-01-09 14:54:00.536382139 +0200 EET m=+0.043073791 image pull  nginx:mainline
Jan 09 14:54:12 Cubert systemd[1]: user@200.service: Got notification message from PID 15150, but reception only permitted for main PID 14978
Jan 09 14:54:12 Cubert podman[15150]: 2022-01-09 14:54:12.141137063 +0200 EET m=+11.647827815 container init 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a (image=docker.io/library/nginx:mainline, name=nginx, PODMAN_SYSTEMD_UNIT=nginx.service, maintainer=NGINX Docker Maintainers <docker-maint@nginx.com>)
Jan 09 14:54:12 Cubert systemd[1]: user@200.service: Got notification message from PID 15150, but reception only permitted for main PID 14978
Jan 09 14:54:12 Cubert podman[15150]: 2022-01-09 14:54:12.145611861 +0200 EET m=+11.652302766 container start 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a (image=docker.io/library/nginx:mainline, name=nginx, PODMAN_SYSTEMD_UNIT=nginx.service, maintainer=NGINX Docker Maintainers <docker-maint@nginx.com>)
Jan 09 14:54:12 Cubert podman[15150]: 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
Jan 09 14:54:12 Cubert conmon[15215]: 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
Jan 09 14:54:12 Cubert conmon[15215]: 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
Jan 09 14:54:12 Cubert conmon[15215]: /docker-entrypoint.sh: Configuration complete; ready for start up
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: using the "epoll" event method
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: nginx/1.21.5
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: OS: Linux 5.15.0-2-amd64
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: start worker processes
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: start worker process 26
Jan 09 14:54:12 Cubert systemd[14978]: Started podman-15271.scope.
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 26#26: gracefully shutting down
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 26#26: exiting
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 26#26: exit
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: signal 17 (SIGCHLD) received from 26
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: worker process 26 exited with code 0
Jan 09 14:54:12 Cubert conmon[15215]: 2022/01/09 12:54:12 [notice] 1#1: exit
Jan 09 14:54:12 Cubert podman[15299]: 2022-01-09 14:54:12.393064442 +0200 EET m=+0.052274069 container remove 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a (image=docker.io/library/nginx:mainline, name=nginx, PODMAN_SYSTEMD_UNIT=nginx.service, maintainer=NGINX Docker Maintainers <docker-maint@nginx.com>)
Jan 09 14:54:12 Cubert podman[15271]: 7c7de83a412558d9ef53592734d3a52df9eecf331f696acfcdaac0ce33cf4c2a
Jan 09 14:54:12 Cubert systemd[14978]: podman-15150.scope: Consumed 7.547s CPU time.
Jan 09 14:54:12 Cubert systemd[1]: nginx.service: Failed with result 'protocol'.
Jan 09 14:54:12 Cubert systemd[1]: Failed to start Nginx.

Describe the results you expected:

Nginx runs until the end of time.

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.5
Built:        Thu Jan  1 02:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 1
  distribution:
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: Cubert
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 200
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 200
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-2-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 1015083008
  memTotal: 2041786368
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/200/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 8h 1m 8.23s (Approximately 0.33 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/lib/nginx/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/nginx/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/200/containers
  volumePath: /var/lib/nginx/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.5
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of apt list podman):

podman/testing,now 3.4.4+ds1-1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes and yes.

Additional environment details (AWS, VirtualBox, physical, etc.):

Machine is a VM.

[mheon]

This is a limitation on the systemd side. They will only accept notifications, or PID files, that are created by or sent by root, for security reasons - even if the User and Group of the unit file are explicitly set to start the process as a non-root user. Their recommendation was to start the container as a user service of the user in question via systemctl --user. There have been a few other issues about this, I'll try and dig them up.

[eriksjolund]

Previous discussion: #9642
It contains links to some issues.

[Gchbg]

Thank you both. For now I've worked around it by managing the service under the user's systemd which is clunky to say the least. I don't understand systemd's security argument - if the process is run as a given user, why would systemd not allow that user's process to send sd_notify? Who else could? But I guess this is no flaw of podman.

#9642 mentions some code changes that need to happen to podman for sd_notify, what are those? And have they progressed since March?

I guess you could close this issue or use it to track progress.

[vrothberg]

#9642 mentions some code changes that need to happen to podman for sd_notify, what are those? And have they progressed since March?

Yes, there is some progress. The main PID is now communicated via sd notify but there are still some remaining issues. For instance, %t resolves to the root's runtime dir - even when User=foo is set.

[vrothberg]

I think the next big thing to tackle is finding a way how to lift the User= setting. While the process in ExecStart itself is run as the specified User/Group, the systemd specifiers (e.g., %t, %U, etc) remain to be root.

[Gchbg]

[...] The main PID is now communicated via sd notify [...]

But even that is rejected by systemd, as seen in the logs above.

[vrothberg]

I fear there's not much Podman can do at the moment.

[wc7086]

Only after solving this problem can become truly rootless.

So I have to keep using the root account for now.

[svdHero]

Is there a quick overview what, at the moment, the best approach / workaround is for starting podman containers with systemd as a specific non-root user?

Furthermore, if a container is run as root, is there a workaround how to change the ownership of files and directories created inside the container (in a bound volume) to a specific host user?

[wc7086]

Furthermore, if a container is run as root, is there a workaround how to change the ownership of files and directories created inside the container (in a bound volume) to a specific host user?

use -e PUID=useruid -e PGID=usergid

use id username check UID and GID

[vrothberg]

Is there a quick overview what, at the moment, the best approach / workaround is for starting podman containers with systemd as a specific non-root user?

The services need to be started and managed as the specific non-root user. Using the User= directive does not work yet.

[Gchbg]

Is there a quick overview what, at the moment, the best approach / workaround is for starting podman containers with systemd as a specific non-root user?

For the moment my workaround is to run such containers in a systemd --user. This means that for every system service I want to run as a rootless container, I need to create a separate system user, enable linger, and run a separate systemd --user instance for that user.

It works but it's clunky, e.g. restarting Nginx is sudo su -l nginx -s /bin/sh -c 'XDG_RUNTIME_DIR="/run/user/$(id -u)" DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus" systemctl --user restart nginx' and running a command inside such a container might be something like sudo su -l nextcloud -s /bin/sh -c 'XDG_RUNTIME_DIR="/run/user/$(id -u)" DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus" podman exec -u www-data -w /var/www/html nextcloud ./occ status'.

Inside these rootless containers root is mapped to the system user, which is a different uid for each service. If something inside the containers runs as non-root, that gets mapped to a high-numbered host uid by default. However with some magic on the host you can map a specific non-root uid in the container to a host uid of your choice, which can then be mapped to a different non-root uid in a different container running under a different user.

I should probably document my setup one of these days...

[eriksjolund]

@Gchbg If you are running a recent systemd version (for instance by running Fedora 35), I think you could run

sudo systemd-run --machine=nginx@ --quiet --user --collect --pipe --wait systemctl --user restart nginx

No need to set DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR