"Authentication or permission failure" during build process

[LeHack]

ISSUE TYPE

• Bug Report

container.yml

version: "2"
settings:
  conductor_base: centos:7
services:
  web:
    ports:
      - 80:8000
    roles:
      - testapp

Where testapp is a python:3 based role.

SUMMARY

When running ansible-container build from Python 3.5, the process crashes right after completing the conductor image and starting to build the service image.
I'm not exactly sure whether this is an ansible-container or ansible bug, but I've been running Ansible with this setup (Python 3.5) for a while and never encountered this issue outside this use case.

STEPS TO REPRODUCE

Run ansible-container build

EXPECTED RESULTS

The service image should be built correctly.

ACTUAL RESULTS

<21d1e7dd4d26e037bf74c25b7f3fee568a15a976f589e91426826db64bf5ec2e> EXEC ['/usr/local/bin/docker', 'exec', '-i', u'21d1e7dd4d26e037bf74c25b7f3fee568a15a976f589e91426826db64bf5ec2e', u'/bin/sh', '-c', u'/bin/sh -c \'( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1496512485.12-114174128545119 `" && echo ansible-tmp-1496512485.12-114174128545119="` echo ~/.ansible/tmp/ansible-tmp-1496512485.12-114174128545119 `" ) && sleep 0\'']
fatal: [web]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp/ansible-tmp-1496512485.12-114174128545119 `\" && echo ansible-tmp-1496512485.12-114174128545119=\"` echo ~/.ansible/tmp/ansible-tmp-1496512485.12-114174128545119 `\" ), exited with result 1",
    "unreachable": true
}

WORKAROUND

Running ansible-container build --use-local-python completes the build correctly.

OS / ENVIRONMENT

Ansible Container, version 0.9.2rc0
Linux, localhost, 4.4.0-79-generic, #100~14.04.1-Ubuntu SMP Fri May 19 18:36:51 UTC 2017, x86_64
3.5.3 (default, May  6 2017, 00:22:52) 
[GCC 4.8.4] /home/.../bin/python3.5
{
  "InitBinary": "docker-init",
  "DriverStatus": [
    [
      "Root Dir",
      "/var/lib/docker/aufs"
    ],
    [
      "Backing Filesystem",
      "extfs"
    ],
    [
      "Dirs",
      "29"
    ],
    [
      "Dirperm1 Supported",
      "true"
    ]
  ],
  "ClusterAdvertise": "",
  "InitCommit": {
    "Expected": "949e6fa",
    "ID": "949e6fa"
  },
  "RuncCommit": {
    "Expected": "54296cf40ad8143b62dbcaa1d90e520a2136ddfe",
    "ID": "54296cf40ad8143b62dbcaa1d90e520a2136ddfe"
  },
  "Containers": 1,
  "Labels": null,
  "MemTotal": 6102839296,
  "Runtimes": {
    "runc": {
      "path": "docker-runc"
    }
  },
  "Plugins": {
    "Network": [
      "bridge",
      "host",
      "macvlan",
      "null",
      "overlay"
    ],
    "Volume": [
      "local"
    ],
    "Authorization": null
  },
  "ContainersPaused": 0,
  "DefaultRuntime": "runc",
  "SystemTime": "2017-06-03T20:26:24.275069044+02:00",
  "BridgeNfIptables": true,
  "NCPU": 4,
  "CpuCfsPeriod": true,
  "CPUSet": true,
  "IPv4Forwarding": true,
  "OomKillDisable": true,
  "Architecture": "x86_64",
  "SwapLimit": false,
  "Images": 14,
  "CPUShares": true,
  "LoggingDriver": "json-file",
  "OperatingSystem": "Ubuntu 14.04.5 LTS",
  "NoProxy": "",
  "KernelVersion": "4.4.0-79-generic",
  "HttpsProxy": "",
  "Isolation": "",
  "ExperimentalBuild": false,
  "BridgeNfIp6tables": true,
  "Name": "localhost",
  "MemoryLimit": true,
  "CgroupDriver": "cgroupfs",
  "SystemStatus": null,
  "IndexServerAddress": "https://index.docker.io/v1/",
  "DockerRootDir": "/var/lib/docker",
  "CpuCfsQuota": true,
  "SecurityOptions": [
    "name=apparmor"
  ],
  "Debug": false,
  "HttpProxy": "",
  "ID": "WATP:X33H:FWGQ:BAEX:CHVS:L3NA:KXEC:RFCU:B3VN:AANJ:PDDJ:OGLE",
  "RegistryConfig": {
    "IndexConfigs": {
      "docker-host:5000": {
        "Secure": false,
        "Name": "docker-host:5000",
        "Official": false,
        "Mirrors": []
      },
      "docker.io": {
        "Secure": true,
        "Name": "docker.io",
        "Official": true,
        "Mirrors": null
      }
    },
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ],
    "Mirrors": []
  },
  "ContainersStopped": 0,
  "ContainersRunning": 1,
  "Swarm": {
    "Nodes": 0,
    "Managers": 0,
    "ControlAvailable": false,
    "LocalNodeState": "inactive",
    "RemoteManagers": null,
    "Cluster": {
      "Spec": {
        "CAConfig": {},
        "TaskDefaults": {},
        "EncryptionConfig": {
          "AutoLockManagers": false
        },
        "Dispatcher": {},
        "Raft": {
          "ElectionTick": 0,
          "HeartbeatTick": 0
        },
        "Orchestration": {}
      },
      "CreatedAt": "0001-01-01T00:00:00Z",
      "UpdatedAt": "0001-01-01T00:00:00Z",
      "Version": {},
      "ID": ""
    },
    "NodeID": "",
    "NodeAddr": "",
    "Error": ""
  },
  "OSType": "linux",
  "NGoroutines": 26,
  "NFd": 20,
  "ServerVersion": "17.03.1-ce",
  "ContainerdCommit": {
    "Expected": "4ab9917febca54791c5f071a9d1f404867857fcc",
    "ID": "4ab9917febca54791c5f071a9d1f404867857fcc"
  },
  "NEventsListener": 0,
  "KernelMemory": true,
  "ClusterStore": "",
  "Driver": "aufs",
  "LiveRestoreEnabled": false
}
{
  "GoVersion": "go1.7.5",
  "BuildTime": "2017-03-27T17:10:36.401799458+00:00",
  "Version": "17.03.1-ce",
  "Arch": "amd64",
  "Os": "linux",
  "GitCommit": "c6d412e",
  "ApiVersion": "1.27",
  "MinAPIVersion": "1.12",
  "KernelVersion": "4.4.0-79-generic"
}

[j00bar]

Thank you for your interest in Ansible Container and for filing this - could you please include the full --debug build output? Thanks!

[LeHack]

Sure, here it is: debug-build.txt

[chouseknecht]

The failure is with setup.py. The generated playbook is attempting to gather_facts.

Here's the full error:

Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<c3db3492970f2e6dfd8e3fc3ff8c682ef4ab12730082fa08d8a7c0d9d6001542> ESTABLISH DOCKER CONNECTION FOR USER: root
<c3db3492970f2e6dfd8e3fc3ff8c682ef4ab12730082fa08d8a7c0d9d6001542> EXEC ['/usr/local/bin/docker', 'exec', '-i', u'c3db3492970f2e6dfd8e3fc3ff8c682ef4ab12730082fa08d8a7c0d9d6001542', u'/bin/sh', '-c', u"/bin/sh -c 'echo ~ && sleep 0'"]
<c3db3492970f2e6dfd8e3fc3ff8c682ef4ab12730082fa08d8a7c0d9d6001542> EXEC ['/usr/local/bin/docker', 'exec', '-i', u'c3db3492970f2e6dfd8e3fc3ff8c682ef4ab12730082fa08d8a7c0d9d6001542', u'/bin/sh', '-c', u'/bin/sh -c \'( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1496518691.11-78913299246777 `" && echo ansible-tmp-1496518691.11-78913299246777="` echo ~/.ansible/tmp/ansible-tmp-1496518691.11-78913299246777 `" ) && sleep 0\'']
fatal: [web]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp/ansible-tmp-1496518691.11-78913299246777 `\" && echo ansible-tmp-1496518691.11-78913299246777=\"` echo ~/.ansible/tmp/ansible-tmp-1496518691.11-78913299246777 `\" ), exited with result 1",
    "unreachable": true
}
	to retry, use: --limit @/tmp/tmpm58NjN/playbook.retry

PLAY RECAP *********************************************************************
web                        : ok=0    changed=0    unreachable=1    failed=0

We've seen this when attempting to use Python 3 inside the conductor, particularly with an Ubuntu base.

[j00bar]

It looks like the service you're building is based on the python:3 image, which is based on debian:jessie. Please try again using conductor_base: debian:jessie in your settings. Thanks!

[LeHack]

Hi.
I just tried it with debian:jessie as you suggested and it didn't crash with the "Authentication or permission failure" this time.
However something still seems off, here's what I got:

$ ansible-container build 
Building Docker Engine context...	
Starting Docker build of Ansible Container Conductor image (please be patient)...	
Parsing conductor CLI args.
Docker™ daemon integration engine loaded. Build starting.	project=ansible-container
Building service...	project=ansible-container service=web

PLAY [web] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [web]

TASK [testapp : Ensure /project/ exists] ***************************************
changed: [web]

TASK [testapp : copy] **********************************************************
changed: [web]

TASK [testapp : Make sure python requirements are met] *************************
fatal: [web]: FAILED! => {"changed": false, "cmd": "/_usr/local/bin/pip2 freeze", "failed": true, "msg": "\n:stderr: Traceback (most recent call last):\n  File \"/_usr/local/bin/pip2\", line 7, in <module>\n    from pip import main\nImportError: No module named pip\n"}
	to retry, use: --limit @/tmp/tmpWhneqG/playbook.retry

PLAY RECAP *********************************************************************
web                        : ok=3    changed=2    unreachable=0    failed=1

ERROR	Error applying role!	engine=<container.docker.engine.Engine object at 0x7f596caa2610> exit_code=2 playbook=[{'hosts': u'web', 'roles': ['testapp'], 'vars': {'playbook_debug': False}}]
Traceback (most recent call last):
  File "/usr/local/bin/conductor", line 11, in <module>
    load_entry_point('ansible-container', 'console_scripts', 'conductor')()
  File "/_ansible/container/__init__.py", line 19, in __wrapped__
    return fn(*args, **kwargs)
  File "/_ansible/container/cli.py", line 359, in conductor_commandline
    **params)
  File "/_ansible/container/__init__.py", line 19, in __wrapped__
    return fn(*args, **kwargs)
  File "/_ansible/container/core.py", line 726, in conductorcmd_build
    raise RuntimeError('Build failed.')
RuntimeError: Build failed.
Conductor terminated. Cleaning up.	command_rc=1 conductor_id=7ca1c55523a11b1a3d0094a1ebf62e655f41413d4db1fd5cfc500878edd51b3f save_container=False
ERROR	Conductor exited with status 1	

And here is the failed task definition:

- name: Make sure python requirements are met
  pip:
    requirements: /project/requirements.txt
    extra_args: --disable-pip-version-check

Does this mean that after changing the conductor_base (but still building the image from python:3) pip suddenly went missing?

[LeHack]

Attached is the build log for:

$ ansible-container --debug build -- -vvvvv >& build.txt

build.txt
Also note that (just to be sure) I also tried without the "extra_args: --disable-pip-version-check" and (as expected) it didn't change a thing.

[SalahAdDin]

Yes, i have the same problem now: OpenSaasAU/edx-docker-ansible#11