Run the tests with udocker in GitHub Actions (matrix)

[kinow]

Closes common-workflow-language/cwltool#782

Running locally, I get an error that udocker is not able to pull the python:3-slim image. After some debugging, I think it's because it's missing the username in the image.

docker pull python:3-slim is smart to translate that into docker pull library/python:3-slim. But udocker doesn't prepend the default library (user for curated images), and fails to pull it locally for me.

If I replace the dockerPull with library/python:3-slim, it works, but some tests fail, e.g.

Test command execution in Docker with stdin and stdout redirection
Compare failure expected: {
    "output_txt": {
        "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
        "class": "File",
        "location": "output.txt",
        "size": 13
    }
}
got: {
    "output_txt": {
        "basename": "output.txt",
        "checksum": "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c",
        "class": "File",
        "location": "file:///tmp/tmpye4qw_do/output.txt",
        "path": "/tmp/tmpye4qw_do/output.txt",
        "size": 73
    }
}
caused by: failed comparison for key 'output_txt': expected: {
    "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
    "class": "File",
    "location": "output.txt",
    "size": 13
}
got: {
    "basename": "output.txt",
    "checksum": "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c",
    "class": "File",
    "location": "file:///tmp/tmpye4qw_do/output.txt",
    "path": "/tmp/tmpye4qw_do/output.txt",
    "size": 73
}
caused by: field 'checksum' failed comparison: expected: "sha1$47a013e660d408619d894b20806b1d5086aab03b"
got: "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c"


Test command execution in Docker with mediumcut stdout redirection
Compare failure expected: {
    "output_file": {
        "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
        "class": "File",
        "location": "cat-out",
        "size": 13
    }
}
got: {
    "output_file": {
        "basename": "cat-out",
        "checksum": "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c",
        "class": "File",
        "location": "file:///tmp/tmphwa0ha0z/cat-out",
        "path": "/tmp/tmphwa0ha0z/cat-out",
        "size": 73
    }
}
caused by: failed comparison for key 'output_file': expected: {
    "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
    "class": "File",
    "location": "cat-out",
    "size": 13
}
got: {
    "basename": "cat-out",
    "checksum": "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c",
    "class": "File",
    "location": "file:///tmp/tmphwa0ha0z/cat-out",
    "path": "/tmp/tmphwa0ha0z/cat-out",
    "size": 73
}
caused by: field 'checksum' failed comparison: expected: "sha1$47a013e660d408619d894b20806b1d5086aab03b"
got: "sha1$b4bfd0fa40eaea6c3441b2994fc8d6e3f0d7185c"

I might be doing something wrong, as I also get errors with --user-space-docker-cmd=docker.

-Bruno

[kinow]

Right now GH Actions has the exact same error I have on my Ubuntu when running the conformance tests with udocker:

ERROR Got workflow error: Command '['udocker', 'pull', 'docker.io/python:3-slim']' returned non-zero exit status 1.

If you try the command locally with the latest release you should get the same error, udocker pull docker.io/python:3-slilm. I tried changing the settings in the udocker.conf file, increasing verbosity, but everything looked OK.

The only way I found to fix it was by replacing in the workflows the image name, prefixing it with library/, creating the command udocker pull library/python:3-slim. Maybe worth asking the udocker devs if that's a known issue, or if there's a workaround? Or is that something we should handle in the workflows, always prefixing the user name (i.e. use <registry>/<user>/<image>:<tag>, as in docker.io/library/python:3.-lim)?

[mr-c]

@kinow what does udocker -D pull docker.io/python:3-slim show for you? It succeeds for me locally using udocker version 1.1.3

[kinow]

udocker -D pull docker.io/python:3-slim

(venv) kinow@ranma:~/Development/python/workspace/cwltool$ udocker --version
Info: using config file:  /home/kinow/.udocker/udocker.conf
Info: using curl executable 
version: 1.3.1
tarball: https://download.ncg.ingrid.pt/webdav/udocker/udocker-englib-1.2.8.tar.gz https://raw.githubusercontent.com/jorge-lip/udocker-builds/master/tarballs/udocker-englib-1.2.8.tar.gz
tarball_release: 1.2.8

# omitted tokens
(venv) kinow@ranma:~/Development/python/workspace/cwltool$ udocker -D pull docker.io/python:3-slim
Info: using config file:  /home/kinow/.udocker/udocker.conf
Info: using curl executable 
Info: using curl executable 
Info: read buf 1.2.8

Info: get imagerepo: docker.io/python tag: 3-slim
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 401 Unauthorized', 'X-ND-CURLSTATUS': 0, 'content-type': 'application/json', 'docker-distribution-api-version': 'registry/2.0', 'www-authenticate': 'Bearer realm="https://auth.docker.io/token",service="registry.docker.io"', 'date': 'Sat, 21 May 2022 12:18:17 GMT', 'content-length': '87', 'strict-transport-security': 'max-age=31536000'}
Info: buffer: b'{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}\n'
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 200 OK', 'X-ND-CURLSTATUS': 0, 'content-type': 'application/json; charset=utf-8', 'date': 'Sat, 21 May 2022 12:18:18 GMT', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=31536000', 'X-ND-HEADERS': []}
Info: buffer: b'{"token":"***","access_token":"***","expires_in":300,"issued_at":"2022-05-21T12:18:18.247276753Z"}\n'
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 200 OK', 'X-ND-CURLSTATUS': 0, 'content-length': '2', 'content-type': 'application/json', 'docker-distribution-api-version': 'registry/2.0', 'date': 'Sat, 21 May 2022 12:18:19 GMT', 'strict-transport-security': 'max-age=31536000', 'X-ND-HEADERS': ['Authorization: Bearer ***']}
Info: buffer: b'{}'
Info: manifest url https://registry-1.docker.io/v2/python/manifests/3-slim
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 401 Unauthorized', 'X-ND-CURLSTATUS': 0, 'content-type': 'application/json', 'docker-distribution-api-version': 'registry/2.0', 'www-authenticate': 'Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:python:pull"', 'date': 'Sat, 21 May 2022 12:18:20 GMT', 'content-length': '149', 'strict-transport-security': 'max-age=31536000'}
Info: buffer: b'{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"python","Action":"pull"}]}]}\n'
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 200 OK', 'X-ND-CURLSTATUS': 0, 'content-type': 'application/json; charset=utf-8', 'date': 'Sat, 21 May 2022 12:18:20 GMT', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=31536000', 'X-ND-HEADERS': []}
Info: buffer: b'{"token":"***","access_token":"***","expires_in":300,"issued_at":"2022-05-21T12:18:20.951289576Z"}\n'
Info: header: {'X-ND-HTTPSTATUS': 'HTTP/1.1 401 Unauthorized', 'X-ND-CURLSTATUS': 0, 'content-type': 'application/json', 'docker-distribution-api-version': 'registry/2.0', 'www-authenticate': 'Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:python:pull",error="insufficient_scope"', 'date': 'Sat, 21 May 2022 12:18:21 GMT', 'content-length': '149', 'strict-transport-security': 'max-age=31536000', 'X-ND-HEADERS': ['Authorization: Bearer ***']}
Info: buffer: b'{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"python","Action":"pull"}]}]}\n'
Error: manifest not found or not authorized
Error: no files downloaded

[mr-c]

@kinow Your tarball_release: 1.2.8 is out of date; should be 1.3.1 as well

[kinow]

@kinow Your tarball_release: 1.2.8 is out of date; should be 1.3.1 as well

That's funny. I installed it in my venv with pip install udocker, but I think others had a similar issue, e.g. indigo-dc/udocker#352

Let me uninstall from the virtual environment and install it some other way.

[kinow]

Hmmm, exact same behavior after installing udocker via curl. Had a look at their docs, the repository they mention for the tools appears to be this one? https://github.com/jorge-lip/udocker-builds/tree/master/tarballs

But I couldn't find a 1.3.1 in that repository. Do you get the same version for both udocker and udocker-englib running udocker version, @mr-c ?

[mr-c]

udocker install should pull down a matching version; Try moving ~/.udocker to a different location and running udocker install again.

Here are the URLs it tries: https://github.com/indigo-dc/udocker/blob/dcc4972f3abf2aac8de12ce8a0453997d8af7cdc/udocker/config.py#L34

[mr-c]

okay, we are seeing behavior difference between udocker 1.1.3 (my version) and the latest 1.3.1;

From the earlier errors, I'm guessing we are hitting some API limit; we'll need to use udocker login I think

[mr-c]

Upgrading to udocker 1.3.1 I also see that udocker pull docker.io/python:3-slim fails, but udocker pull python:3-slim succeeds 😿

[mr-c]

I opened a new issue over there: indigo-dc/udocker#359

[kinow]

I opened a new issue over there: indigo-dc/udocker#359

Thanks @mr-c !

[mr-c]

Interesting. Maybe better to build the matrix with options, a string gets used later to set CWLTOOL_OPTIONS? That way we can also use --singularity, for example

[kinow]

Good idea! I'll update it later, thanks Michael!

[kinow]

Done @mr-c. I did a very (really very) quick testing, and I think the issue wasn't fixed on udocker side yet. So probably both jobs will fail. For the default value, docker, we probably want to comment that out or remove that line I think? Since that won't work on GH Actions as it's already running with Docker (unless we want to try that docker on docker approach). Cheers.

[mr-c]

Getting closer with the udocker 1.3.5 release; we also need common-workflow-language/cwltool#1761

[netlify]

✅ Deploy Preview for cwl-v1-2-dev ready!

Name	Link
🔨 Latest commit	4d68050
🔍 Latest deploy log	https://app.netlify.com/sites/cwl-v1-2-dev/deploys/64de304af46dc0000981f461
😎 Deploy Preview	https://deploy-preview-180--cwl-v1-2-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.