cwltool does not properly detect available podman containers

[jmfernandez]

Next workflow https://github.com/inab/hello-workflows/tree/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl depends on container bash:5.2.15. The execution of it in podman mode with --disable-pull tag does not work as expected when the container is already in the local cache.

For instance, if you run

podman pull bash:5.2.15

it either fetches or updates the container with that tag, having it cached for its later usage. But it is also true that podman lists the available containers with a fully qualified tag:

podman images

REPOSITORY                             TAG            IMAGE ID      CREATED       SIZE
docker.io/library/bash                 5.2.15         3c04497fad88  4 days ago    14.4 MB

so internal cwltool code is failing to find the container as it relies on the output of podman images --no-trunc --all

cwltool/cwltool/docker.py

Lines 116 to 145 in 2fedbd6

	for line in (
	subprocess.check_output([self.docker_exec, "images", "--no-trunc", "--all"]) # nosec
	.decode("utf-8")
	.splitlines()
	):
	try:
	match = re.match(r"^([^ ]+)\s+([^ ]+)\s+([^ ]+)", line)
	split = docker_requirement["dockerImageId"].split(":")
	if len(split) == 1:
	split.append("latest")
	elif len(split) == 2:
	# if split[1] doesn't match valid tag names, it is a part of repository
	if not re.match(r"[\w][\w.-]{0,127}", split[1]):
	split[0] = split[0] + ":" + split[1]
	split[1] = "latest"
	elif len(split) == 3:
	if re.match(r"[\w][\w.-]{0,127}", split[2]):
	split[0] = split[0] + ":" + split[1]
	split[1] = split[2]
	del split[2]
	# check for repository:tag match or image id match
	if match and (
	(split[0] == match.group(1) and split[1] == match.group(2))
	or docker_requirement["dockerImageId"] == match.group(3)
	):
	found = True
	break
	except ValueError:
	pass

Maybe you could use a more targeted method, like using podman inspect bash:5.2.15, in order to learn whether the tag resolves to something in podman cache or not.

Expected Behavior

Once the container is in the local podman cache, running next command, where pulls are disabled, should work:

cwltool --debug --disable-pull --podman https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello.yml

Actual Behavior

cwltool complies about not finding the container image (see dump below)

Workflow Code

Workflow is available at https://github.com/inab/hello-workflows/tree/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl

Full Traceback

cwltool --debug --disable-pull --podman https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello.yml 
INFO /home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/bin/cwltool 3.1.20230719185429
DEBUG [workflow ] initialized from https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow
INFO [workflow ] start
DEBUG [workflow ] inputs {
    "an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
INFO [workflow ] starting step hello
DEBUG [step hello] job input {
    "https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow/hello/an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
DEBUG [step hello] evaluated job input to {
    "https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow/hello/an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
INFO [step hello] start
DEBUG [job hello] initializing from https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-tool.cwl#hello as part of step hello
DEBUG [job hello] {
    "an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
DEBUG [job hello] path mappings is {
    "https://raw.githubusercontent.com/etc/passwd": [
        "/tmp/tmppfeyow9v",
        "/var/lib/cwl/stgbcc097b6-d1f3-43fd-be23-3d2cc92c5991/passwd",
        "File",
        true
    ]
}
DEBUG [job hello] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bash"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "hello-world.bash"
    }
]
DEBUG Docker error
Traceback (most recent call last):
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 771, in run
    self.get_from_requirements(
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/docker.py", line 225, in get_from_requirements
    raise WorkflowException("Docker image %s not found" % r["dockerImageId"])
cwltool.errors.WorkflowException: Docker image bash:5.2.15 not found
ERROR Workflow or tool uses unsupported feature:
Docker is required to run this tool: Docker image bash:5.2.15 not found
Traceback (most recent call last):
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 771, in run
    self.get_from_requirements(
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/docker.py", line 225, in get_from_requirements
    raise WorkflowException("Docker image %s not found" % r["dockerImageId"])
cwltool.errors.WorkflowException: Docker image bash:5.2.15 not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/main.py", line 1298, in main
    (out, status) = real_executor(
                    ^^^^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 61, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 144, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 240, in run_jobs
    job.run(runtime_context)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 812, in run
    raise UnsupportedRequirement(
cwltool.errors.UnsupportedRequirement: Docker is required to run this tool: Docker image bash:5.2.15 not found

Your Environment

• cwltool version: 3.1.20230719185429
• podman version:

Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.20.5
Git Commit:   75e3c12579d391b81d871fd1cded6cf0d043550a
Built:        Sun Jun 25 00:09:49 2023
OS/Arch:      linux/amd64

[mr-c]

Fixed in #1891