Cannot import a dumpdir containing constrained indices

[amoutaux]

Hello,

I created a dumpdir using devpi-export. My devpi server currently runs with constrained index. The devpi-export command fails with:

Traceback (most recent call last):
  File "/usr/local/bin/devpi-import", line 8, in <module>
    sys.exit(import_())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/importexport.py", line 154, in import_
    do_import(config.args.directory, xom)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/importexport.py", line 111, in do_import
    importer.import_all(path)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/importexport.py", line 482, in import_all
    stage = user.create_stage(index, type=indexconfig['type'])
  File "/usr/local/lib/python3.10/site-packages/devpi_server/model.py", line 430, in create_stage
    return self.parent.create_stage(self, index, type=type, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/keyfs.py", line 503, in create_stage
    self.model_cache[key] = super().create_stage(
  File "/usr/local/lib/python3.10/site-packages/devpi_server/model.py", line 173, in create_stage
    stage._modify(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/model.py", line 1044, in _modify
    self.customizer.validate_config(oldconfig, newconfig)
  File "/usr/local/lib/python3.10/site-packages/devpi_constrained/main.py", line 62, in validate_config
    raise self.InvalidIndexconfig(errors)
devpi_server.model.InvalidIndexconfig: ['A constrained index requires at least one base']

Extract of the dataindex.json for a test/dev index failing to be created:

    "test/dev": {
      "indexconfig": {
        "acl_upload": [
          "test"
        ],
        "mirror_whitelist": [
          "xgboost"
        ],
        "volatile": true,
        "mirror_whitelist_inheritance": "union",
        "description": "test",
        "acl_toxresult_upload": [":ANONYMOUS:"],
        "title": "test",
        "bases": [
          "test/upload",
          "root/pypi-wheels"
        ],
        "constraints": [
          "virtualenv<v20.24.6",
          "pytest<=8.1.2",
          "twine!=5.1.0"
        ],
        "type": "constrained"
      },
      "projects": {},
      "files": []
    }

If I am not mistaken, the issue lies in the user.create_stage being called before bases are set on the newly created indices:

devpi/server/devpi_server/importexport.py

Lines 462 to 482 in be69c6f

	bases = indexconfig.pop('bases', None)
	stage = None
	if stagename == "root/pypi":
	# see if the stage was already created
	stage = self.xom.model.getstage(stagename)
	if self.xom.config.no_root_pypi:
	continue
	if stage is None:
	# we create a stage without any config
	stage = user.create_stage(index, type=indexconfig['type'])
	if stage is not None:
	# we use modify with _keep_unknown, so data from
	# removed plugins is preserved
	stage.modify(**indexconfig, _keep_unknown=True)
	if "bases" in import_index["indexconfig"]:
	# we are changing bases directly to allow import with
	# removed bases without changing the data from the export
	with stage.user.key.update() as userconfig:
	indexconfig = userconfig['indexes'][stage.index]
	indexconfig["bases"] = tuple(bases)
	stages.append(stage)

Thanks for this great product ❤️

[fschulze]

I think it is better to change the validation in devpi-constrained. A constrained index doesn't really require a base, it just doesn't do anything useful without one. Changing the import semantics would be way harder because of the ways bases work.

[amoutaux]

Thanks for your prompt feedback. I raised fschulze/devpi-constrained#1 so we can iterate on the devpi-constrained repository :).

[amoutaux]

Hi @fschulze , I just realized I targeted the wrong directory for the PR.. Here is the correct one: devpi/devpi-constrained#8