package alternatives in Pkg3?

[stevengj]

I wonder whether we want Pkg3 to contain something like Debian alternatives: a package A can depend on package B or package C or package D.

For example, the Plots.jl package might want to depend on having at least one plotting package installed (cc @tbreloff).

Or, now that FFTW.jl is being split off into its own package, and that package in turn will probably depend on an AbstractFFTs package (JuliaMath/FFTW.jl#2), other packages may want to depend on having some package installed that implements FFTs following the AbstractFFT interface (e.g. FFTW, or a native Julia FFT, or FFTPACK, or MKL, etcetera).

[simonbyrne]

Similarly, there should be a way for users to specify which option they want to use when multiple are available, presumably via a TOML file?

[oxinabox]

The situtation perhaps is more complex.
Though we might be able to ignore the complex case and get the low hanging fruit.

• JuMP requires you to have a Solver package installed. of which there are 15
• If you want to solve a MIP then JuMP requires a MIP solver. Of which there are 8
• If you want to solve a MIP with lazy (callback) constrains then you need a MIP solver with that support. Of which there are 3 or 4
• if it is a NLMIP with lazy constraints you need such a solver. Of which there are none.
• If you both want to solve LP and MIP you need either two different solvers or one of 6 solvers that support both

Etc etc,
the constraints are fairly complex.

[stevengj]

@oxinabox, that doesn't seem too much more complex to me. It just means:

• A package can have multiple sets of optional dependencies.

• Optional dependencies should include "none" as an option (in which case the package should disable the corresponding functionality).

[tkelman]

Would this be mutually exclusive, with only one package allowed to actively satisfy the dependency at a time?

[stevengj]

@tkelman, I don't think so. At Pkg.build time, the package should be able to find out which (if any) of the optional dependencies were installed, and can then decide to use all of them or just one of them depending on how the package works.

But there should be a way for the user to indicate a preference if the package is just going to use one of the options.

[stevengj]

I'm thinking of an interface something like the following. Suppose MyPackage wants at least one of three different FFT packages, and optionally can use a plotting package or can disable plotting if no such package is present. All of these optional packages are declared as optional=true packages in the Pkg3 TOML file. In addition, you add section declaring the alternatives preference:

[option.fft]
packages = ["FFTW", "MKL", "FFTPACK"]

[option.plots]
packages = ["", "PyPlot", "Gadfly"]  # "" means it is okay to install nothing

Then, in Pkg.add("MyPackage"), if one of the options is not already present, it will install the default ("FFTW" for fft and nothing for plots).

Pkg.option("MyPackage", :plots) will return a list of all installed plots options (or an empty list if none), in the default order ["PyPlot", "Gadfly"] unless the user has changed the default. The build script for MyPackage will typically call this function to decide which option to configure with (if any).

Pkg.option("MyPackage", fft="MKL") will change the fft option to "MKL", installing "MKL" if needed, and rebuilding MyPackage if the option has changed; Pkg.option("MyPackage", :fft) will now list "MKL" first (returning ["MKL", "FFTW"] if FFTW is also installed).

You can specify more than one package, e.g. Pkg.option("MyPackage", fft=("MKL","FFTPACK")), and it will install both of these dependencies. Pkg.option("MyPackage", :fft) will return the dependencies with the user-specified ones first: ["MKL", "FFTPACK", "FFTW"]

You can also specify the option as keywords in Pkg.add or Pkg.build. e.g. Pkg.add("MyPackage", fft="MKL").

[StefanKarpinski]

@KristofferC and I have been brainstorming and debating this for a while and so far the best scheme we've come up with is something like this made up example:

[deps]
ABC   = "<uuid>"
Gtk   = "<uuid>"
TclTk = "<uuid>"
Qt    = "<uuid>"
XYZ   = "<uuid>"

[alts]
graphics = ["Gtk", "TclTk", "Qt"]

[build.deps]
Builder = "<uuid>"
Clang   = "<uuid>"
GCC     = "<uuid>"
MSVC    = "<uuid>"

[build.alts.compiler]
linux   = ["GCC", "Clang"]
macos   = ["Clang"]
windows = ["MSVC"]

[test.deps]
Test       = "<uuid>"
QuickCheck = "<uuid>"

There are two different things at play here – alternatives and targets:

• build and test are targets which add dependencies on top of the main set of dependencies.
• graphics is a set of alternatives for the default target, letting you choose between one of three graphics backends – Gtk, TclTk and Qt
• compiler is a set of alternatives for the build target that lets you choose a compiler in a platform-specific way – between GCC and Clang on Linux, Clang on macOS and MSVC on Windows.

The general rules are that:

• Every package in deps is requires by the default target unless it appears on the RHS of a set an entry in the atls section.
• The LHS of every entry in the alts section is required in place of its RHS alternatives.
• Thus, the default target on the example requires ABC, XYZ and graphics which may be one of Gtk, TclTk or Qt

Each target is a microcosm of the top-level with its own deps and alts sections. These can be merged as dicts, as in:

target_deps = merge(project["deps"], project[target]["deps"])
target_alts = merge(project["alts"], project[target]["alts"])

If there are any key collisions it should be an error – targets allowed to override the identity of a package. A generalization of this maybe this:

target_project = recursive_merge(project, project[target])

where recursive_merge recurses into dictionary values and errors on key collisions. The resulting target_project can be saved as a project file and used for executing the target – build, test, etc.

[StefanKarpinski]

The motivation for naming sets of alternatives is that it allows configuration files to specify them as in:

[config.SomePackage]
graphics = "Qt"
compiler = "Clang"

[mauro3]

Maybe its worth spelling alts out as I think it wouldn't be used that much? Unlike deps it's not a "word". Another word would be choices or choiceof, which reads well in above example.

[StefanKarpinski]

I was originally writing out alternatives but I got tired of typing it over again.

[stevengj]

I find the compiler to be a bit out of place … how is that fundamentally different from any other non-Julia dependency? Hopefully, the vast majority of projects will use something like BinaryProvider.jl and won't require a compiler.