Add toolchain options API to WORKSPACE and Bzlmod

[mbland]

Description

Updates scala_toolchains() to accept either boolean or dict arguments for specific toolchains, and updates //scala/extensions:deps.bzl to generate them from tag classes. Part of #1482.

Notable qualities:

• Adds toolchain options support to the scala_toolchains() parameters scalafmt, scala_proto, and twitter_scrooge, and to the scalafmt tag class.

• Eliminates the scalafmt_default_config, scala_proto_options, and twitter_scrooge_deps option parameters from scala_toolchains().

• Provides uniform, strict evaluation and validation of toolchain options passed to scala_toolchains().

• Configures enabled toolchains using root module settings or the default toolchain settings only.

• Introduces the shared TOOLCHAIN_DEFAULTS dict in //scala/private:toolchains_defaults.bzl to aggregate individual TOOLCHAIN_DEFAULTS macro parameter dicts.

This change also:

• Replaces the non-dev dependency scala_deps.scala() tag instantiation in MODULE.bazel with dev_deps.scala().

• Renames the options parameter of the scala_deps.scala_proto tag class to default_gen_opts to match setup_scala_proto_toolchain().

• Introduces _stringify_args() to easily pass all toolchain macro args compiled from scala_toolchains_repo() attributes through to the generated BUILD file.

• Extracts _DEFAULT_TOOLCHAINS_REPO_NAME and removes the scala_toolchains_repo() macro.

• Includes docstrings for the new private implementation functions, and updates all other docstrings, README.md, and other relevant documentation accordingly.

Motivation

Inspired by @simuons's suggestion to replace toolchain macros with a module extension in:

• Enable Bzlmod #1722 (comment)

Though a full replacement is a ways off, this is a step in that direction that surfaced several immediate improvements.

First, extensibility and maintainability:

• The new implementation enables adding options support for other toolchains in the future while maintaining backward compatibility, for both the WORKSPACE and Bzlmod APIs. Adding this support will only require a minor release, not a major one.

• The scala_deps module extension implementation is easier to read, since all toolchains now share the _toolchain_settings mechanism.

Next, improved consistency of the API and implementation:

• Toolchain options parameters should present all the same parameters as the macros to which they correspond, ensured by the TOOLCHAIN_DEFAULTS mechanism. This is to make it easier for users and maintainers to see the direct relationship between these separate sets of parameters. (They can also define additional parameters to those required by the macro, like default_config from the scalafmt options.)

  This principle drove the renaming of the scala_deps.scala_proto tag class parameter from options to default_gen_opts. It also inspired updating scala_toolchains_repo() to pass toolchain options through _stringify_args() to generate BUILD macro arguments.

• The consolidated TOOLCHAIN_DEFAULTS dict reduces duplication between the scala/extensions/deps.bzl and scala/toolchains.bzl files. It ensures consistency between tag class attr default values for Bzlmod users and the scala_toolchains() default parameter values for WORKSPACE users.

  The TOOLCHAINS_DEFAULTS dicts corresponding to each toolchain macro do duplicate the information in the macro argument lists. However, the duplicated values in this case are physically adjacent to one another, minimizing the risk of drift.

• Extracting _DEFAULT_TOOLCHAINS_REPO_NAME is a step towards enabling customized repositories based on the builtin toolchains, while specifying different options. This extraction revealed the fact that the scala_toolchains_repo() macro was no longer necessary, since only scala_toolchains() ever called it.

Finally, fixes for the following design bugs:

• Previously, scala_deps.scala_proto(options = [...]) compiled the list of default_gen_opts from all tag instances in the module graph. This might've been convenient, but didn't generalize to other options for other toolchains. In particular, it differed from the previous toolchains, scalafmt, and twitter_scrooge tag class behavior.

  The new semantics are unambiguous, consistent, and apply to all toolchains equally; they do not show a preference for any one toolchain over the others. They also maintain the existing scalafmt and twitter_scrooge tag class semantics, but now using a generic mechanism, simplifying the implementation.

• Instantating scala_deps.scala() was a bug left over from the decision in Enable Bzlmod #1722 not to enable the builtin Scala toolchain by default under Bzlmod.

  The previous scala_deps.toolchains() tag class had a default scala = True parameter. The user could set scala = False to disable the builtin Scala toolchain. After replacing toolchains() with individual tag classes, the documented behavior was that the user must enable the builtin Scala toolchain by instantiating scala_deps.scala().

  By instantiating scala_deps.scala() in our own MODULE.bazel file, we ensured that rules_scala would always instantiate the builtin Scala toolchain. While relatively harmless, it violated the intention of allowing the user to avoid instantiating the toolchain altogether.

[mbland]

@simuons @liucijus I'd consider this the last essential change before releasing the next version, since it updates both the WORKSPACE and Bzlmod APIs.

[mbland]

@meteorcloudy I've noticed the cla/google check hanging today when pushing commits to this branch. Is this a consequence of the move to the bazel-contrib org?

[mbland]

@alexeagle took off the cla/google check, since the bazel-contrib org doesn't require it, so that's no longer blocking anything.

@simuons Separate from this pull request, I've continued to experiment a bit with generating toolchains from the module extension in the new bzlmod-toolchains-experiment branch of mbland/rules_scala. I've managed to generate a separate toolchain in @rules_scala_toolchains//scala after updating the scala_deps.scala() tag class.

This confirms that, after this pull request lands, and after the new release, we can indeed expand the tag classes to pass through more parameters and/or generate new toolchains in @rules_scala_toolchains. Such updates can remain backward compatible, requiring only minor version releases. It may even be possible to support an arbitrary number of toolchains and/or separate toolchain repos in a minor version update, though I've yet to get around to experimenting with that yet.

[simuons]

Thanks, @mbland