Set default resource source based on environment

[nawatts]

Currently, resources will alway default to being read from Google Cloud Public Datasets.

gnomad_methods/gnomad/resources/config.py

Lines 14 to 16 in 748c5e4

	DEFAULT_GNOMAD_PUBLIC_RESOURCE_SOURCE = (
	GnomadPublicResourceSource.GOOGLE_CLOUD_PUBLIC_DATASETS
	)

Preferably, the default would be based on the environment, such that jobs running on EMR automatically default to reading resources from the Registry of Open Data on AWS and jobs running on HDInsight default to reading from Azure Open Datasets.

Maybe Hail already has something for identifying the environment (at least in clusters started with hailctl)?

[gtiao]

I can ask Dan if this is possible

[nawatts]

If nothing else, we can read the default source from an environment variable that can be set by either the user or hailctl. I was curious if Hail had anything to "guess" which cloud platform code was running on.

[gtiao]

Agreed, it would be nice if this could be automatic

[danking]

We did not have such a function, but we do have some heuristics as well as some environment variables that we set in all hailctl invocations. I've codified this as a new function in hail.utils: hail-is/hail#11230. If you have an EMR cluster running and can list the environment variables available there, I'm happy to update the heuristics function to include that too. I'm also happy to receive feedback on the returned strings (currently: "dataproc" and "hdinsight").

[nawatts]

Added the option to configure the default source with an environment variable in #435. Once hail-is/hail#11230 is merged and that API finalized, we can update get_default_public_resource_source to also use Hail's guess for the cloud provider.