Reading Credential data for GCP

[dacbd]

With a gs:// remote DVC will look at the env var of GOOGLE_APPLICATION_CREDENTIALS for a path to the JSON file for a service account but doesn't accept GOOGLE_APPLICATION_CREDENTIALS_DATA which contains the actual JSON

[daavoo]

Is GOOGLE_APPLICATION_CREDENTIALS_DATA supported by Google Cloud API ?

[daavoo]

For the record, we rely on https://github.com/fsspec/gcsfs for handling gs remotes so we support what they support for credentials: https://gcsfs.readthedocs.io/en/latest/#credentials

[dacbd]

Is GOOGLE_APPLICATION_CREDENTIALS_DATA supported by Google Cloud API ?

Fair enough it does look like CML is just special in accepting that 🙈

[daavoo]

Closing it for now. @dacbd feel free to reopen if you would like to request that feature

[bmabir17-asj]

@dacbd as CML supports GOOGLE_APPLICATION_CREDENTIALS_DATA but DVC does not. How do you suggest we store service account credentials/keys and feed it into DVC?

[dacbd]

@dacbd as CML supports GOOGLE_APPLICATION_CREDENTIALS_DATA but DVC does not. How do you suggest we store service account credentials/keys and feed it into DVC?

@bmabir17-asj in general or from cicd / a cml runner instance?

CML supports reading the JSON key file where GOOGLE_APPLICATION_CREDENTIALS is a path to that file, which DVC supports.

If you are running DVC cmds from a CML made instance within something like GitHub actions I would add this: --cloud-permission-set=dvc-sa@[gcp-project].iam.gserviceaccount.com,scopes=storage-rw to your cml runner command (where this service account has permissions to the gs bucket)

I have been passively improving the documentation to make this clearer. In the mean time, feel free to the reach out if you have anymore questions discord

[bmabir17-asj]

@dacbd yes, thank you for the info. I did not find anything regarding this in the documentation except in this issue .
If possible please add this to the documentation as well.

[dacbd]

@bmabir17-asj indeed you are correct, I have my "in progress" update here: iterative/cml.dev#208 but I put a pause on working on it for "reasons" 😶‍🌫️ but intend would like to have something helpful there by the end of this week!