-
Notifications
You must be signed in to change notification settings - Fork 31
Support multiple resource buckets #373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
62ff039
to
b868fa9
Compare
return f"gs://gcp-public-data--gnomad{relative_path}" | ||
|
||
return ( | ||
f"{resource_source.rstrip('/')}{relative_path}" # pylint: disable=no-member |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't figure out why Pylint insisted this was an error. The tests show that this works.
Pylint also only complained about this when running in the PR checks. It passed when running Pylint locally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly questions for my knowledge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just one comment, but lgtm. Thank you for answering all my questions and adding all the extra tests!
assert ds == read_block_matrix.return_value | ||
|
||
|
||
def gnomad_public_resource_test_parameters(path: str): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add return type
Co-authored-by: jkgoodrich <[email protected]>
92fe502
to
842768c
Compare
This adds support for multiple resource buckets in order to provide access to resources hosted on different cloud providers through our resources framework (#255, #263, #264).
This adds a
GnomadPublicResource
resource class and subclasses for each resource type (GnomadPublicTableResource
,GnomadPublicMatrixTableResource
, etc). When reading data,GnomadPublicResource
rewrite theirpath
attribute based on the value ofgnomad.resources.config.gnomad_public_resource_configuration.source
.Thus, to read resources from
gcp-public-data--gnomad
instead ofgnomad-public-requester-pays
,Alternatively, a custom path can be used (for example, if someone has created their own copy of the gnomAD public buckets).