AMP-97070 Move Spark metadata upon transfer completion #8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fzqgriffin
reviewed
Apr 4, 2024
| print(f'Identified bucket: {bucket}, prefix: {prefix}') | ||
|
|
||
| # List all files in the s3_path directory | ||
| response = s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix, Delimiter='/') |
There was a problem hiding this comment.
Will this return folders as well? If only files are returned, then we are good.
There was a problem hiding this comment.
only files. Same for listing objects in our Java repo using Amplitude's S3 wrapper
The example job mentioned in the description has the logs of what was discovered (note that /meta folder exists at the point of execution of this method and is not listed)
| if '://' not in s3_uri_with_spark_metadata: | ||
| raise ValueError(f'Invalid s3 URI: {s3_uri_with_spark_metadata}. Expected to contain "://".') | ||
| bucket, prefix = s3_uri_with_spark_metadata.split('://')[1].split('/', 1) | ||
| bucket = replace_double_slashes_with_single_slash(bucket) |
There was a problem hiding this comment.
curious where double slashes are coming from?
There was a problem hiding this comment.
for bucket, it shouldn't
It's just a sanitization in case some unnormalized input for s3uri is provided (e.g. s3:////bucket////prefix////)
| expected_output = '/path/to/file/with/double/slashes/end/' | ||
| self.assertEqual(expected_output, replace_double_slashes_with_single_slash(input_string)) | ||
|
|
||
| def test_move_spark_metadata_to_separate_s3_folder(self): |
There was a problem hiding this comment.
thanks for adding tests!
fzqgriffin
approved these changes
Apr 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Move spark transfer metadata to a subdirectory of the target export S3 location
Testing
DatabricksToS3WorkerServiceIntegrationTest.testUnloadData