feat: batch posix fs connector (__for_testing_only_batch_posix_fs) #22733
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 tasks
github-actions
bot
added
type/feature
This was referenced Jul 28, 2025
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
cf7922e to
032c049
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
f1e5606 to
82f268b
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
a6138ae to
f74bdf5
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
82f268b to
ce71aab
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
f74bdf5 to
b4aa469
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
2 times, most recently
from
129a087 to
c0fe9da
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
b4aa469 to
cd08960
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
c0fe9da to
d44e849
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
cd08960 to
82da810
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Bugbot free trial expires on July 29, 2025
Learn more in the Cursor dashboard.
| Ok(vec![BatchPosixFsSplit::new( | ||
| self.properties.root.clone(), // file_path is the root | ||
| "114514".into(), // split_id does not matter | ||
| )]) |
There was a problem hiding this comment.
Bug: BatchPosixFs Connector Violates Unique Split ID Requirement
The BatchPosixFs connector hardcodes the split_id to "114514" for all instances. This violates the documented requirement for a unique split identifier, leading to conflicts and incorrect behavior when multiple BatchPosixFs sources are used, as SplitId serves as a unique key for split tracking and state management. The accompanying comment "split_id does not matter" is misleading.
xxchan
commented
Jul 29, 2025
| @@ -44,6 +46,7 @@ pub const GCS_CONNECTOR: &str = "gcs"; | |||
| /// If user inputs `connector='s3_v2'`, it will be rejected. | |||
| pub const OPENDAL_S3_CONNECTOR: &str = "s3_v2"; | |||
| pub const POSIX_FS_CONNECTOR: &str = "posix_fs"; | |||
| pub const BATCH_POSIX_FS_CONNECTOR: &str = "__for_testing_only_batch_posix_fs"; | |||
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
82da810 to
1739d0b
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
92a451a to
be94be6
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
1739d0b to
bdba77b
Compare
10 tasks
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
bdba77b to
f446d39
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
be94be6 to
0266dcb
Compare
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
f446d39 to
3b205df
Compare
xxchan
changed the title
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
3b205df to
e14e9c3
Compare
Signed-off-by: xxchan <[email protected]> add note for recovery add note for spawn resolve comments
xxchan
force-pushed
the
feat_add_batch_posix_fs_connector
branch
from
e14e9c3 to
3e3193c
Compare
xxchan
force-pushed
the
07-14-feat_refreshable_source_sourceexec_part
branch
from
a0b55a0 to
26605cd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Checklist
Documentation
Release note