[DISCUSSION] Proposal move `object_store` to its own github repo?

[alamb]

Which part is this question about
This guthub repository contains an implementation of arrow and parquet and object_store which are related but are in separate crates and reasonably could be in separate repos

. arrow and parquet are still released in lockstep (have to be released at the same time, use the same Arrow voting thread)

However, object_store is released on a different schedule, with a different voting thread, a diffrent process, and has a non-trivally different set of maintainers and a substantial number of other users

While we have tags to separate issues and PRs of object_store I still find it confusing that this repo has content related to object_store

I believe the reason object_store is in the arrow-rs repo in the first place was convenience for the maintainers after it was first donated: #2030

Now that we have settled the API down and its development and release cycle becomes decoupled from the other crates in this repo I think the overhead of keeping it in the same crate is greater than the value we get from keeping it in the same one

Describe your question

1. Do you think we move the object_store crate and associated Dev process (tickets, etc) to its own repository
2. If so, to which one (perhaps apache/arrow-rs-object-store )
3. Are you willing to help 🎣 make it happen?

Additional context

cc @tustvold and @crepererum

[Xuanwo]

Hi, @alamb, thanks a lot for raising this discussion.

Do you think we move the object_store crate and associated Dev process (tickets, etc) to its own repository

Most of my contributions, both in coding and reviewing, were focused on object_store. I feel that object_store is distinct from other arrow-related crates. I believe it would be beneficial to move object_store and its development process to a separate repository.

If so, to which one (perhaps apache/arrow-rs-object-store )

Since object_store is a subproject of arrow-rs, the natural name for it would be apache/arrow-rs-object-store.

• The arrow- prefix is required unless we decided to move it out of arrow community (in incubator or other TLP).
• The rs- prefix is used to associate it with rust and avoid confusion with other subprojects.

Are you willing to help 🎣 make it happen?

I'm willing to help do this.

But I'm guessing we need:

• A PMC member or a committer to create repo apache/arrow-rs-object-store
• A committer to extract the git history to new repo.
• A committer to move all object_store related issues/discussions to new repo

[alamb]

But I'm guessing we need:

I agree -- thank you. I happen to be both a PMC member of Arrow / committer so I have the relevant permissions.

As long as there are no concerns I'll send a note to the arrow list in a few days

A committer to move all object_store related issues/discussions to new repo

I recently learned about "collaborators" in asf.yml https://github.com/apache/infrastructure-asfyaml?tab=readme-ov-file#triage where we could give someone rights to manage issues so in theory someone else could also help with thi

[alamb]

I'm willing to help do this.

(thank you!)

[Xuanwo]

I recently learned about "collaborators" in asf.yml https://github.com/apache/infrastructure-asfyaml?tab=readme-ov-file#triage where we could give someone rights to manage issues so in theory someone else could also help with thi

I'm happy to help with this.

ONE question: Do we need to migrate already closed issues/discussions?

[tustvold]

Since object_store is a subproject of arrow-rs, the natural name for it would be apache/arrow-rs-object-store.

I think given it lives under the arrow PMC and not a separate arrow-rs PMC, I think it would be arrow-object-store. This would be consistent with DataFusion which was arrow-datafusion.

I think the overhead of keeping it in the same crate is greater than the value we get from keeping it in the same one

Perhaps we could articulate what these overheads are? There might be ways to alleviate them that require less effort/churn

Do you think we move the object_store crate and associated Dev process (tickets, etc) to its own repository

I don't feel very strongly either way, I think it really depends what the long-term goals for the project are. My vision was for it to parallel the arrow filesystem abstractions present in arrow-cpp / pyarrow. This would naturally entail it residing long-term as part of the arrow project, and being largely developed within that context. With this as the goal splitting it into a separate repository seems like a lot of work and disruption for relatively minor return, if not actively detrimental to that goal. If there is instead an initiative to split it out into its own top-level apache project, splitting it out into a new repository would be a natural next step.

[alamb]

Perhaps we could articulate what these overheads are? There might be ways to alleviate them that require less effort/churn

In my mind it is:

1. The release scripts for arrow / object_store require filtering on tags / are close but not quite the same
2. The tag name convention for object store is annoying
3. When I look at the list of issues in arrow-rs I also see a bunch of object store things which is distracting

We could certainly make this better by making separate release documentation and I could add some filters on tags to my view.

With this as the goal splitting it into a separate repository seems like a lot of work and disruption for relatively minor return, if not actively detrimental to that goal. If there is instead an initiative to split it out into its own top-level apache project, splitting it out into a new repository would be a natural next step.

The "what is the vision of the project" is a great discussion.

I agree if the vision is that object_store will mostly be used by arrow-rs and related technologies (like DataFusion) splitting into a new repo would likely be more detrimental

An alternate vision for object_store is a Rust ecosystem wide library for interacting with object stores, where arrow-rs / DataFusion were one of many downstream projects.

The reverse dependencies at the moment would suggest DataFusion and related projects are the largest users at the moment: https://crates.io/crates/object_store/reverse_dependencies

[crepererum]

I'm not super involved w/ release cutting and admin things of object_store, so this is mostly a user/contributor PoV: it's rather strange that object_store in the arrow repo. I know the historical cause of this, but it is fully independent. Having a dedicated repo would IMHO make the following things easier:

• issue tracking
• google/etc. searches
• community building
• reasoning about the grant picture or niche that this crate tries to fill

However I also understand that an extra repo may come w/ extra overhead.

I think my answer is mostly in line with @alamb.

[alamb]

Another annoyance of mine is that that the commit history is intermixed

Specifically, if you look at https://github.com/apache/arrow-rs/commits/master/

it is not clear which commits modify object_store and which modify arrow

[tustvold]

I think if there is a group of people willing to undertake the work of splitting it out, and to foster, maintain and build a community around said project, I think that is a very exciting path forward. My concern would be that it gets split out without a very clear community around it, and this then hampers its ongoing development and maintenance. I'm especially wary given my reduced capacity going forwards, which would leave the repository with very few active maintainers.

it is not clear which commits modify object_store and which modify arrow

One could make the argument that is what the changelogs are for, but I take your point

[edmondop]

+1

[andygrove]

1. +1 for moving to a new repo
2. the suggested repo name LGTM
3. I don't have bandwdth to help with this, and I am not a regular contributor to arrow-rs or object-store

[alamb]

I think if there is a group of people willing to undertake the work of splitting it out, and to foster, maintain and build a community around said project, I think that is a very exciting path forward.

I am willing to help with this

I'm especially wary given my reduced capacity going forwards, which would leave the repository with very few active maintainers.

I agree this is is a risk.

However, I think it may actually be that moving to a new repo makes it easier to attract new contributors and maintainers (I am thinking about @Xuanwo for example who has been happy to help) -- I think the number of other things in the arrow-rs repo could make the barrier to contribution of object_store higher