Context for the move of Arrow JS and version policy

[rajsite]

1. I see the issue and PR that moved the Arrow JS implementation out of the arrow monorepo. Is there discussion for that direction change one can reference?
2. We use arrow for interoperability from .NET C# (Blazor) and JS and just relied on the monorepo version alignment to not worry about compatibility. Now it's possible to be out of sync (arrow js is currently v21 while arrow dotnet is v20).
   • With the change is there a policy on versioning and compatibility between monorepo and arrow-js?
   • Arrow has Format Versions which I think is supposed to address this concern but I can't find documentation of what format version libraries are supporting. Is that what we should rely on now that we can't rely on monorepo versioning and how would we do that?

[westonpace]

AFAIK, transmission between C# and JS would have had to go through the IPC format. This is covered by the IPC format version. In other words, you are no worse off, in terms of compatibility, then you were before.

As it says on the page you linked:

When these new features are not used, the new minor format versions are compatible with format version 1.0.0.

So as long as the two implementations support the feature they should be able to communicate. This page documents implementation status.

So, for example, if you have decimal32 arrays in C# you will not be able to deserialize them in JS. This was true even when the two libraries were in the monorepo and versioned together (just because they were the same version did not mean they had the same completeness of implementation)

[westonpace]

The only real exceptions were the implementations that were merely bindings on top of C++ (and these are all still in the monorepo). For example, pyarrow and arrow-cpp are more tightly linked and can transmit arrays without going through the C data interface or the IPC interface. JS never behaved this way.

[amoeba]

@westonpace beat me to answer to the compatibility question. Well-said!

For context on the move, the discussion started out on the mailing list, see [DISCUSS] Split JS release process. It was also posted as a GitHub Discussion but the mailing list is where decision-making happens on Arrow (and other Apache projects).

[pitrou]

Also, the Arrow in-memory format has been stable since version 1.0. New datatypes have been added, but unless you're using a datatype that one of your Arrow implementations of choice doesn't support, you'll be good.