Remove `dict_id` from `arrow_schema::field::Field` and make dictionary IDs an internal implementation detail of flight encoding/decoding

[thinkharderdev]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently the dict_id field is only used for the purposes of arrow flight encoding/decoding so dictionaries can be mapped to there associated fields.

This is annoying and error-prone as the user is left the responsibility of assigning these dictionary IDs and ensuring that they are unique.

#5971 adds the option to auto-assign dictionary IDs during arrow flight encoding. This can be enabled by setting the preserve_dict_id option in IpcWriteOptions to false (current default is true

Describe the solution you'd like

This can be done in stages but ultimately would like to

1. Make preserve_dict_id default to false
2. Remove the preserve_dict_id option altogether
3. Remove the dict_id field from arrow_schema::schema::Field entirely as it no longer has any purpose

Describe alternatives you've considered

We can leave this is as a configurable option and either only do 1 above or we can leave auto-assigning of dictionary IDs as an opt-in feature

Additional context

[alamb]

I think this would be a nice thing to do -- we have talked about it for a long time.

I think the dict_id field in the schema is left over from very very early versions of arrow-rs

[brancz]

Now that dict IDs are automatically assigned, how about we create a new type eg. FlightSchema and FlightField in the flight crate that are created in prepare_schema_for_flight and wrap the canonical Schema and Field? Then we can remove dict_id from Field.

[thinkharderdev]

Now that dict IDs are automatically assigned, how about we create a new type eg. FlightSchema and FlightField in the flight crate that are created in prepare_schema_for_flight and wrap the canonical Schema and Field? Then we can remove dict_id from Field.

Seems reasonable. We still need to do step 1 and 2 though. I realize now that we missed the 53.0.0 release so we can't do that until December and the next major release since it is a breaking change

[brancz]

Now that dict IDs are automatically assigned, how about we create a new type eg. FlightSchema and FlightField in the flight crate that are created in prepare_schema_for_flight and wrap the canonical Schema and Field? Then we can remove dict_id from Field.

After trying to apply this change I realize it's actually entirely useless since like you said, after 1 and 2 there won't be any use for dict_id anymore. Since the dict ID is just set in the prepare_schema_for_flight function on the dict tracker directly.

[brancz]

Once #6711 merges, we can change the default to not preserving dict IDs.

Then the next breaking release we remove the ability to set dict IDs in the first place?

[brancz]

Here is the follow up: #6788

[alamb]

See also

• Add deprecation warnings for everything related to dict_id #6873

[andygrove]

Currently the dict_id field is only used for the purposes of arrow flight encoding/decoding so dictionaries can be mapped to there associated fields.

This is not entirely true. It is part of Arrow's public API and is used by downstream systems such as Comet.