Question about the `map_varchar_to_utf8view` config

[xudong963]

DataFusion CLI v48.0.0
> create table t1(a varchar, b char);
0 row(s) fetched. 
Elapsed 0.009 seconds.

> show columns from t1;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion    | public       | t1         | a           | Utf8View  | YES         |
| datafusion    | public       | t1         | b           | Utf8      | YES         |
+---------------+--------------+------------+-------------+-----------+-------------+
2 row(s) fetched. 
Elapsed 0.012 seconds.

Currently, map_varchar_to_utf8view is enabled by default, I noticed char will be mapped to Utf8 in nature, so if map_varchar_to_utf8view is true, should we also map char to Utf8View

FYI: I found the issue while upgrading DF48 for mv repo: https://github.com/datafusion-contrib/datafusion-materialized-views/pull/61/files#r2131717796

[xudong963]

cc @zhuqi-lucas @alamb

[zhuqi-lucas]

Good question @xudong963 .

I agree that we can map both CHAR and VARCHAR to Utf8View. In DataFusion, both types are ultimately represented as Arrow string arrays, so using Utf8View avoids a full copy for either one. The only semantic difference is that, per SQL standards, CHAR(n) is fixed-length (right-padded with spaces or truncated to exactly n characters) while VARCHAR(n) is variable length (0–n characters). I am not sure if it has some issues about the mapping for CHAR(n), otherwise, treating both CHAR and VARCHAR as Utf8View in the planner/executor is fine and incurs no extra runtime cost.

[xudong963]

The only semantic difference is that, per SQL standards, CHAR(n) is fixed-length (right-padded with spaces or truncated to exactly n characters) while VARCHAR(n) is variable length (0–n characters).

The char will be mapped to varchar currently(as the example in the issue shows), so I guess there won't be an issue to map it to Utf8View

[zhuqi-lucas]

Updated, the current code mapping Char/Text/String to Utf8, so we should change all to make it consistent?

SQLDataType::Char(_) | SQLDataType::Text | SQLDataType::String(_) => {
          Ok(DataType::Utf8)
}

And also fix the related tests.

[xudong963]

I think so, if map_varchar_to_utf8view is true, the code should be:

SQLDataType::Char(_) | SQLDataType::Text | SQLDataType::String(_) => {
          Ok(DataType::Utf8view)
}

[zhuqi-lucas]

I start the work in:

#16290

It is similar to

#15096

But most work we have done, so it's easier for us to finish this ticket.

[zhuqi-lucas]

@alamb @xudong963 I think we may finish it at next release, i found some additional things need to be done if we change to

SQLDataType::Char(_) | SQLDataType::Text | SQLDataType::String(_) => {
          Ok(DataType::Utf8view)
}

I am in progress, or we can wait some time for this release?

[xudong963]

No rush, it’s okay to include it in next round

[alamb]

Thanks @xudong963 and @zhuqi-lucas -- this makes sense to me

[alamb]

It seems we are tracking the work in

• Mapping Char/Text/String default to Utf8View #16288

So perhaps we can close this ticket