Comparing a `Timestamp` to a `Date32` fails

[crepererum]

Describe the bug
Comparing a Timestamp to a Date32 fails.

To Reproduce
Add the following test to timestamps.slt:

# Test that we can compare a timestamp to a string casted to a date
query C rowsort
select * from foo where ts > '2000-01-01'::date;
----
2 2000-02-01T00:00:00
3 2000-03-01T00:00:00

This fails w/:

[timestamps.slt] Running query: "select * from foo where ts > '2000-01-01'::date;"
Error: SqlLogicTest(query failed: DataFusion error: Internal error: The type of Timestamp(Nanosecond, None) > Date32 of binary physical should be same. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior
It works.

Additional context
Tested on 5c558e9 .

This is likely due to a like in temporal_coercion as well as this workaround:

https://github.com/apache/arrow-datafusion/blob/5c558e9f6b3fe29ef0d5f78ec9465852d020d989/datafusion/optimizer/src/type_coercion.rs#L232-L239

Also see #3419.

[comphead]

@crepererum @waitingkuo I can take that as a next ticket related to timestamp issues

[waitingkuo]

thank you @comphead

do you plan to do Timestamp without Timezone only or both Timestamp and TImestampTz?

[waitingkuo]

fyi

this is straightforward

willy=# select timestamp '2000-01-01T00:00:00.00000' = date '2000-01-01';
 ?column? 
----------
 t
(1 row)

and there're more to consider for timestamptz (i'm in UTC+8)

willy=# select timestamptz '2000-01-01T00:00:00.00000+00' = date '2000-01-01';
 ?column? 
----------
 f
(1 row)

willy=# select timestamptz '1999-12-31T16:00:00.00000+00' = date '2000-01-01';
 ?column? 
----------
 t
(1 row)

willy=# select timestamptz '2000-01-01T00:00:00.00000+08' = date '2000-01-01';
 ?column? 
----------
 t
(1 row)

[comphead]

Simpler reproduce case

❯ select * from (select now() as n) a where n = '2000-01-01'::date;
Internal("The type of Timestamp(Nanosecond, Some(\"+00:00\")) = Date32 of binary physical should be same")

[comphead]

@waitingkuo those ts to date tests, did you run in datafusion cli?
I'm going through some final tests and cannot reproduce your test

❯ select timestamptz '2000-01-01T00:00:00.000000+00' = date '2000-01-01';
Plan("'Timestamp(Nanosecond, Some(\"+00:00\")) = Date32' can't be evaluated because there isn't a common type to coerce the types to")

I'm running it on master branch

[waitingkuo]

@comphead sorry i didn't mention it clearly. it's the behavior for postgresql

[liukun4515]

I want to reopen this issue to discuss the implementation of the comparing between date and timestamp data type.

cc @waitingkuo @alamb @crepererum

I have comments about #4726 (comment)

1. type coercion should be in the right phase, and not be in the physical phase
2. what is the the common type/ result type for comparing timestamp and date

[liukun4515]

Describe the bug Comparing a Timestamp to a Date32 fails.

To Reproduce Add the following test to timestamps.slt:

# Test that we can compare a timestamp to a string casted to a date
query C rowsort
select * from foo where ts > '2000-01-01'::date;
----
2 2000-02-01T00:00:00
3 2000-03-01T00:00:00

This fails w/:

[timestamps.slt] Running query: "select * from foo where ts > '2000-01-01'::date;"
Error: SqlLogicTest(query failed: DataFusion error: Internal error: The type of Timestamp(Nanosecond, None) > Date32 of binary physical should be same. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior It works.

Additional context Tested on 5c558e9 .

This is likely due to a like in temporal_coercion as well as this workaround:

https://github.com/apache/arrow-datafusion/blob/5c558e9f6b3fe29ef0d5f78ec9465852d020d989/datafusion/optimizer/src/type_coercion.rs#L232-L239

This rule will not affect the comparison, we just need to support them in the temporal_coercion

Also see #3419.

[liukun4515]

we need to try to make align with behavior of PG