Support Time Parquet Data Page Statistics #11187
Conversation
| [<$stat_type_prefix Int32DataPageStatsIterator>]::new($iterator).flatten(), | ||
| )), | ||
| _ => { | ||
| let len = $iterator.count(); |
There was a problem hiding this comment.
I have referenced get_statistics macro to decide on returing a null array, my other idea was to conclued with unimplemented!().
There was a problem hiding this comment.
You beat me to it ;-).
I think this can just be:
_ => new_empty_array(&DataType::Time32(unit.clone())),And then you need to add new_empty_array into the use statement. Time64 gets a similar treatment.
There was a problem hiding this comment.
Much cleaner. Thanks for the idea @efredine
| )), | ||
| _ => { | ||
| // don't know how to extract statistics, so return an empty array | ||
| new_empty_array(&DataType::Time32(unit.clone())) |
There was a problem hiding this comment.
Technically speaking I think Time32 must be either Seconds or Milliseconds otherwise it is not a valid type.
https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#variant.Time32
However, the arrow-rs library doesn't encode this in the types so I think this is a reasonable behavior
* add parquet page stats for time * return empty array instead of null * fix typos
3 participants
Which issue does this PR close?
Closes #11114
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Covered by existing test cases
Are there any user-facing changes?
No