Consider using `Vec` directly in builders

[alamb]

          I think many of these instances could be changed to use `Vec` directly (and use optimized extend, etc. from them)

Originally posted by @Dandandan in #7309 (comment)

[liamzwbao]

Hi @alamb and @Dandandan, I'm looking into this issue and wanted to clarify what you meant by "using Vec directly." From what I see, the code is already using a Vec, so I assume you're suggesting a refactor like this:

let mut intermediate = Vec::with_capacity(offsets.len() - 1);
for &offset in &offsets[1..] {
    intermediate.push(offset + shift);
}

to use extend

let mut intermediate = Vec::with_capacity(offsets.len() - 1);
intermediate.extend(offsets[1..].iter().map(|&offset| offset + shift));

so that the code is more concise and the performance may improve.

Am I right? Thanks!

[alamb]

Hi @alamb and @Dandandan, I'm looking into this issue and wanted to clarify what you meant by "using Vec directly." From what I see, the code is already using a Vec, so I assume you're suggesting a refactor like this:

let mut intermediate = Vec::with_capacity(offsets.len() - 1);
for &offset in &offsets[1..] {
intermediate.push(offset + shift);
}
to use extend

let mut intermediate = Vec::with_capacity(offsets.len() - 1);
intermediate.extend(offsets[1..].iter().map(|&offset| offset + shift));
so that the code is more concise and the performance may improve.

Am I right? Thanks!

i think @Dandandan was referring to using the ArrayBuilders

So for example, things that use a builder like

let mut builder = Int32Builder::new()
for i in 0..100 {
  builder.append_value(i)
}
let array = builder.build()

Is typically much faster to implement using Vec directly:

let data: Vec<_> = (0..100).collect();
let array = Int32Array::from(data)

This is certainly true for primitive arrays. It is not clear to me it is the same for StringArrays, StringViewArray, etc.

[alamb]

So specifically, for the PR referenced in #7309 (comment)

I think it means changing stuff like

https://github.com/apache/arrow-rs/blob/721150286b00bece40ffcc6f5ac14ebb5d64785b/arrow-array/src/builder/generic_bytes_builder.rs#L33-L32

pub struct GenericByteBuilder<T: ByteArrayType> {
    value_builder: UInt8BufferBuilder,
    offsets_builder: BufferBuilder<T::Offset>,
    null_buffer_builder: NullBufferBuilder,
}

To

pub struct GenericByteBuilder<T: ByteArrayType> {
    value_builder: Vec<u8>,   // Use of a Vec here is likely faster
    offsets_builder: Vec<T::Offset::Native>, // likewise here
    null_buffer_builder: NullBufferBuilder,
}