Skip to content

Commit 06ccae2

Browse files
alamb2010YOUY01
andauthored
Document copy_array_data function with example (#16361)
* Document `copy_array_data` function with example * Update datafusion/common/src/scalar/mod.rs Co-authored-by: Yongting You <[email protected]> * fixup --------- Co-authored-by: Yongting You <[email protected]>
1 parent a91e042 commit 06ccae2

File tree

1 file changed

+30
-3
lines changed
  • datafusion/common/src/scalar

1 file changed

+30
-3
lines changed

datafusion/common/src/scalar/mod.rs

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3527,9 +3527,36 @@ impl ScalarValue {
35273527
}
35283528
}
35293529

3530-
pub fn copy_array_data(data: &ArrayData) -> ArrayData {
3531-
let mut copy = MutableArrayData::new(vec![&data], true, data.len());
3532-
copy.extend(0, 0, data.len());
3530+
/// Compacts the data of an `ArrayData` into a new `ArrayData`.
3531+
///
3532+
/// This is useful when you want to minimize the memory footprint of an
3533+
/// `ArrayData`. For example, the value returned by [`Array::slice`] still
3534+
/// points at the same underlying data buffers as the original array, which may
3535+
/// hold many more values. Calling `copy_array_data` on the sliced array will
3536+
/// create a new, smaller, `ArrayData` that only contains the data for the
3537+
/// sliced array.
3538+
///
3539+
/// # Example
3540+
/// ```
3541+
/// # use arrow::array::{make_array, Array, Int32Array};
3542+
/// use datafusion_common::scalar::copy_array_data;
3543+
/// let array = Int32Array::from_iter_values(0..8192);
3544+
/// // Take only the first 2 elements
3545+
/// let sliced_array = array.slice(0, 2);
3546+
/// // The memory footprint of `sliced_array` is close to 8192 * 4 bytes
3547+
/// assert_eq!(32864, sliced_array.get_array_memory_size());
3548+
/// // however, we can copy the data to a new `ArrayData`
3549+
/// let new_array = make_array(copy_array_data(&sliced_array.into_data()));
3550+
/// // The memory footprint of `new_array` is now only 2 * 4 bytes
3551+
/// // and overhead:
3552+
/// assert_eq!(160, new_array.get_array_memory_size());
3553+
/// ```
3554+
///
3555+
/// See also [`ScalarValue::compact`] which applies to `ScalarValue` instances
3556+
/// as necessary.
3557+
pub fn copy_array_data(src_data: &ArrayData) -> ArrayData {
3558+
let mut copy = MutableArrayData::new(vec![&src_data], true, src_data.len());
3559+
copy.extend(0, 0, src_data.len());
35333560
copy.freeze()
35343561
}
35353562

0 commit comments

Comments
 (0)