Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions parquet-variant-compute/benches/variant_kernels.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@ use arrow::array::{Array, ArrayRef, StringArray};
use arrow::util::test_util::seedable_rng;
use criterion::{Criterion, criterion_group, criterion_main};
use parquet_variant::{Variant, VariantBuilder};
use parquet_variant_compute::variant_get::{GetOptions, variant_get};
use parquet_variant_compute::{VariantArray, VariantArrayBuilder, json_to_variant};
use parquet_variant_compute::{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved all the functions into the main parquet_variant_compute module

GetOptions, VariantArray, VariantArrayBuilder, json_to_variant, variant_get,
};
use rand::Rng;
use rand::SeedableRng;
use rand::distr::Alphanumeric;
Expand Down
2 changes: 1 addition & 1 deletion parquet-variant-compute/src/cast_to_variant.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ use arrow_schema::ArrowError;
/// ```
/// # use arrow::array::{Array, ArrayRef, Int64Array};
/// # use parquet_variant::Variant;
/// # use parquet_variant_compute::cast_to_variant::cast_to_variant;
/// # use parquet_variant_compute::cast_to_variant;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise, I removed this extra level of namespacing

/// // input is an Int64Array, which will be cast to a VariantArray
/// let input = Int64Array::from(vec![Some(1), None, Some(3)]);
/// let result = cast_to_variant(&input).unwrap();
Expand Down
17 changes: 11 additions & 6 deletions parquet-variant-compute/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,14 @@
//! ## Main APIs
//! - [`VariantArray`] : Represents an array of `Variant` values.
//! - [`VariantArrayBuilder`]: For building [`VariantArray`]
//! - [`json_to_variant`]: Function to convert a batch of JSON strings to a `VariantArray`.
//! - [`variant_to_json`]: Function to convert a `VariantArray` to a batch of JSON strings.
//! - [`mod@cast_to_variant`]: Module to cast other Arrow arrays to `VariantArray`.
//! - [`variant_get`]: Module to get values from a `VariantArray` using a specified [`VariantPath`]
//!
//! # Compute Kernels
//! - [`json_to_variant()`]: Function to convert Arrays of JSON strings to a `VariantArray`.
//! - [`variant_to_json()`]: Function to convert a `VariantArray` to arrays of JSON strings.
//! - [`cast_to_variant()`]: Cast Arrow arrays to `VariantArray`.
//! - [`variant_get()`]: Convert `VariantArray` (or an inner path) to a strongly-typed Arrow array.
//! - [`shred_variant()`]: Shred a `VariantArray` according to the provided shredding schema
//! - [`unshred_variant()`]: Unshred a `VariantArray` to pure binary variant.
//!
//! ## 🚧 Work In Progress
//!
Expand All @@ -36,15 +40,15 @@
//! [Variant issue]: https://github.com/apache/arrow-rs/issues/6736

mod arrow_to_variant;
pub mod cast_to_variant;
mod cast_to_variant;
mod from_json;
mod shred_variant;
mod to_json;
mod type_conversion;
mod unshred_variant;
mod variant_array;
mod variant_array_builder;
pub mod variant_get;
mod variant_get;
mod variant_to_arrow;

pub use variant_array::{BorrowedShreddingState, ShreddingState, VariantArray, VariantType};
Expand All @@ -56,3 +60,4 @@ pub use shred_variant::shred_variant;
pub use to_json::variant_to_json;
pub use type_conversion::CastOptions;
pub use unshred_variant::unshred_variant;
pub use variant_get::{GetOptions, variant_get};
13 changes: 5 additions & 8 deletions parquet/src/variant.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,11 @@
//! Note: Requires the `variant_experimental` feature of the `parquet` crate to be enabled.
//!
//! # Features
//! * [`Variant`] represents variant value, which can be an object, list, or primitive.
//! * [`VariantBuilder`] for building `Variant` values.
//! * [`VariantArray`] for representing a column of Variant values.
//! * [`json_to_variant`] and [`variant_to_json`] for converting to/from JSON.
//! * [`cast_to_variant()`] for casting other Arrow arrays to `VariantArray`.
//! * [`VariantType`] Arrow ExtensionType for Parquet Variant logical type.
//! [`variant_get`] to extracting a value by path and functions to convert
//! between `Variant` and JSON.
//! * Representation of [`Variant`], and [`VariantArray`] for working with
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was largely a copy of what was in parquet_variant and parquet_variant_compute so I changed the docs to highlight the most important functions and structures and link to the rest of the docs

//! Variant values (see [`parquet_variant`] for more details)
//! * Kernels for working with arrays of Variant values
//! such as conversion between `Variant` and JSON, and shredding/unshredding
//! (see [`parquet_variant_compute`] for more details)
//!
//! # Example: Writing a Parquet file with Variant column
//! ```rust
Expand Down
Loading