ObjectStore WASM32 Support #7226

tustvold · 2025-03-03T10:57:23Z

Which issue does this PR close?

Closes #6818.

Rationale for this change

This allows building object_store with all features enabled on wasm32-wasip1.

It is worth noting, providing an HttpConnector and by extension HttpClient is an exercise for the user, but I could see this changing - it just needs someone to go away and figure out how best to do this (#7227). I am also aware that wasm32-wasip2 adds proper socket support, and so there is the possibility it could use a more "normal" HTTP client eventually.

What changes are included in this PR?

Some feature gating to make it possible to enable the cloud features on WASM platforms.

Are there any user-facing changes?

FYI @kylebarron

.github/workflows/object_store.yml

tustvold · 2025-03-03T11:10:33Z

.github/workflows/object_store.yml

        run: cargo build --target wasm32-unknown-unknown
      - name: Build wasm32-wasip1
-        run: cargo build --target wasm32-wasip1
+        run: cargo build --all-features --target wasm32-wasip1


alamb · 2025-03-03T13:48:57Z

I will review this shortyl

alamb

Makes sense to me -- thank you @tustvold

It is somewhat unfortunate that we need to sprinkle cfgs around, but I don't see any way around this and I think the new CI checks should prevent regressions

    #[cfg(not(target_arch = "wasm32"))]

alamb · 2025-03-03T13:50:17Z

object_store/src/client/connection.rs

+    match custom {
+        Some(x) => Ok(x),
+        None => Err(crate::Error::NotSupported {
+            source: "WASM32 architectures must provide an HTTPConnector"


At some point it would be great to have an example of implementing WASM / a documentation guide for how to do it

I think it would be good to even go as far as providing first-party support, but I don't have time to figure out the complexities involved.

#7227 is my attempt to fish for contributions

crepererum · 2025-03-03T15:24:55Z

I am also aware that wasm32-wasip2 adds proper socket support, and so there is the possibility it could use a more "normal" HTTP client eventually.

So why are we bothering w/ wasip1 at all? It's outdated (not just the APIs that we can use, but also how it works under the hood), it's unnecessarily limiting, and I would argue that if you wanna use WASI, at least you the 0.2 preview version.

kylebarron · 2025-03-03T15:35:09Z

Does this compile on wasm32-unknown-unknown as well, or just wasm32-wasip1?

cc'ing also @H-Plus-Time who made an ObjectStore implementation for wasm32-unknown-unknown here: https://github.com/H-Plus-Time/object-store-wasm

tustvold · 2025-03-03T16:28:24Z

Does this compile on wasm32-unknown-unknown as well, or just wasm32-wasip1?

It should do but requires some shenanigans with getrandom - https://docs.rs/getrandom/latest/getrandom/#webassembly-support

So why are we bothering w/ wasip1 at all?

My understanding is wasip2 is not properly standardised yet, and there are real platforms that support wasip1 and that people might want to target.

That being said I am really just using wasip1 as it is a well-defined WASM32 target, see above for the challenges of wasm32-unknown-unknown.

Ultimately this PR is to allow people to support wasm32 targets at all, it is then up to the user to provide an HttpConnector that exploits whatever APIs are available in their particular WASM32 flavour.

crepererum · 2025-03-03T16:30:59Z

My understanding is wasip2 is not properly standardised yet

They are BOTH standardized equally: as previews. See https://wasi.dev/

tustvold · 2025-03-03T16:33:11Z

My understanding is wasip2 is not properly standardised yet

I'm not sure what you are suggesting here, are you saying we should not support wasip1, we should add a test showing we support wasip2 (which seems redundant given it is a superset), something else?

This PR is not about supporting particular WASM32 flavours, it is about supporting WASM32 at all...

crepererum · 2025-03-03T16:41:53Z

My understanding is wasip2 is not properly standardised yet

I'm not sure what you are suggesting here, are you saying we should not support wasip1, we should add a test showing we support wasip2 (which seems redundant given it is a superset), something else?

This PR is not about supporting particular WASM32 flavours, it is about supporting WASM32 at all...

The CI specifically cares about WASIp1, not about p2. And as you said: plain WASM32 is gonna be a pain, because you to get random numbers, it's a side effect that the host needs to support (either via JS or via WASI). So I think my questions are:

do we need random numbers at all?
if we need any side effects, then I think we should settle on WASIp2.

tustvold · 2025-03-03T18:14:39Z

The CI specifically cares about WASIp1, not about p2.

This is fair, although this is simply because it predates wasmp2 existing, rather than any particular design decision.

do we need random numbers at all?

Yes for retries and crypto

I think we should settle on WASIp2.

At least at the time of writing wasip2 is explicitly labelled an experimental target here, whereas the same language is not used for wasip1.

As such, and because wasip1 is closer to the browser target I suspect 90% or more of users are actually interested in, I think we should leave the CI unchanged. Although I don't feel strongly on this matter, my main goal with this PR is just to make WASM32 possible, I don't intend to personally work on wasm32 support beyond that.

crepererum · 2025-03-04T12:24:57Z

At least at the time of writing wasip2 is explicitly labelled an experimental target here, whereas the same language is not used for wasip1.

That's a good point. I find that phrasing somewhat confusing given that both are tier 2 targets, so I've opened a thread in the community forums: https://users.rust-lang.org/t/is-wasm32-wasip2-experimental/126541

H-Plus-Time · 2025-03-05T16:21:39Z

Just an FYI on this one, especially with the 0.12.0 release imminent: anything that uses the RetryExt (all the remote ObjectStore implementations, including http::client::Client) will trigger a panic on the first request in wasm32-unknown-unknown and emscripten (effectively neutering http_connector). The following, in the object_store crate (plainly, this can't be worked around downstream), avoids this:

// Cargo.toml
[target.'cfg(target_arch = "wasm32")'.dependencies]
web-time = { version = "1.1.0" }

// src/client/retry.rs
#[cfg(not(target_arch = "wasm32"))]
use std::time::{Duration, Instant};
#[cfg(target_arch = "wasm32")]
use web_time::{Duration, Instant};

tustvold · 2025-03-05T16:30:08Z

Good to know, unless the fix would require a breaking change I'd be inclined to not hold the release for this.

I also think WASI platforms are not impacted by this.

I would be very keen for volunteers if someone wanted to better flesh out the WASM32 support #7227 apache/arrow-rs-object-store#26

H-Plus-Time · 2025-03-05T16:37:48Z

Yep, I'll fold it in with my PR for #7227.

kylebarron · 2025-03-05T17:37:29Z

@H-Plus-Time would the fix require a breaking change? The next breaking object_store release won't be for a while I think

If it doesn't need a breaking change (based on my read above, it doesn't?) then I agree we can get this fixed in the next patch release

H-Plus-Time · 2025-03-05T22:06:24Z

Provided additional dependencies are not treated as breaking (surely not), yes, a patch release is fine.

* ObjectStore WASM32 Support * Docs * Update .github/workflows/object_store.yml

ObjectStore WASM32 Support

871e29d

github-actions bot added the object-store label Mar 3, 2025

tustvold mentioned this pull request Mar 3, 2025

Support object_store with wasm: Default WASM32 HttpConnector #7227

Closed

Docs

569e63d

tustvold commented Mar 3, 2025

View reviewed changes

.github/workflows/object_store.yml Outdated Show resolved Hide resolved

Update .github/workflows/object_store.yml

149ca7b

tustvold mentioned this pull request Mar 3, 2025

Release object_store 0.12.0 (API breaking) Around Feb 30 2025 apache/arrow-rs-object-store#28

Closed

8 tasks

tustvold commented Mar 3, 2025

View reviewed changes

tustvold requested review from alamb and crepererum March 3, 2025 13:34

alamb mentioned this pull request Mar 3, 2025

Weekly Plan (Andrew Lamb) March 3, 2025 apache/datafusion#14978

Closed

12 tasks

alamb approved these changes Mar 3, 2025

View reviewed changes

tustvold merged commit 3674073 into apache:main Mar 3, 2025
33 checks passed

alamb pushed a commit to alamb/arrow-rs that referenced this pull request Mar 20, 2025

ObjectStore WASM32 Support (apache#7226)

e975177

* ObjectStore WASM32 Support * Docs * Update .github/workflows/object_store.yml

PinkCrow007 pushed a commit to PinkCrow007/arrow-rs that referenced this pull request Mar 20, 2025

ObjectStore WASM32 Support (apache#7226)

0078827

* ObjectStore WASM32 Support * Docs * Update .github/workflows/object_store.yml

ObjectStore WASM32 Support #7226

ObjectStore WASM32 Support #7226

Uh oh!

Conversation

tustvold commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Uh oh!

tustvold Mar 3, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 3, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Mar 3, 2025

Choose a reason for hiding this comment

Uh oh!

tustvold Mar 3, 2025

Choose a reason for hiding this comment

Uh oh!

crepererum commented Mar 3, 2025

Uh oh!

kylebarron commented Mar 3, 2025

Uh oh!

tustvold commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crepererum commented Mar 3, 2025

Uh oh!

tustvold commented Mar 3, 2025

Uh oh!

crepererum commented Mar 3, 2025

Uh oh!

tustvold commented Mar 3, 2025

Uh oh!

Uh oh!

crepererum commented Mar 4, 2025

Uh oh!

H-Plus-Time commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tustvold commented Mar 5, 2025

Uh oh!

H-Plus-Time commented Mar 5, 2025

Uh oh!

kylebarron commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

H-Plus-Time commented Mar 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tustvold commented Mar 3, 2025 •

edited

Loading

tustvold commented Mar 3, 2025 •

edited

Loading

H-Plus-Time commented Mar 5, 2025 •

edited

Loading

kylebarron commented Mar 5, 2025 •

edited

Loading