Create UTF-8 version of `OsStr`/`OsString` #147932

thaliaarchi · 2025-10-21T04:53:41Z

Implement a UTF-8 version of OsStr/OsString, in addition to the existing bytes and WTF-8 platform-dependent encodings.

This is applicable for several platforms, but I've currently only implemented it for Motor OS:

WASI uses Unicode paths, but currently reexports the Unix bytes-assuming OsStrExt/OsStringExt traits.
- wasi:filesystem APIs:
  
  Paths are passed as interface-type strings, meaning they must consist of a sequence of Unicode Scalar Values (USVs). Some filesystems may contain paths which are not accessible by this API.
- In wasi-filesystem#17, it was decided that applications can use any Unicode transformation format, so we're free to use UTF-8 (and probably already do). This was chosen over specifically UTF-8 or an ad hoc encoding which preserves paths not representable in UTF-8.
  
  The current API uses strings for filesystem paths, which contains sequences of Unicode scalar values (USVs), which applications can work with using strings encoded in UTF-8, UTF-16, or other Unicode encodings.
  
  This does mean that the API is unable to open files which do not have well-formed Unicode encodings, which may want separate APIs for handling such paths or may want something like the arf-strings proposal, but if we need that we should file a new issue for it.
As of Redox OS 0.7.0, "All paths are now required to be UTF-8, and the kernel enforces this". This appears to have been implemented in commit d331f72f (Use UTF-8 for all paths, 2021-02-14). Redox does not have OsStrExt/OsStringExt.
Motor OS guarantees that its OS strings are UTF-8 in its current OsStrExt/OsStringExt traits, but they're still internally bytes like Unix.

This is an alternate approach to #147797, which reuses the existing bytes OsString and relies on the safety properties of from_encoded_bytes_unchecked. Compared to that, this also gains efficiency from propagating the UTF-8 invariant to the whole type, as it never needs to test for UTF-8 validity.

Note that Motor OS currently does not build until #147930 merges.

cc @tgross35 (for earlier review)
cc @alexcrichton, @rylev, @loganek (for WASI)
cc @lasiotus (for Motor OS)
cc @jackpot51 (for Redox OS)

rustbot · 2025-10-21T04:53:46Z

r? @ibraheemdev

rustbot has assigned @ibraheemdev.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

tgross35

Looks pretty good to me, just a few small suggestions.

r? tgross35
@rustbot author

View changes since this review

library/std/src/sys/os_str/utf8.rs

library/std/src/os/motor/ffi.rs

rustbot · 2025-10-21T21:26:10Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

rustbot · 2025-10-21T22:37:21Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

thaliaarchi · 2025-10-21T22:38:56Z

@tgross35 Thanks for the helpful feedback. I've addressed all of your comments.

@rustbot ready

thaliaarchi · 2025-10-21T22:41:07Z

Unless WASI and/or Redox OS platform maintainers speak up here, I am creating separate PRs for those platforms for more focused attention.

tgross35 · 2025-10-21T22:47:25Z

Thanks!

@bors r+ rollup

bors · 2025-10-21T22:47:27Z

📌 Commit 7e2b76e has been approved by tgross35

It is now in the queue for this repository.

Rollup of 7 pull requests Successful merges: - #141445 (Add `FromIterator` impls for `ascii::Char`s to `String`s) - #142339 (Add NonNull pattern types) - #147768 (Code refactoring on hir report_no_match_method_error) - #147788 (const Cell methods) - #147932 (Create UTF-8 version of `OsStr`/`OsString`) - #147933 (os_str: Make platform docs more consistent) - #147948 (PassWrapper: Access GlobalValueSummaryInfo::SummaryList via getter for LLVM 22+) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 7 pull requests Successful merges: - rust-lang/rust#141445 (Add `FromIterator` impls for `ascii::Char`s to `String`s) - rust-lang/rust#142339 (Add NonNull pattern types) - rust-lang/rust#147768 (Code refactoring on hir report_no_match_method_error) - rust-lang/rust#147788 (const Cell methods) - rust-lang/rust#147932 (Create UTF-8 version of `OsStr`/`OsString`) - rust-lang/rust#147933 (os_str: Make platform docs more consistent) - rust-lang/rust#147948 (PassWrapper: Access GlobalValueSummaryInfo::SummaryList via getter for LLVM 22+) r? `@ghost` `@rustbot` modify labels: rollup

Create UTF-8 version of `OsStr`/`OsString` Implement a UTF-8 version of `OsStr`/`OsString`, in addition to the existing bytes and WTF-8 platform-dependent encodings. This is applicable for several platforms, but I've currently only implemented it for Motor OS: - WASI uses Unicode paths, but currently reexports the Unix bytes-assuming `OsStrExt`/`OsStringExt` traits. - [wasi:filesystem](https://wa.dev/wasi:filesystem) APIs: > Paths are passed as interface-type `strings`, meaning they must consist of a sequence of Unicode Scalar Values (USVs). Some filesystems may contain paths which are not accessible by this API. - In [wasi-filesystem#17](WebAssembly/wasi-filesystem#17 (comment)), it was decided that applications can use any Unicode transformation format, so we're free to use UTF-8 (and probably already do). This was chosen over specifically UTF-8 or an ad hoc encoding which preserves paths not representable in UTF-8. > The current API uses strings for filesystem paths, which contains sequences of Unicode scalar values (USVs), which applications can work with using strings encoded in UTF-8, UTF-16, or other Unicode encodings. > > This does mean that the API is unable to open files which do not have well-formed Unicode encodings, which may want separate APIs for handling such paths or may want something like the arf-strings proposal, but if we need that we should file a new issue for it. - As of Redox OS [0.7.0](https://www.redox-os.org/news/release-0.7.0/), "All paths are now required to be UTF-8, and the kernel enforces this". This appears to have been implemented in commit [d331f72f](https://gitlab.redox-os.org/redox-os/kernel/-/commit/d331f72f2a51fa577072f24bc2587829fd87368b) (Use UTF-8 for all paths, 2021-02-14). Redox does not have `OsStrExt`/`OsStringExt`. - Motor OS guarantees that its OS strings are UTF-8 in its [current `OsStrExt`/`OsStringExt` traits](https://github.com/moturus/rust/blob/a828ffcf5f04be5cdd91b5fad608102eabc17ec7/library/std/src/os/motor/ffi.rs), but they're still internally bytes like Unix. This is an alternate approach to rust-lang#147797, which reuses the existing bytes `OsString` and relies on the safety properties of `from_encoded_bytes_unchecked`. Compared to that, this also gains efficiency from propagating the UTF-8 invariant to the whole type, as it never needs to test for UTF-8 validity. Note that Motor OS currently does not build until rust-lang#147930 merges. cc `@tgross35` (for earlier review) cc `@alexcrichton,` `@rylev,` `@loganek` (for WASI) cc `@lasiotus` (for Motor OS) cc `@jackpot51` (for Redox OS)

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#141445 (Add `FromIterator` impls for `ascii::Char`s to `String`s) - rust-lang#142339 (Add NonNull pattern types) - rust-lang#147768 (Code refactoring on hir report_no_match_method_error) - rust-lang#147788 (const Cell methods) - rust-lang#147932 (Create UTF-8 version of `OsStr`/`OsString`) - rust-lang#147933 (os_str: Make platform docs more consistent) - rust-lang#147948 (PassWrapper: Access GlobalValueSummaryInfo::SummaryList via getter for LLVM 22+) r? `@ghost` `@rustbot` modify labels: rollup

Create UTF-8 version of `OsStr`/`OsString` Implement a UTF-8 version of `OsStr`/`OsString`, in addition to the existing bytes and WTF-8 platform-dependent encodings. This is applicable for several platforms, but I've currently only implemented it for Motor OS: - WASI uses Unicode paths, but currently reexports the Unix bytes-assuming `OsStrExt`/`OsStringExt` traits. - [wasi:filesystem](https://wa.dev/wasi:filesystem) APIs: > Paths are passed as interface-type `strings`, meaning they must consist of a sequence of Unicode Scalar Values (USVs). Some filesystems may contain paths which are not accessible by this API. - In [wasi-filesystem#17](WebAssembly/wasi-filesystem#17 (comment)), it was decided that applications can use any Unicode transformation format, so we're free to use UTF-8 (and probably already do). This was chosen over specifically UTF-8 or an ad hoc encoding which preserves paths not representable in UTF-8. > The current API uses strings for filesystem paths, which contains sequences of Unicode scalar values (USVs), which applications can work with using strings encoded in UTF-8, UTF-16, or other Unicode encodings. > > This does mean that the API is unable to open files which do not have well-formed Unicode encodings, which may want separate APIs for handling such paths or may want something like the arf-strings proposal, but if we need that we should file a new issue for it. - As of Redox OS [0.7.0](https://www.redox-os.org/news/release-0.7.0/), "All paths are now required to be UTF-8, and the kernel enforces this". This appears to have been implemented in commit [d331f72f](https://gitlab.redox-os.org/redox-os/kernel/-/commit/d331f72f2a51fa577072f24bc2587829fd87368b) (Use UTF-8 for all paths, 2021-02-14). Redox does not have `OsStrExt`/`OsStringExt`. - Motor OS guarantees that its OS strings are UTF-8 in its [current `OsStrExt`/`OsStringExt` traits](https://github.com/moturus/rust/blob/a828ffcf5f04be5cdd91b5fad608102eabc17ec7/library/std/src/os/motor/ffi.rs), but they're still internally bytes like Unix. This is an alternate approach to rust-lang#147797, which reuses the existing bytes `OsString` and relies on the safety properties of `from_encoded_bytes_unchecked`. Compared to that, this also gains efficiency from propagating the UTF-8 invariant to the whole type, as it never needs to test for UTF-8 validity. Note that Motor OS currently does not build until rust-lang#147930 merges. cc `@tgross35` (for earlier review) cc `@alexcrichton,` `@rylev,` `@loganek` (for WASI) cc `@lasiotus` (for Motor OS) cc `@jackpot51` (for Redox OS)

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#141445 (Add `FromIterator` impls for `ascii::Char`s to `String`s) - rust-lang#142339 (Add NonNull pattern types) - rust-lang#147768 (Code refactoring on hir report_no_match_method_error) - rust-lang#147788 (const Cell methods) - rust-lang#147932 (Create UTF-8 version of `OsStr`/`OsString`) - rust-lang#147933 (os_str: Make platform docs more consistent) - rust-lang#147948 (PassWrapper: Access GlobalValueSummaryInfo::SummaryList via getter for LLVM 22+) r? `@ghost` `@rustbot` modify labels: rollup

rustbot assigned ibraheemdev Oct 21, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Oct 21, 2025

tgross35 reviewed Oct 21, 2025

View reviewed changes

rustbot assigned tgross35 and unassigned ibraheemdev Oct 21, 2025

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2025

thaliaarchi added 2 commits October 21, 2025 16:29

os_str: Create UTF-8 platform encoding

206ffcc

motor: Use UTF-8 guarantee for OS strings

7e2b76e

thaliaarchi force-pushed the utf8-osstring branch from 087647c to 7e2b76e Compare October 21, 2025 22:37

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 21, 2025

tgross35 approved these changes Oct 21, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2025

This was referenced Oct 21, 2025

Tracking Issue for motor_ext #147456

Open

motor: Use UTF-8 guarantee for OS strings #147797

Closed

matthiaskrgr mentioned this pull request Oct 22, 2025

Rollup of 7 pull requests #147957

Merged

bors merged commit aa65c31 into rust-lang:master Oct 22, 2025
11 checks passed

rustbot added this to the 1.92.0 milestone Oct 22, 2025

thaliaarchi deleted the utf8-osstring branch October 22, 2025 20:35

thaliaarchi mentioned this pull request Oct 23, 2025

Unify OsStringExt/OsStrExt traits across platforms #148050

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Create UTF-8 version of `OsStr`/`OsString` #147932

Create UTF-8 version of `OsStr`/`OsString` #147932

Uh oh!

thaliaarchi commented Oct 21, 2025

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

tgross35 left a comment •

edited by rustbot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

thaliaarchi commented Oct 21, 2025

Uh oh!

thaliaarchi commented Oct 21, 2025

Uh oh!

tgross35 commented Oct 21, 2025

Uh oh!

bors commented Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Create UTF-8 version of OsStr/OsString #147932

Create UTF-8 version of OsStr/OsString #147932

Uh oh!

Conversation

thaliaarchi commented Oct 21, 2025

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

tgross35 left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

rustbot commented Oct 21, 2025

Uh oh!

thaliaarchi commented Oct 21, 2025

Uh oh!

thaliaarchi commented Oct 21, 2025

Uh oh!

tgross35 commented Oct 21, 2025

Uh oh!

bors commented Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Create UTF-8 version of `OsStr`/`OsString` #147932

Create UTF-8 version of `OsStr`/`OsString` #147932

tgross35 left a comment •

edited by rustbot

Loading