Discussion about an implicit (or explicit) "node" scheme

[SMotaal]

This compliments #168 with a more open discussion about the notion of assuming an implicit (or explicit) node scheme which builds on the existing internal/implicit use of node: in experimental-modules.

This thread opens the discussion about the notion that a theoretical (or experimental) Node.js ESM loader can be considered the one and only implementation for a node+file: scheme.

Other schemes which are not supported by such a loader (like node+https:) can still be valid schemes that live in the realm of other platforms like Electron, or handled by third-party handlers that may or may not interface with node's own loader.

[SMotaal]

@jkrems

Would a protocol like commonjs:// be more clear?

Absolutely, which is more ambiguously than node+commonjs: (dropping node+ since it is always within the node scheme. Funny enough commonjs: is not relevant for CommonJS modules since the only time a specifier is actually a file: URL (not an OS-specific path) is "theoretically" when importing into ESM or when using import(…) instead of require(…) in cjs.

The disadvantage of a protocol solution (last time I tried it) was that it becomes weird for non-absolute things. commonjs://./foo.node looks really weird and isn't quite how "normal" URLs would work.

Certainly, those are merely conceptual schemes for design purposes (and possibly internal resolution algorithms). The only time you would need to explicitly refer to commonjs://./foo.node is when commonjs: is defined in isolation. Within Node (or electron) the idea that an implied node scheme exists (which predicates that an explicit referrer (an absolute URI for the requesting module) along with any absolute or relative specifiers, the main module maybe being the one exception. This means that it resolves to commonjs://./foo.js only internally, which defers to the commonjs loader that converts this file-like URI to a OS-specific file-system path

[devsnek]

strong -1 on commonjs: or node+commonjs or whatever else... schemes refer to the protocol used to access a resource, not the type of the resource itself. it would be a misuse of urls to do so.

[SMotaal]

@devsnek that commonjs: is not a protocol nor a scheme per say, but within the context of a URI-based module system (like ESM) that can theoretically load a commonjs module through an interface with that system, such specifiers must assume a "non-standard" scheme which takes the form of a URI that makes it possible to infer concrete and consistent parameters that will result in exactly one "wrapped" module instance throughout the runtime span

[ljharb]

I don't think it's likely that import will work in node with any schemes, including http/https, and i don't even think it should work with file:// - unless we have consensus on those, I don't think it makes sense to add a new scheme.

[SMotaal]

@ljharb so I've thought about this for a while and wanted to push this discussion further but with more clarity.

Assume for a moment that I not proposing that import … from 'node:…' or import('node:…') would be handled by node, I am simply saying that for a theoretical third-party implementation (ie bundlers, renderers... etc) parsing an ESM file that is somehow identified as a "node" ESM file (maybe MIME or not) a bare specifier will need to be associated with a scheme, at the very least to construct a URL relative to a base or referrer.

So this whole direction of the discussion about supporting or not protocols in node files has nothing to do with my intent. Let's try extra hard to not let hot topics overshadow seemingly related issues which can be very critical.

Can you clarify if you are just against the node resolving part (which is not the intent) and if not, are you against specifiers being URLs in general? I am just trying to relay my own thinking to steer the conversation in a more productive path.

[ljharb]

Although anyone will be able to use loaders and transpilers to use any specifier they want - I’m against node having specifiers work by default. Every path in node is file:// and so does not need an explicit protocol, http/https URLs would be dangerous imo (altho these might end up being acceptable), and i don’t see the benefit of adding additional protocols. A protocol talks about how you access the format - but not what the format is. What access method needs disambiguation here?

(In addition, I’m firmly against encoding the module format anywhere but in the module itself - either in its contents (altho this won’t work for JS scripts vs modules), in its filename (ie, it’s extension), or in “closest package.json” metadata that overrides one of those two options)

[SMotaal]

http/https URLs would be dangerous imo
I’m firmly against encoding the module format

Again, that is all besides the points I am trying to raise. And as far as node is concerned, I don't think I disagree with your logic.

Every path in node is file:// and so does not need an explicit protocol

This imho is where I might be able to articulate my thinking better. My version of this is more like "while every path in node is file:// and so does not need an explicit protocol" not every specifier is a path.

A protocol talks about how you access the format - but not what the format is. What access method needs disambiguation here?

For instance, bare specifiers are not a path nor a URL until they are resolved by node's ESM subsystem to a file in some package. More importantly though, bare specifiers that resolve to a dynamic module instance, including at the very least every single builtin module, not to mention all non-esm if you consider the --experimental-modules implementation, those are not paths, but they are resources, just like blobs are resources, and they too have a scheme.

But here is where this all matters from my point of view, and @demurgos's example is spot on, we should not just be focusing on specifiers from the resolution aspect (not touching this one), we should also consider how they interoperate at the very least inside the parent runtime, which happens to be v8/chrome+webkit, possible chakra... and so on. This is just conceptual, not implementational.

[ljharb]

I could certainly see the internal - not user-visible - use of an implicit node:// protocol on bare specifiers, just like resolved file paths might have file:// and source text modules might have another protocol. However, i wouldn’t want users to be able by default to have two ways to represent a specifier to the same module (noting that ./foo and ../src/foo might refer to the same module, but I’d still consider these the “same way”). In other word, i don’t want such a user-visible signal, as mentioned in the linked comment.

[SMotaal]

i wouldn’t want users to be able by default to have two ways to represent a specifier to the same module (noting that ./foo and ../src/foo …

absolutely, my own thinking too, the source text specifier node:foo would be invalid (unless maybe a custom loader wants to play with that) and assuming it makes it somehow to the module wrap, the parent runtime will see it node:[…]node:foo and this will be an issue for the custom loader(s) and never a node thing to address.

An instantiated module "node:process" in an electron app for instance, is only accessible by an ES module when the source text is requiring or importing "process". It cannot be accessed from a <script type=module> unless you register a custom scheme "node:" and somehow map it to the module instance for that same module wrap (this last part is secondary to the issue at hand)

[devsnek]

https://github.com/nodejs/node/blob/317b894b122e2c45499d6a0032adc27d85f0dd99/lib/internal/modules/esm/loader.js#L77-L78

[SMotaal]

@devsnek I know, I blame you for all this discussion. If I did not see this, I think I would have remained relatively ignorant of the notion of the ModuleWrap's src and url and how they interplay with v8's Modules...

My concerns are with all other non-path ModuleWraps (dynamic modules like cjs) and how a simple spinoff on "node:builtins" which do not factor into node's ESM resolution at all, could potentially prevent conflicting coercions or assumptions in other realms of node's own ecosystem (for instance Electron & NW.js)