New testsuite organization

[andreaTP]

Hello 👋

I’m following up here as suggested in this conversation.

Before the PR landed, we didn’t notice that a new testsuite folder structure was introduced.
As runtime developers, we now face a challenge: it’s much harder to implement proposals one at a time.

I’m looking for guidance on how to proceed and what to expect going forward.

In chicory, we generate unit tests based on .wast files.
Previously, we relied on the folder structure to include only the proposals implemented at each stage.
Now that everything is under wasm-3.0, we either need a way to separate files by proposal or provide at least a minimal implementation of all proposals opcodes upfront just to get parsing to pass.

I also noticed that in some cases (e.g. binary.wast and binary0.wast) files have been duplicated.
Would it be an acceptable middle ground to keep proposals separated at the file level and use more descriptive names, such as binary_eh.wast, to indicate which proposal they cover?

Thanks!

[rossberg]

The wasm-3.0 branch (which will be merged into main very soon in one form or the other) doesn't actually change the test structure, it merely has additional proposals merged. Some are in their own directory, some aren't. Unfortunately, the CG did not create any guidelines regarding that, so it was left to the proposal champions to organise themselves. It would make sense to me, though, to e.g. move all tests related to exception handling into a new subdirectory.

In general, however, it is unrealistic that we can divide everything strictly by proposal — for example, because there are tests that only live in the cross product of multiple proposals. Moreover, proposals are ultimately just historical accident and less relevant or useful as a structuring mechanism for future implementers wanting to implement a Wasm engine from scratch. Structuring by proposal in fact makes it much more difficult to find all tests for a given instruction, for example (one reason why I'm not happy with the duplicated tests under multi-memory).

For any incremental implementation work you're gonna need a more fine-grained mechanism anyway. In the past we have briefly discussed some ways of naming or marking tests with annotations, but that discussion didn't go far. Maybe it's time to restart such a discussion.

[bvisness]

It might be possible to add "guards" to the wast, e.g. (if-supports "multi-memory" (module ...)). In SpiderMonkey we generate JS from the wast, and then patch many of the JS files afterwards to add e.g. if (wasmMultiMemorySupported()) around anything that can fail in certain modes.

I think this approach might scale better than organizing by files, since the overlap of proposals makes it all but impossible to organize all the tests into a nice tree.

[rossberg]

I moved exceptions and memory64 tests to their own directory in #1951.

[andreaTP]

 it is unrealistic that we can divide everything strictly by proposal

Right, and we have mechanisms in place to exclude even single assertions, but having cross cutting concerns, for example, isolated in fewer files(as opposed to possibly everywhere) is still immensely helpful.

proposals are ultimately just historical accident

This is absolutely right, but, from a pragmatic point of view that's the only viable separation we can leverage as of today.

It might not be optimal, correct or anything, but makes it possible to easily split large chunks of work while working on a runtime.

Unless we have an(any!) alternative, I'm worried that giving up on this point is going to, just, significantly rise the bar for new runtime to be implemented.

At this historical point, and in absence of real alternatives, I believe that "proposals" are a "good enough" way to draw lines and ease implementors.

At the end of the day, they are single bodies of work from the spec standpoint and the connection clearly shows.

The fact that it maps 1:1 the evolution of the reference interpreter has also demonstrated to be very helpful during implementation.

For any incremental implementation work you're gonna need a more fine-grained mechanism anyway. 

Again, very correct, but we are(often) able to deal with the real world intricacies given a baseline.

I'm worried to lose an "emerged" baseline while aiming for a perfect solution.
On this particular aspect I'd favor any quick and dirty pragmatic approach over long term plans.

[rossberg]

@bvisness, guards would make sense to me, though since it's meta-information, I'd use annotation syntax for that: (@if-supports "multi-memory") (module ...).

But yeah, marking up the entire test suite would be a substantial effort. Somebody would have to volunteer to do that.

@andreaTP, the current organisation structure is primarily by construct, although there are historical exceptions to that. If we could clean up some of the problematic exceptions, then I think that's an even better and more flexible approximation. Since most constructs can be grouped by proposal, it should at least be no worse.

The tricky bits are the cross-cutting extensions, as you mention. These are primarily things like multi-value, multi-memory/table, memory/table-64, and new value types. The former three are perhaps best treated as fundamental at this point, with no attempt of separating them out. But new value types and their use in various constructs will be a repeated issue that I'm not sure about how to organise best, because it's an NxM problem. (And we already suck badly at filling that NxM test matrix!)

[andreaTP]

Adding annotation syntax (@if-supports "multi-memory") (module ...) to the wast files seems appropriate, probably it can be 90% generated by LLM after a few initial examples. Guiding the LLM implementation with what is already in SpiderMonkey is going to be smooth for someone that knows where to look.

[dschuff]

I've actually also been thinking about this lately, in the context of importing spec tests into WPT. WPT also likes to have functionality organized by "feature" and of course browsers typically do implement features one proposal at a time, because they aren't usually starting from scratch.
In the absence of annotating the whole test suite, I was noting that most of the individual wast files target a specific proposal, and was thinking that encoding that file->proposal association somewhere, perhaps combined with duplicating some tests in case of overlap (and handling one-offs where tests are superseded) would be sufficient for that use case.

An alternative to annotation syntax could be some kind of out-of-band file (JSON or something) that could associate whole files and individual tests to constructs and/or proposals could be less work than annotating each module, but it would be less robust to e.g. insertion of new tests and easy to forget to update.

So all that is to say, I'd be in favor of reorganizing and annotating the tests by proposal, and willing to help out. Having a directory for each proposal would be nice, but if there's a way to annotate each module, that seems fine to me too.

Some specific thoughts:

Multi-memory seems like it should be easy enough to separate (and the fact that it is relatively recent and still missing from many implementations, and that other tests and proposals tend not to depend on it), that I think we should just separate it, even if it's within the same file as single-memory tests.

Likewise for memory64, actually.

Multi-value seems more fundamental. For the cases where one proposal actually depends on another, then it's obviously necessary for the tests to do so too, but ideally there wouldn't be tests that use another proposal where there's no actual dependence. I'm willing to be pragmatic on this though and if there are lots of unrelated tests that use multi-value I would be ok with keeping it as fundamental.

If we're going way back to the beginning, mutable globals is another one that's very fundamental.

For the NxM cases, maybe it would be sufficient to add multiple @if-supports annotations to a module that tests the intersection of 2 proposals (and maybe put it in its own file and/or in a common directory or something)? For cases where there is a true dependence between proposals, I don't think it's really necessary to do anything (e.g. no need to annotate all of the GC tests with @if-supports "reference-types").