Incorporating and merging with the compliance test suite

[gregsdennis]

Both this project and the CTS have a lot of useful test cases. I think it would be ideal to maintain these in a single place.

To that end, I'd like to propose merging the CTS test cases into this project. This project seems to be ideal because it already contains test harnesses for many well-known implementations.

To do this, I'd like to add some properties to the test cases to indicate what the spec says:

• spec-valid - A boolean that determines whether the path is valid (can be parsed) according to the spec. There are a number of test cases in the CTS to ensure that the implementation returns an error when it should.
• spec-result - The expected result as defined by the spec. I think it's still valuable to collect consensus results as well.

There may be more needed, but this is what I can think of so far.

[gregsdennis]

Related to #127

[glyn]

@gregsdennis If the CTS test cases were merged into this project, would the CTS project still be maintained separately? I don't think every implementor will want to add their implementation to this project (e.g. if it is closed source) just to find out if it passes the CTS.

[gregsdennis]

would the CTS project still be maintained separately?

No, that project could be retired.

I don't think every implementor will want to add their implementation to this project (e.g. if it is closed source) just to find out if it passes the CTS.

There's no requirement to add an implementation to the comparison to see if it passes the CTS. The "CTS" portion of the project is really just a list of test cases and expected results, just like the current CTS. It could still be run privately.

---

Benefits of doing this are (not limited to):

• we maintain the test cases in one place
• there are a lot of spec-invalid test cases in this project that are useful
• we get to see how closely (these) implementations align with the spec
• those implemnetations which have been implemented here get a free CTS implementation and report