Bundling - managing references

[FTravaglini-crif]

Hello, the opeani-generator plugin doesn't support the '$ref' keyword according to json schema specifications, and before looking in the defs for the referenced schema tries to reach the URL https://example.com/string, to use the example in the documentation.

Let's say I have this schema on my apicurio registry

{
  "$id": "https://example.com/person"
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$ref": "https://example.com/party",
  "$defs": {
    "https://example.com/party": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "$id": "https://example.com/party",
      "type": "string"
    }
  }
}

In this case, if I reference in my openapi.yaml the API GET (https://apicurio.com/apis/registry/v3/groups/jsonschemas/artifacts/person/versions/1/content) I can retrieve the json from the registry, but than the openapi generator, to generate the PersonDTO will try to reach "https://example.com/party", and won't resolve it with the local schema in the "$defs".

I was wondering if it could be useful to add a step to the bundling process: the addition of '#/$defs/' to the reference of the schema ( -> "$ref": "#/$defs/https://example.com/party"). It's probably not that common but I don't see downsides to this modification

[jviotti]

Hey @FTravaglini-crif ,

Thanks for reaching out!

the opeani-generator plugin doesn't support the '$ref' keyword according to json schema specifications

Interesting to hear that!

I was wondering if it could be useful to add a step to the bundling process: the addition of '#/$defs/' to the reference of the schema ( -> "$ref": "#/$defs/https://example.com/party"). It's probably not that common but I don't see downsides to this modification

The current approach strictly follows the JSON Schema Bundling algorithm (https://json-schema.org/blog/posts/bundling-json-schema-compound-documents) but I'm all for an improvement like this to help with less compliant tooling, at least as an opt-in option.

Let me get into this on Monday. I'll probably have it ready and released by then (it's not a hard one, I think!)

if I reference in my openapi.yaml the API GET (https://apicurio.com/apis/registry/v3/groups/jsonschemas/artifacts/person/versions/1/content)

As an aside, I'm very curious about your experience with Apicurio, how you are using it, and if you are facing any limitations. We at Sourcemeta are actively working on a JSON Schema Registry (https://www.sourcemeta.com) that aims to be very performant and compliant, and any feedback from your Apicurio use case would be really interesting for us to hear about!

[FTravaglini-crif]

About the Apicurio Registry experience we are just starting to play with it. Our idea is to take advantage of both JSON Schemas and OpenAPI management in the design and development process of our solution, making the JSON schemas our "source of truth" as the foundation of the development

[jviotti]

@FTravaglini-crif

Our idea is to take advantage of both JSON Schemas and OpenAPI management in the design and development process of our solution, making the JSON schemas our "source of truth" as the foundation of the development

Awesome! This is exactly the problem we are trying to solve with a lot of our tooling, including the Registry. If you are up for it, I'd love to continue discussing your specific use case in more depth. I'm a member of the JSON Schema Technical Steering Committee and can definitely provide some advice and how we are designing the Sourcemeta Registry with this specific use case in mind! Feel free to reach out at [email protected]!

[FTravaglini-crif]

I spoke with my colleagues and for now it seems there is no intention of searching for substitute for tools in our solution.
Thank you very much for the offer though! I'll be sure to keep it in mind if we face some issues in our usage of Apicurio, to consider if the Sourcemeta Registry could be a valid alternative

[jviotti]

No worries! Do let us know how it goes with Apicurio and if there is anything you like or don't like about it. What are aiming for is:

• Make the Registry dependency-free (not even a database or separate storage layer)
• Make the Registry deployable out of Git repositories of schemas (for governance purposes) and disallow dynamically registering new schemas through i.e. an API
• Add various high-level monitoring features such as scoring the health and strictness of schemas and display that on the HTML UI
• Tight programming language integration (for example, we are working on an NPM integration to consume schemas as dependencies in JavaScript)
• High-performance schema serving and validation APIs (using our record breaking JSON Schema validator: https://github.com/sourcemeta/blaze)
• Automatic upgrade/downgrade of schemas between versions of JSON Schema (to easily bridge to different kinds of tooling)

If you think we are missing anything on the roadmap, let us know! 🙏🏻 We are actively working in all of this and any early super-honest feedback is super valuable!

[jviotti]

@FTravaglini-crif Actually, before I try this improvement, have you tried the --without-id option on the bundle command? That option does something like what you suggested already. It was for workarounding limitations of Visual Studio Code, but I wonder the principle is the same.

[jviotti]

Hey @FTravaglini-crif ,

makes the references result in string where some special characters are not processed correctly.

I mean what tool, if any, is telling you that the special characters are not processed correctly?

The escaping we do here (with ~ and %) are standard JSON Pointer URI fragment escaping (https://www.rfc-editor.org/rfc/rfc6901#section-6). A JSON Schema implementation that is fully URI and JSON Pointer compliant should have no issues with the schema you get using --without-id.

When a standard implementation encounters the $ref, it must interpret it as a URI, thus decoding it as such. And when the URI fragment is a JSON Pointer (like in this case), further JSON Pointer escaping is expected to take place, making it correctly match the JSON location you actually have in $defs (which does not need escaping at the JSON level).

So after double checking, I can confirm that the escaping that --without-id is doing on the JSON Pointer is correct. But again, maybe some less compliant implementations do not implement $ref unescaping correctly!

Let me try the schema you provided with the Java generator and see if I find anything! If we indeed find that some tooling is non-standard, I can study it further to see in which ways it is not standard, and see if we can workaround it somehow!

[jviotti]

@FTravaglini-crif So this is what I tried:

openapi: 3.1.0
info:
  title: API
  version: 1.0.0
paths:
  /:
    get:
      responses:
        '200':
          description: Test
          content:
            application/json:
              schema:
                $ref: ./Organisation.json

Where Organisation.json is what you shared:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Organisation",
  "$comment": "Data of an organisation/legal entity",
  "type": "object",
  "allOf": [
    {
      "$ref": "#/$defs/https%3A~1~1apicurio.com~1jsonschemas~1party~1v1.schema.json"
    },
    {
      "type": "object",
      "required": [
        "organisationName"
      ],
      "properties": {
        "addresses": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/https%3A~1~1apicurio.com~1jsonschemas~1address~1v1.schema.json"
          }
        },
        "contacts": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/https%3A~1~1apicurio.com~1jsonschemas~1contact~1v1.schema.json"
          }
        },
        "organisationName": {
          "type": "string"
        }
      },
      "unevaluatedProperties": false
    }
  ],
  "$defs": {
    "https://apicurio.com/jsonschemas/address/v1.schema.json": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "title": "Address",
      "$comment": "Address details",
      "type": "object",
      "required": [
        "addressTypeCode",
        "address",
        "city",
        "country"
      ],
      "properties": {
        "address": {
          "type": "string"
        },
        "addressTypeCode": {
          "type": "string"
        },
        "city": {
          "type": "string"
        },
        "country": {
          "type": "string"
        },
        "county": {
          "type": "string"
        },
        "deleted": {
          "type": "boolean"
        },
        "guid": {
          "type": "string",
          "format": "uuid"
        },
        "postcodeAreaCode": {
          "type": "string"
        }
      },
      "patternProperties": {
        "^extension$": {
          "type": "object"
        }
      },
      "unevaluatedProperties": false
    },
    "https://apicurio.com/jsonschemas/contact/v1.schema.json": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "title": "Contact",
      "$comment": "Contact details for a party",
      "type": "object",
      "required": [
        "contactTypeCode"
      ],
      "properties": {
        "contactTypeCode": {
          "type": "string"
        },
        "deleted": {
          "type": "boolean"
        },
        "email": {
          "description": "Email",
          "type": "string"
        },
        "guid": {
          "type": "string",
          "format": "uuid"
        },
        "mobileNumber": {
          "description": "Mobile Phone Number",
          "type": "string"
        },
        "phoneNumber": {
          "description": "Phone Number",
          "type": "string"
        }
      },
      "patternProperties": {
        "^extension$": {
          "type": "object"
        }
      },
      "unevaluatedProperties": false
    },
    "https://apicurio.com/jsonschemas/party/v1.schema.json": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "title": "Party",
      "$comment": "Base party/customer entity",
      "type": "object",
      "required": [
        "partyTypeCode"
      ],
      "properties": {
        "description": {
          "type": "string"
        },
        "categoryCode": {
          "type": "string"
        },
        "code": {
          "type": "string"
        },
        "endDate": {
          "type": "string"
        },
        "guid": {
          "type": "string",
          "format": "uuid"
        },
        "partyTypeCode": {
          "enum": [
            "PERSON",
            "ORGANISATION"
          ]
        },
        "startDate": {
          "type": "string"
        }
      },
      "patternProperties": {
        "^extension$": {
          "type": "object"
        }
      }
    }
  }
}

If I run this, the OpenAPI generator seems to succeed (or at least there are no visible errors):

openapi-generator generate -i openapi.yaml -g java

Thought I'm not proficient enough in Java to know if there is anything odd in there!

[jviotti]

At least based on the generated bundled OpenAPI spec, it does seem like the generator traversed those pointer references just fine! See this:

openapi: 3.1.0
info:
  title: API
  version: 1.0.0
servers:
- url: /
paths:
  /:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Organisation"
          description: Test
      x-accepts:
      - application/json
components:
  schemas:
    Address:
      example:
        country: country
        addressTypeCode: addressTypeCode
        address: address
        deleted: true
        postcodeAreaCode: postcodeAreaCode
        city: city
        county: county
        guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
      properties:
        address:
          type: string
        addressTypeCode:
          type: string
        city:
          type: string
        country:
          type: string
        county:
          type: string
        deleted:
          type: boolean
        guid:
          format: uuid
          type: string
        postcodeAreaCode:
          type: string
      required:
      - address
      - addressTypeCode
      - city
      - country
      title: Address
    Organisation:
      allOf:
      - properties:
          description:
            type: string
          categoryCode:
            type: string
          code:
            type: string
          endDate:
            type: string
          guid:
            format: uuid
            type: string
          partyTypeCode:
            enum:
            - PERSON
            - ORGANISATION
            type: string
          startDate:
            type: string
        required:
        - partyTypeCode
        title: Party
      - properties:
          addresses:
            items:
              $ref: "#/components/schemas/Address"
            type: array
          contacts:
            items:
              $ref: "#/components/schemas/Contact"
            type: array
          organisationName:
            type: string
        required:
        - organisationName
      example:
        partyTypeCode: PERSON
        addresses:
        - country: country
          addressTypeCode: addressTypeCode
          address: address
          deleted: true
          postcodeAreaCode: postcodeAreaCode
          city: city
          county: county
          guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
        - country: country
          addressTypeCode: addressTypeCode
          address: address
          deleted: true
          postcodeAreaCode: postcodeAreaCode
          city: city
          county: county
          guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
        organisationName: organisationName
        code: code
        endDate: endDate
        description: description
        guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
        categoryCode: categoryCode
        startDate: startDate
        contacts:
        - deleted: true
          phoneNumber: phoneNumber
          contactTypeCode: contactTypeCode
          mobileNumber: mobileNumber
          guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
          email: email
        - deleted: true
          phoneNumber: phoneNumber
          contactTypeCode: contactTypeCode
          mobileNumber: mobileNumber
          guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
          email: email
      title: Organisation
      $defs:
        https://apicurio.com/jsonschemas/address/v1.schema.json:
          $schema: https://json-schema.org/draft/2020-12/schema
          title: Address
          $comment: Address details
          type: object
          required:
          - addressTypeCode
          - address
          - city
          - country
          properties:
            address:
              type: string
            addressTypeCode:
              type: string
            city:
              type: string
            country:
              type: string
            county:
              type: string
            deleted:
              type: boolean
            guid:
              type: string
              format: uuid
            postcodeAreaCode:
              type: string
          patternProperties:
            ^extension$:
              type: object
          unevaluatedProperties: false
        https://apicurio.com/jsonschemas/contact/v1.schema.json:
          $schema: https://json-schema.org/draft/2020-12/schema
          title: Contact
          $comment: Contact details for a party
          type: object
          required:
          - contactTypeCode
          properties:
            contactTypeCode:
              type: string
            deleted:
              type: boolean
            email:
              description: Email
              type: string
            guid:
              type: string
              format: uuid
            mobileNumber:
              description: Mobile Phone Number
              type: string
            phoneNumber:
              description: Phone Number
              type: string
          patternProperties:
            ^extension$:
              type: object
          unevaluatedProperties: false
        https://apicurio.com/jsonschemas/party/v1.schema.json:
          $schema: https://json-schema.org/draft/2020-12/schema
          title: Party
          $comment: Base party/customer entity
          type: object
          required:
          - partyTypeCode
          properties:
            description:
              type: string
            categoryCode:
              type: string
            code:
              type: string
            endDate:
              type: string
            guid:
              type: string
              format: uuid
            partyTypeCode:
              enum:
              - PERSON
              - ORGANISATION
            startDate:
              type: string
          patternProperties:
            ^extension$:
              type: object
    Contact:
      example:
        deleted: true
        phoneNumber: phoneNumber
        contactTypeCode: contactTypeCode
        mobileNumber: mobileNumber
        guid: 046b6c7f-0b8a-43b9-b35d-6489e6daee91
        email: email
      properties:
        contactTypeCode:
          type: string
        deleted:
          type: boolean
        email:
          description: Email
          type: string
        guid:
          format: uuid
          type: string
        mobileNumber:
          description: Mobile Phone Number
          type: string
        phoneNumber:
          description: Phone Number
          type: string
      required:
      - contactTypeCode
      title: Contact

It also generated the models well as far as I can tell:

      ▾ �model/
          �AbstractOpenApiSchema.java
          �Address.java
          �Contact.java
          �Organisation.java

[FTravaglini-crif]

Ah okay, I see! I didn't try honestly to generate the code through the openapi generator cause I was not comprehending correctly what you meant about the JSON Pointer URI fragment escaping. Thanks for the explanation, I'll have a look at the link you shared but I concur with you that the java generation is correct. Thank you again for the time, I guess now it is just a choice for us about keeping the references with the JSON Pointer URI fragments or have a more readable JSON schema

[jviotti]

Any time!

Thank you again for the time, I guess now it is just a choice for us about keeping the references with the JSON Pointer URI fragments or have a more readable JSON schema

Note that bundling without --without-id does yield much nicer schemas. But I guess the limitation you shared was around $id and nested schema resources, which your tooling didn't seem to like very much.

Overall, the --without-id option is just for workarounding implementation limitations (like these ones) and not optimised for readability at all. I suggest only using the option when feeding the schemas to non-compliant tooling, and then you don't see the ugliness when not needed.

That said, maybe improving the location pointers does make it a bit less ugly anyway