Data value JSON format

[larshelge]

Context

From a client and in particular the new data entry web app perspective, the current data value format is cumbersome to use, as it requires the client to specify the category option combo (and until recently, the attribute option combo) specifically. The option combos could be considered internal details which should not be exposed to the client. Having to load all option combos to the client uses unnecessary time and bandwidth.

Considerations

JSON is not suited for custom property names in the same way XML is, meaning we need a fixed format for the category combination and category options. The naming of "combo" and "attribute" is not ideal, but we are constrained by the existing model and should not make drastic changes in format.

Format

Proposed data value JSON format can be seen below. The new properties will be supported both for the individual and bulk format. The main goal is to avoid having to expose category option combinations, and instead rely on category combo and category options.

{
  "dataElement": "fbfJHSPpUQD",
  "period": "202201",
  "orgUnit": "DiszpKrYNg8",
  "category": {
    "combo": "jCNGsC2NawV",
    "options": [
      "apsOixVZlf1",
      "zs2Ra6sikup"
    ]
  },
  "attribute": {
    "combo": "O4VaNks6tta",
    "options": [
      "CW81uF03hvV",
      "i4Nbp8S2G6A"
    ]
  },
  "value": "10"
}

[vietnguyen]

Link to doc of current format

[vietnguyen]

Changing the class org.hisp.dhis.dxf2.datavalue.DataValue will affect both api/dataValues and api/dataValueSets endpoints.

But in api/dataValueSet endpoint should we also have an outer attribute object ?

{
  "attribute": {
    "combo": "O4VaNks6tta",
    "options": [
      "CW81uF03hvV",
      "i4Nbp8S2G6A"
    ]
  },
  "dataValues": [
    {
      "dataElement": "fbfJHSPpUQD",
      "period": "202201",
      "orgUnit": "DiszpKrYNg8",
      "value": "10",
      "category": {
        "combo": "jCNGsC2NawV",
        "options": [
          "apsOixVZlf1",
          "zs2Ra6sikup"
        ]
      }
    },
    {
      "dataElement": "fbfJHSPpUQD",
      "period": "202201",
      "orgUnit": "DiszpKrYNg8",
      "value": "11",
      "category": {
        "combo": "jCNGsC2NawV",
        "options": [
          "apsOixVZlf1",
          "zs2Ra6sikup"
        ]
      }
    }
  ]
}

[larshelge]

Good point. I think we need to have that. Though I think we could potentially split this into two iterations and only support it for the inner data value object (and the single data value endpoint) in first iteration.

[bobjolliffe]

Lars I wonder why you want to expose the "combo" uid instead of the individual categories. In a sense that is an internal detail like the categoryoptioncombo. Which seems to require the order of the options which follow to be significant. And I dont think the order of categories within a categorycombo is really significant. It might be I am misunderstanding. But wouldn't something similar to the following be more robust (and also easily parseable):`

"dataValues": [
  {
    "dataElement": "fbfJHSPpUQD",
    "period": "202201",
    "orgUnit": "DiszpKrYNg8",
    "value": "10",
    "categories": [
      { "category": "jCNGsC2NawV", "option": "apsOixVZlf1"},
      { "category": "huhyy8yuhhl", "option": "zs2Ra6sikup"}
    ]
  }
]

Note that this would also be directly mappable" to ADX. which I know is not a design goal here but might have some advantages. For exdample, using codes instead of uids would render something like:

{
  "dataElement": "MALARIA_CASES",
  "period": "202201",
  "orgUnit": "DiszpKrYNg8",
  "value": "10",
  "categories": [
    { "category": "SEX", "option": "FEMALE"},
    { "category": "AGE", "option": "P0Y--P1Y"}
  ]
}

[jbee]

Is there any real use case to have same set of categories defined multiple times? Are people using this as a work-around or is this just unintentional duplication?

[larshelge]

No, not a useful use-case from what I know.

[abyot]

Yes, we could end-up with multiple category combos for the same set of categories. However, since we have a data element which is linked only to a single category combo, I think we will be fine with just categories and corresponding options.

[jbee]

Maybe we can also work towards adding a unique constraint or error messages to make this into an actual system property?

[bobjolliffe]

Lars it strikes me that the possibility of having multiple identical categorycombos is an unfortunate one, and rather than have it dictate the design we might hope to eradicate it instead. I wonder how often this actually occurs in practice and could it be detected and rectified as part of metadata assessment and cleanup scripts? I think there is also a different logical problem with the original proposal. maybe also an edge case of sorts. It is quite possible for the same category options to appear in the different categories in a categorycombo. I think particularly of generic labels like YES, NO, UNKNOWN, UNSURE, UNTESTED, etc etc but your mind could wander and think of more. So it is quite possible to have two categoryoptions and a categorycombo, but not be able to say which category each option relates to. The only way to be absolutely unambiguous would be to combine both approaches and specify both the categorycombo and the category/option tuples. Which starts to get verbose. Probably it would be better to try and (i) eliminate the multiple identical categories or (ii) rely on what Abyot has pointed out, that the categorycombo is linked to the dataelement (though I think it can be overridden in the dataset so need to be careful).

[Birkbjo]

Note that my remarks below are in the context of the Data Entry app.

Thanks Lars! It would be great to not need to "match" options to their categoryOptionCombo on the client. However, we would still need a way to uniquely an input field. In our current implementation we closely mirror how this was done in the old Data Entry app; by concatenating the dataElementId and categoryOptionComboId. This is also how it's done in the custom-forms. So this API-change would also need an update to how the custom-forms are handled, right?

My first idea is to just concatenate the optionIds - eg. dataElementId-optId1.optId2.

This do however make the order of the optionIds significant. This is not necessarily a problem on our client-side generated form, as that is mostly internal to our rendering logic, and we can work around that. But we would still prefer the options returned in this API to be stable.

[larshelge]

Right. We could continue to support categoryOptionCombo as a field in the new format too, so that the client could decide. Would that work, so that we can continue to handle custom forms the way we do now?