Datastore Paging, Ordering and Filtering API

[jbee]

Filtering

In contrast to schema based metadata the datastore is schema-less. Values can be any type of JSON node.
This is especially relevant for filtering as proper comparison may depend on the actual type of JSON node.

Instead of creating the proper filter based on schema information the filter used is based on the operator and provided value:

• if the value is null this is a null check for any node type
• if the value is true or false it is a check for a boolean node
• if the value is a valid JSON number expression it is a check for a number node (numeric comparison)
• otherwise it is a text based check (this might restrict to string nodes)
• if a text based check should be forced even if one of the other conditions is true the value can be quoted: '<text>' is same as <text> except that the target node type is string

Some operators have special semantics.

• empty is a check for a empty object, array or string node
• !empty is a check for non empty object, array or string node

Property Expressions

Property Expression	Semantics
_	the entry key is...
.	the entry (root) value is...
[<index>]	the array element at index is...
<name>	the object member with name is ...
<root>.<child>	the object member at path root.child is ...
<root>[0].<child>	the member child of the first element of the array held by the root member of the root object is ...

Operators

Operator	Description	Boolean	Number	String	Array	Object
null	is null	✓	✓	✓	✓	✓
!null	is not null	✓	✓	✓	✓	✓
empty	is empty	x	x	✓	✓	✓
!empty	is not empty	x	x	✓	✓	✓
eq	equals	✓	✓	✓	x	x
!eq, neq, ne	not equals	✓	✓	✓	x	x
lt	less than	x	✓	✓	x	x
lte, le	less than or equal	x	✓	✓	x	x
gt	greater than	x	✓	✓	x	x
gte, ge	greater than or equal	x	✓	✓	x	x
like	text pattern matches	x	x	✓	x	x
!like	text pattern matches not	x	x	✓	x	x

• Other pattern matching operators like startsWith, endsWith etcetera are just variants of like/!like.
• If a number or string node is targeted for those operators that support both then depends upon the provided value

Ordering

Similar to filtering the lack of schema information is a problem for ordering as it is unclear if we should do an alphanumeric or numeric order. Therefore 4 sort order types can be distinguished:

Name	Description
asc	ascending alphanumeric (default)
desc	descending alphanumeric
nasc	ascending numeric
ndesc	descending numeric

Alphanumeric could also be considered the natural order that is always defined.
Numeric order will require a conversion which can fail should a value not be numeric.

To start with it seems sufficient to only support single property sorting.

When a filter is used on the sort property we could infer numeric order in case we use a numeric filter.
This is not implemented but worth considering.

Paging

Like in metadata results are paged by default.

A paged result has the form

{
  "pager": { "page": 1, "pageSize": 50 },
  "entries": []
}

As usual page and pageSize have a minimum value of 1.
If paging is set false or if headless=true is used the response has the form

[]

This array being the same array that otherwise lists entries as the entries property in the root object.

Paging does not support counting the total number of matches or total number of pages.

Fields

For completion some notes on fields filtering.

Since we always include the entry key as key property of the resulting entries it does make sense to allow an empty fields= (nothing) parameter list so only extract the keys of matching entries.

Outlook

In the future we might support

• allow to explicitly filter on JSON node type, e.g. name:isa:string (isa would be a new operator for node type checks)
• allow JSON objects or arrays as filter value to filter equality or non-equality
• add special operators to filter based on keys present in an object nodes
• allow filtering based on array node length
• allow filtering based on string node length
• offer a parameter to turn key only results from normal format [{"key":"x"}, {"key":"y"}] into just ["x","y"] (which is same as the long time supported listing of all keys - this would basically allow to filter keys as they were returned for a long time)