[API] v2 interface design - output (predict, backtesting etc) interface

[fkiraly]

The current v2 interface design lacks a clear predict interface (or similarly named).

It should be possible, for all models:

• to query k-step-ahead predictions, if there is no future-known data
• to query k-step-ahead predictions, if there is some future-known data
• to pass a future batch and get predictions for it (without specifying k, which is assumed to be the index) - optional

Further, the data that is passed is optimally also a TimeSeries, not the D2 module.

DSIPTS has something similar in the inference method, if I am not mistaken (@agobbi?)
https://github.com/DSIP-FBK/DSIPTS/blob/main/dsipts/models/base.py

FYI @phoeenniixx, @PranavBhatP, @xandie985

[agobbifbk]

1. most of the DL models can use the forward method as predict. There are some exception (for example diffusion models). For this reason in the base class it should be implemented a predict method that, by default, returns the result of the forward function
2. The output of the forward should be standardize: in my opinion having a 4d tensor covers all the possible situation. Most of the time series models return a 3d tensor in the shape [batch_size, output_length, output_channels]. If we add a final dimension it can be used for the Quantile loss having something of the shape [batch_size, output_length, output_channels, quantiles]. If no quantile loss is used, quantiles=1. If we predict just one variable output_channels=1
3. output_length is related to the initialization of the model during the train procedure, can not be changed by the user at inference time. If we need less horizion step, better to initialize a smaller model, if we need more better to enlarge it. We can think to reuse the prediction for making new prediction, I personally do not like it because there are a lot of checks to do (for example covariates) and the error can explode quickly. Again, if we need longer horizon better to train the model to do it
4. The predict method returns a tensor, we need to unroll it and chose an output format for the user. I think an O (output layer) can be something to investigating:

• for back-testing, where we have the real values, the maximum information should be returned (something like the structure below, if multiple timeseries are digested, there will be a column called group related to those groups)

• for predictions in reatime, where the batch size is 1 (you want just one prediction) we can think to return a timeseries, but if we keep the same structure it can be easier to maintain

5. Ideally the flow of the prediction can be something like:

• creating the 'DataLoaderor leverage thesetup of the LighteningDataModule (stage == "predict"`)
• obtain the prediction (and the real values if in back-testing mode)
• stack the predictions
• scale the results
• create a pandas.DataFrame with all the information (groups, timestamps and quantiles)
  See here for an idea (maybe there is a cleaner solution)

6. If in the D2 layer the checks are implemented correctly, the point 5. works also in case of realtime prediction. Before creating the dataloader we need to ensure that all the future covariates are in the dataset. There can be NaNs in the dataset for example the target future variable. This is not a problem because in the D2 layers, when the validity of a sample is checked, in inference mode it should not check the target future values.

Hope this helps, happy to discuss in a tech meeting.

[phoeenniixx]

Hi, @fkiraly please assign me and @PranavBhatP this issue

[fkiraly]

Sure! @PranavBhatP still needs to comment on it first