Very slow to extract predictions from large projects

[Fredrik00]

Is your feature request related to a problem? Please describe.
I have a large project in Label Studio for annotating bounding boxes (over 100k tasks). As part of my workflow I want to extract the annotations for all manually labelled tasks, followed by all predictions with a score above some threshold. I attempted to extract predictions through the task API, but the predictions information was empty. If I attempt to use the predictions API instead, it times out before I can get a response.
My only option right now appears to be using the predictions API with a task filter, fetching the predictions one task at a time, but this takes a very long time, even when running the requests in parallel (90+ minutes in my case).

Describe the solution you'd like
I think there are two options for improving prediction extraction:

1. (Preferred) Make it possible to include prediction/annotation data from the tasks API (api/tasks). This would make it possible to apply filters and paging, which I believe would be much faster than the current workaround.
   Additionally filtering the predictions by model_versions to reduce the payload size would also be great here.
2. Add paging/filtering to the predictions API (api/predictions), similar to that of the tasks API. This would allow us to fetch predictions for multiple tasks at a time, with a reasonable page size to ensure the request does not time out.

Describe alternatives you've considered
Attempted workarounds described above.

Additional context

[heidi-humansignal]

Hello,

Thanks for outlining your workflow. You can already page through tasks and pull predictions in bulk by using the tasks endpoint with the right query parameters—no need to fetch each task one-by-one. Here’s how:

1. Include predictions (and annotations) in the paginated tasks response

By default,

GET /api/tasks?project=<PROJECT_ID>

returns only basic task fields. To get predictions and annotations in each page, add

fields=all

or explicitly request the prediction/annotation sub-fields with

include=predictions,annotations

Example:

GET /api/tasks?
project=<PROJECT_ID>&
fields=all&
page_size=200

This returns batches of up to 200 tasks, each including a predictions array (with score, model_version, etc.) and annotations.

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[Fredrik00]

I attempted to use include with "predictions,annotations" before, but this resulted in no data. Turns out I had to combine it with fields=all for the data to be included.
I find it a bit confusing that we have to specify fields=all when the included fields are specified, but at least this solves my initial issue.
A filter on model_versions to include predictions for would still be great though, there is quite a lot of data that has to be sent when there are multiple predictions for each task.