Our current approach to CI is not scalable

[jessebraham]

The jobs we're running in CI have already exploded in numbers, and this problem is only going to be amplified as we add support for additional devices and features.

We need to fundamentally rethink how we're handling CI in this repository.

[bjoernQ]

I think this is not easy to solve since if we want to (and we have to) check all the combinations etc. there is no way around running all the steps

Only thing I can think of is to combine the steps the individual jobs do ... that way we would reduce the overhead of setting up the runner and preparing the toolchain - but compared to the actual build that is not much and it would be much harder to maintain (and also to see what exactly failed if something fails) - so probably not worth it

[i404788]

Maybe we can cache dependencies & incremental build artifacts, then it should only take a few seconds to build everything unless there is a major change.

[jessebraham]

I definitely would like to look into caching, that will help a lot. I think we can probably shave a fair bit of runtime off with this alone.

I'm personally more concerned about the rapid growth in the number of jobs, we can only run 20 concurrently and with support for C6 and H2 expected in the coming months we will likely have >50 jobs to process using our current approach.

[i404788]

Well if it's purely about the number of jobs you can combine most of them into their board type.
Currently up to 7 jobs per board, are run mostly relating to different permutations of features. But you can check multiple permutations of features in one job but separate steps, and get it down to 1 (or 2 if async is separated) jobs per board.

It would also give a slight speed up in processing time, but does require needing a refactor ci.yml into a potentially less elegant structure