[ci] Default to n=2 for test parallelism (apache#12414)

* Revert "[skip ci] Revert "[ci] Default to n=2 for test parallelism (apache#12376)" (apache#12413)"

This reverts commit 478b672.

* [ci] Default to n=2 for test parallelism

This is attempt #2 of apache#12376 which was reverted in apache#12413. The changes
in `plugin.py` should keep all the tests on the same node so sporadic
failures don't happen due to scheduling.

Co-authored-by: driazati <[email protected]>

Author: driazati