ci: add randomized test matrix for better test coverage

[vlsi]

This enables significantly improve test coverage while still keeping test feedback fast and easy configuration.

For example, this matrix captures locale-dependent issues like #1414
Here's how the matrix looks in my fork: https://github.com/vlsi/spock/actions/runs/1702274076

The idea is that you declare test axis like os, java, java_distribution, timezone, locale, etc, and matrix.js produces several jobs for you.

Note: I do not understand what variant=2.5 and variant=3.0 mean, and I do not understand why did you exclude {java: 17, variant: '2.5', os: 'ubuntu-latest'}.
If the only reason to exclude 2.5 with java 17 on ubuntu was performance, then you might remove that exclusion from matrix.js.

See https://github.com/vlsi/github-actions-random-matrix

If you want to test matrix changes locally, you can launch node matrix.js and it prints something like

...
  {
    java_distribution: 'liberica',
    java8_distribution: 'liberica',
    jit: 'hotspot',
    java: '17',
    tz: 'America/New_York',
    os: 'ubuntu-latest',
    variant: '3.0',
    locale: { language: 'de', country: 'DE' },
    name: '17, liberica, ubuntu, America/New_York, de_DE, stress JIT',
    testExtraJvmArgs: '-Duser.country=DE -Duser.language=de -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM'
  },
  {
    java_distribution: 'zulu',
    java8_distribution: 'liberica',
    jit: 'hotspot',
    java: '17',
    tz: 'America/New_York',
    os: 'ubuntu-latest',
    variant: '3.0',
    locale: { language: 'tr', country: 'TR' },
    name: '17, zulu, same hashcode, ubuntu, America/New_York, tr_TR, stress JIT',
    testExtraJvmArgs: '-XX:+UnlockExperimentalVMOptions -XX:hashCode=2 -Duser.country=TR -Duser.language=tr -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM'
  }
]
::set-output name=matrix::{"include":[{"java_distribution":"liberica","java8_distribution":"temurin","jit":"hotspot","java":"8","tz":"UTC","os":"windows-latest","variant":"3.0","locale":{"language":"fr","country":"FR"},"name":"8, liberica, windows, UTC, fr_FR, stress JIT","testExtraJvmArgs":"-Duser.country=FR -Duser.language=fr -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM"},{"java_distribution":"temurin","java8_distribution":"zulu","jit":"hotspot","java":"11","tz":"America/New_York","os":"ubuntu-latest","variant":"2.5","locale":{"language":"de","country":"DE"},"name":"11, temurin, ubuntu, America/New_York, de_DE","testExtraJvmArgs":"-Duser.country=DE -Duser.language=de"},{"java_distribution":"liberica","java8_distribution":"liberica","jit":"hotspot","java":"17","tz":"UTC","os":"macos-latest","variant":"3.0","locale":{"language":"de","country":"DE"},"name":"17, liberica, macos, UTC, de_DE","testExtraJvmArgs":"-Duser.country=DE -Duser.language=de"},{"java_distribution":"liberica","java8_distribution":"liberica","jit":"hotspot","java":"17","tz":"America/New_York","os":"ubuntu-latest","variant":"3.0","locale":{"language":"de","country":"DE"},"name":"17, liberica, ubuntu, America/New_York, de_DE, stress JIT","testExtraJvmArgs":"-Duser.country=DE -Duser.language=de -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM"},{"java_distribution":"zulu","java8_distribution":"liberica","jit":"hotspot","java":"17","tz":"America/New_York","os":"ubuntu-latest","variant":"3.0","locale":{"language":"tr","country":"TR"},"name":"17, zulu, same hashcode, ubuntu, America/New_York, tr_TR, stress JIT","testExtraJvmArgs":"-XX:+UnlockExperimentalVMOptions -XX:hashCode=2 -Duser.country=TR -Duser.language=tr -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM"}]}

[leonard84]

Hi @vlsi, this looks interesting but I don't want PRs and normal master builds to behave randomly or have too many builds, with Groovy 4 we already got another bunch of builds. However, I could see this used in a nightly/weelkly scheduled build, where it is not slowing down normal usage and would still catch possible issues.

[vlsi]

with Groovy 4 we already got another bunch of builds

That is the reason I suggest using the randomized matrix.
Note that "randomized matrix" != "random failures".

I am sure the randomizer is way better to keep the matrix organized, test feed back fast, etc, etc.

[leonard84]

Note that "randomized matrix" != "random failures".

If I understand it correctly,it will produce a different combination each time. Currently, github only allows to re-run all jobs, so you would have a different set ob jobs for the re-run. The builds are not 100% stable, as we have some tests with tight timeouts, so as not to slow down the whole build, but this causes the occasional failure with the need to re-run. Furthermore, I think that the set of jobs for PRs should be the same as the ones run on master.

So moving that randomness to scheduled jobs sound like a viable solution.

[szpak]

So moving that randomness to scheduled jobs sound like a viable solution.

I also would prefer to have it that way due to the reasons Leonard mentioned.

[vlsi]

so you would have a different set ob jobs for the re-run

Running different tests on "re-run" is completely fine, and it does not hurt.

Approaches are like property-based testing and fuzzing create new tests every time you launch them, and that is considered to be a power of testing approach rather than a weakness. The more time you devote to fuzzing, the more bugs it finds.

That is exactly the same with randomized CI: the more launches you have the more build variations you explore.

---

I believe you are stuck with a wrong mindset like "tests must be exactly the same every time", and I am sure the requested "re-run must launch the same tests" adds nothing to the test coverage or robustness.

---

Hey, if you absolutely want that every "re-run jobs" in GitHub UI to relaunch exactly the same set of jobs, then you can have it: use "workflow id" (or PR id or username or whatever) as a seed for the random, then the matrix builder would yield exactly the same results for a given PR.

[vlsi]

All in all, I have absolutely no need to persuade you.

It is up to you how you test Spock, however, it was really sad that JMeter testing in tr_TR locale was blocked by Spock not supporting tr_TR locale (see #1414 ). It was not me adding Spock tests to JMeter codebase, and I just added locale/timezone/... testing variations.

Of course, everybody makes mistakes, however, the most demotivating outcome for me is that you literally decline adding tests to the testing framework.

I'm closing the PR as I see you are not interested in it.
Feel free to ping me if you have issues/questions regarding github-actions-random-matrix

[leonard84]

Of course, everybody makes mistakes, however, the most demotivating outcome for me is that you literally decline adding tests to the testing framework.

@vlsi When did I decline it? I just said that I don't think it is a good idea to have them on push, but instead to setup a separate workflow that uses schedule. Running these jobs nightly will also run more tests/combinations, as the commit frequency for Spock isn't that high.