Keep service tests green 💚 #1359
Description
We want our service tests to hit the real services. This makes some of them unreliable – out of necessity, not sloppiness. There's a long discussion of the test strategy in #927.
I’d like to use this issue as a place to identify and fix flaky service tests.
IcedFrisby/IcedFrisby#71 will allow us to set a per-test timeout() and per-test retry(), which should allow us to keep flaky tests green most of the time.
Of course we want the requests to be fast. Right now they fail after 2 seconds, which is a fairly long time to be waiting for a badge. Since readme images are served through GitHub’s proxy (camp) which has its own timeout, perhaps we should choose a threshold in relation to that. IcedFrisby has a separate method .expectMaxResponseTime() which we could use to assert the response time. We could also generate a tidy report of slow badges and print it as part of our daily run. An often-red build is not the best way to accomplish this! 😄