com.example.spanner.SpannerSampleIT: testSample failed

[flaky-bot]

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.

---

commit: d7cfb2d
buildURL: Build Status, Sponge
status: failed

Test output

com.google.common.truth.AssertionErrorWithFacts: 
expected to contain:
    1 1 NULL
but was:
    1 1 100000
    1 2 NULL
    2 1 NULL
    2 2 500000
    2 3 NULL
    Closed client
at com.example.spanner.SpannerSampleIT.testSample(SpannerSampleIT.java:137)

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (d7cfb2d), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[olavloite]

This problem is probably caused because of temporary slowness of either the backend or the client itself. What this test does is the following:

1. Insert data into table A.
2. Wait 16 seconds.
3. Update the data in table A.
4. Execute a query on table A that reads with an exact staleness of 15 seconds.

The above steps should normally produce the data that was in the table after the insert in step 1, and without the update in step 3. However, if for some reason the client is slowed down a lot between step 3 and 4, or the backend is very slow to respond to the query request in step 4, it could be that more than 15 seconds has elapsed since the update in step 3 was executed. That will produce unexpected results for the query in step 4.

Fixing this in a 100% watertight way without changing the sample itself seems to be impossible. I would suggest:

1. The above will probably happen very infrequently (I can't remember having seen this before, and this code has been unchanged for very long time). We could just close it for now and assume that it is very unlikely to come back, and if it does; re-evaluate.
2. Change the sample to a standalone sample file and standalone test. The test could then retry if this specific error occurs, and potentially also check the time it took between step 3 and 4 which is where I suspect that the error is coming from.

@skuruppu @thiagotnunes Any thoughts on the above?

[thiagotnunes]

Hey @olavloite thanks for the analysis.

I would vote for option (2), since it brings us closer to where we would like the samples to be. I would, however, add a small change: increase the waiting time in step (2) to reduce the flaky behaviour.

[thiagotnunes]

Oh wait, I just realised these failures are in the java-docs-samples, so these samples are outdated. We should remove the cloud-client samples and associated tests from here if possible. I will discuss this with @skuruppu

[zoercai]

@thiagotnunes any updates for this?