[TEST][FLAKY] test_detection_models

[tqchen]

Seems to be quite frequent in recent PRs, might be related to #7346

https://ci.tlcpack.ai/job/tvm/job/main/495/execution/node/449/log/

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-7360/1/pipeline(this one is timeout)

[tqchen]

cc @zhiics @masahi

[masahi]

So is this a flaky segfault? Not sure how I can reproduce this. Are there common characteristics in nodes that failed (What GPU, which CUDA version etc)? If one node fails, does it fail consistently?

[tqchen]

based on my current read it could also be a timeout. In which case we need to look into if the detection model itself runs too long and if we could build a faster unittest

[masahi]

It shouldn't take more than a few minutes. It should run much faster than TF SSD test, which takes like 20 min (done in a separate thread

tvm/tests/python/frontend/tensorflow/test_forward.py

Lines 3090 to 3095 in dda8f5d

	def test_forward_ssd():
	run_thread = threading.Thread(target=_test_ssd_impl, args=())
	old_stack_size = threading.stack_size(100 * 1024 * 1024)
	run_thread.start()
	run_thread.join()
	threading.stack_size(old_stack_size)

).

Disabled one of rewrites in #7365. The other rewrite added in #7346 should be harmless. We'll see.

[zhiics]

Would this also affect the production test cases although I haven't really seen it.

[zhiics]

Close by @7365

[masahi]

I ran MaskRCNN with that rewrite countless times, I didn't see any problem. Weird.

[altanh]

Before we move too far on from this, can someone compare with our experience in fixing this bug? #7010

If there is some hard to find libomp conflict popping up again, it will crash with "Aborted" like seen the first test

[masahi]

hmm, I see what's common are the use of pytorch, and the way CI dies. But other than that I don't know if two flaky issues are related.

[tqchen]

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/503/pipeline seems #7365 didn't solve the problem