runtime-staging (i.e. iOS and tvOS) legs are timing out in PRs

[ViktorHofer]

Looking at the runtime-staging builds from the last 24 hours, a lot mobile legs, i.e. iOS and tvOS timed out. Those legs already have a quite large timeout (180min). As this is causing PRs to turn red, we should fix this immediately.

@steveisok @akoeplinger @MattGal

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Looking at the runtime-staging builds from the last 24 hours, a lot mobile legs, i.e. iOS and tvOS timed out. Those legs already have a quite large timeout (180min). As this is causing PRs to turn red, we should fix this immediately.

@steveisok @akoeplinger @MattGal

Author:	ViktorHofer
Assignees:	-
Labels:	untriaged, area-Infrastructure-mono
Milestone:	-

[steveisok]

@ViktorHofer when you say a lot, how many are we talking? Also, do some fall in the classification of getting a super slow mac? Or are they just getting slightly slower macs?

@akoeplinger Can you think of any immediate action we can take outside of shutting PR runs down temporarily?

[ViktorHofer]

I did a manual check as I don't know how to query for timed out legs in AzDO or in Kusto. I looked at ~10 builds and half of those timed out.

[ViktorHofer]

Also, do some fall in the classification of getting a super slow mac?

The ones that I looked at weren't slow mac machines. Building the repo didn't take longer than 20-30min but generating the app bundles took over two hours.

[MattGal]

I found these runs while trying to hunt super-slow macs, but I don't see any evidence of real slowness when looking at their stages that should be basically constant, e.g. cloning the repo. There's two major problems IMO:

• They always build all the tests even if they don't send to helix (@steveisok is checking that out)
• Building the tests takes up most of the time, but then when they do send to helix, they need to understand that this is an external, load-sensitive thing (other jobs send these machines work) they're waiting on, so when actually building all these tests and sending to helix they need to accommodate the amount of time this takes... we're talking 15+ GB of apps being created, uploaded to azure storage, downloaded in the pacific northwest to on-premises machines, installed on an emu or device, run, and having the results be re-uploaded, per pipeline in this job. That is a lot!

Some sort of system where we don't build (Number of test assemblies) macOS app packages and only doing so when we actually need them will improve performance a lot.

[ViktorHofer]

Still timing out, i.e. https://github.com/dotnet/runtime/pull/58011/checks?check_run_id=3543565778.

[MattGal]

Still timing out, i.e. https://github.com/dotnet/runtime/pull/58011/checks?check_run_id=3543565778.

Your archives here are just too big (it's something in the 15-20 GB range). I'd like to point out something about one of the listed runs to make my case.

2021-09-08T09:53:59.8783710Z   Sending Job to OSX.1015.Amd64.Open...
2021-09-08T10:32:17.8976280Z   Sent Helix Job; see work items at https://helix.dot.net/api/jobs/cb1b61c0-2f46-42f1-8def-876c46d0777f/workitems?api-version=2019-06-17

It took this hosted macos machine 39 minutes to upload all those zips (wow), but the job it sent? It ran the entire 3 hours, 20 minutes of test work items in a mere 5 minutes, 18 seconds.

WorkItems
| where JobName == "cb1b61c0-2f46-42f1-8def-876c46d0777f"
| extend runtime = Finished-Started
| summarize sum(runtime), min(Queued), max(Finished), entireTime=max(Finished)-min(Queued)

I can't tell you what to do here, but something has to make these payloads smaller somehow. I've long though about combining test assemblies into a batches for app packages, but the real decision will of course be owned by the team.