Out of disk space building on Docker NanoServer

[BruceForstall]

stress-http and stress-ssl jobs are failing to build due to out of disk space, e.g.:

D:\a\1\s\artifacts\obj\coreclr\Linux.x64.Release\crossgen\src\utilcode\stdafx.utilcode_dac.cpp(1,1): fatal error C1085: Cannot write precompiled header file: 'D:/a/1/s/artifacts/obj/coreclr/Linux.x64.Release/crossgen/src/utilcode/Release/stdafx.utilcode_dac.pch': There is not enough space on the disk. [D:\a\1\s\artifacts\obj\coreclr\Linux.x64.Release\crossgen\src\utilcode\utilcode_dac.vcxproj]
  D:\a\1\s\artifacts\obj\coreclr\Linux.x64.Release\crossgen\src\md\runtime\Release\mdruntime_dac.lib : fatal error LNK1180: insufficient disk space to complete link [D:\a\1\s\artifacts\obj\coreclr\Linux.x64.Release\crossgen\src\md\runtime\mdruntime_dac.vcxproj]
##[error]BUILD: Error: cross-arch components build failed. Refer to the build log files for details.

https://dev.azure.com/dnceng/public/_build/results?buildId=592971&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6

https://dev.azure.com/dnceng/public/_build/results?buildId=592970&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6

@dotnet/runtime-infrastructure

[jaredpar]

@MattGal

[MattGal]

Hi guys.

First I want to say I'm impressed you filled the whole disk in only 24 minutes, that's impressive. Also, there's nothing about the 2nd job that says disk space full, just a 2 hour timeout?

Anyways, I've had conversations with the MMS folks recently around similar issues in MacOS hosted pools, and I learned that the amount of free space goes up and down randomly with the payloads installed and no one actually knows what it is so they can't tell you how to keep your build from blowing up the machine.

Your options include:

• Move to buildpool.windows Helix queues, they have huge disks; this is not a great time to go there with the core-reduction stuff we've done, and if you need docker on the build agent I don't think we have that combination ready to go (based on the pipeline's title, probably?)

• Remove any file you like from the build agent, but only if it's hosted This is not good behavior in general and I'd recommend against it, but it's literally what AzDO suggested. I'd recommend an explicit check on the behavior to decide you're not in a Helix machine since we don't recycle machines every build (nor would you need to do this) if you might ever do so.

[BruceForstall]

@dotnet/ncl Can someone look at this? It's failing every run.

[safern]

I think we should just move these builds out of the Hosted pools as they use considerable disk space.

cc: @eiriktsarpalis

[jaredpar]

Agree. This is the only way we're going to get the desired reliability here.

[karelz]

@alnikola can you please help us here?
Why does the stress cause so much trouble?

[eiriktsarpalis]

FWIW this is happening when building the clr and libraries on the host machine.

[karelz]

Are the machines supposed to handle the build?
Do we use the right design for this workflow?

[eiriktsarpalis]

The failures seem to have started abruptly 10 days ago, which suggests that one change may have broken it.

…

On Tue, 14 Apr 2020, 22:01 Karel Zikmund, ***@***.***> wrote: Are the machines supposed to handle the build? Do we use the right design for this workflow? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVO3M4QKLU354DNXEMPPVDRMTFK5ANCNFSM4ME5ZZDA> .