FreeBSD CI workers got ENOMEM randomly

[iblislin]

ERROR (unhandled task failure): write: not enough memory (ENOMEM)
Stacktrace:
 [1] try_yieldto at ./event.jl:189 [inlined]
 [2] wait() at ./event.jl:248
 [3] uv_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:811
 [4] unsafe_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:832
 [5] write(::Pipe, ::String) at ./strings/string.jl:71
 [6] write_cookie(::Base.Process) at ./distributed/cluster.jl:1120
 [7] launch(::Base.Distributed.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./distributed/managers.jl:328
 [8] (::Base.Distributed.##39#42{Base.Distributed.LocalManager,Dict{Any,Any}})() at ./event.jl:73

But the available memory observed from top on each work is enough.

Not sure it's bsd only issue or not.

Ref builds:

1. https://julia.iblis.cnmc.tw/#/builders/1/builds/903
2. https://julia.iblis.cnmc.tw/#/builders/1/builds/887
3. https://julia.iblis.cnmc.tw/#/builders/1/builds/891
4. https://julia.iblis.cnmc.tw/#/builders/1/builds/772
   ...

[ararslan]

I believe @fredrikekre said he's seen this happen locally on Ubuntu as well. This is triggered by the parallel tests, right?

[fredrikekre]

I think it was the same, can not give any more info until getting back to work in couple of weeks though, since I have only seen it there.

[iblislin]

I bumped JULIA_TEST_MAXRSS_MB to 1000 now.
hope it's helpful. 🙏

[iblislin]

some build reports don't contain the statistic testsuit

• https://julia.iblis.cnmc.tw/#/builders/1/builds/988
• https://julia.iblis.cnmc.tw/#/builders/1/builds/986
• https://julia.iblis.cnmc.tw/#/builders/1/builds/983

Any unreasonable memoy usage in that testsuit?

[ararslan]

This is odd, I haven't seen this locally on FreeBSD with JULIA_TEST_MAXRSS_MB=600 on a fairly limited memory machine. Seems like setting it to 600 would be more likely to trigger ENOMEM than 1000?

[iamed2]

I have seen this when SIGINTing while doing parallel work with Dispatcher. Maybe processes are being killed?

[iblislin]

dmesg keep showing this:

Aug  8 06:17:17 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 06:17:19 ionic last message repeated 2 times
Aug  8 06:17:20 ionic kernel: pid 82904 (julia), uid 1001: exited on signal 6 (core dumped)
Aug  8 06:17:21 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 07:34:07 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 07:34:10 ionic last message repeated 3 times
Aug  8 08:52:07 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 08:52:10 ionic last message repeated 3 times
Aug  8 10:10:26 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 10:10:29 ionic last message repeated 3 times

My local machine also have these messages, but do not raise any ENOMEM.

I have seen this when SIGINTing while doing parallel work with Dispatcher. Maybe processes are being killed?

any suggestion/tools about how to obseving that?

[iblislin]

after bumped MAX_RSS up, the frequence of ENOMEM ⬇️ , but freezing ⬆️

• https://julia.iblis.cnmc.tw/#/builders/1/builds/1005
• https://julia.iblis.cnmc.tw/#/builders/1/builds/1001

this ENOMEM is related to freezing? 😱

[mbauman]

I don't think we're seeing this anymore — is this still relevant with the new buildbot infrastructure?

[ararslan]

We still get freezes occasionally, but not as often as we used to, and I haven't seen the ENOMEM in a long time, probably since we switched to the new buildbots.

[ViralBShah]

Reopen if necessary?