Fix deadlock due to infinite retry

[thampiotr]

PR Description

Fixes a deadlock when using modules and running a lot of components (or being starved of CPU).

The symptom was getting a repeated "failed to submit node for evaluation - Alloy is likely overloaded and cannot keep up with evaluating components - will retry" logs and slowly growing memory because of getting many calls to /metrics that are stuck, until OOMKill in one user's case.

Notes to the Reviewer

• When EvaluateDependants is called, we submit the dependants of a node that published new exports to a global worker pool for evaluation, so the new export changes can propagate through the graph.
• The worker pool has a queue that is bounded to limit the memory issues.
• In extreme cases where there are, for example, 3k components on one Alloy instance, the worker pool's default queue of 1024 items can become full.
• We don't error when queue is full, we instead retry with backoff with hope that it's temporary overload due to a slow component evaluation / resource starvation and we will manage later.
• We retry indefinitely.
• There are other code paths (see below) that interact with the loader on component evaluation and will require the write mutex to complete. Only once these code paths get the write lock on loader mutex, will they complete and create some more space in the worker pool queue.
• The problem is that these code paths are stuck forever, because we hold on to the loader read mutex while we backoff. We have a deadlock where the backoff is holding the read mutex, unable to add more tasks to the full queue, but the evaluation tasks queue is not draining because it's waiting for the write mutex.

The code path that grabs the loader write mutex:

• Every custom component has its own controller and loader.
• When these custom components are evaluated (e.g. when the component definition changes because of import.http found a new definition), they will call LoadBody which will call loadSource which will call applyLoaderConfig which will call loader.Apply which will grab the write lock of the loader mut.

I think that the default queue size of 1024 is still a good default, more than enough for normal, correct operation. So choosing not to expose it as a configuration knob in order to keep Alloy simpler.

I also think that the backoff behaviour doesn't need to be changed other than aborting it after some time. This way we can recover from other deadlocks while printing a good number of error logs in the process.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[wildum]

LGTM, nice catch