[API Proposal]: GC.RefreshMemoryLimit

[cshung]

Background and motivation

In a cloud service scenario, demand comes and goes. To be cost-effective, it is important that our services can scale up and scale down on resource consumption as the demand fluctuates.

While the GC is capable of de-commit unused memory (using aggressive GC manually or waiting until the automatic aging de-commit kicks in), there is no guarantee that the application is not going to allocate more as requests come back. It is much safer if we allow the container to shrink its memory limit after the scale down happened so that the operator can put more instances on the box without worrying about running out of memory just because the GC heuristics believe they are more memory than it is.

As of now, we can change the container limit as we wish, but the GC won't notice. Here I propose an API to notify the GC when that happens.

API Proposal

namespace System;
public static class GC
{
        /// <summary>
        ///
        /// Instructs the Garbage Collector to reconfigure itself by detecting the various memory limits on the system.
        ///
        /// In addition to actual physical memory limit and container limit settings, these configuration settings can be overwritten:
        ///
        /// - GCHeapHardLimit
        /// - GCHeapHardLimitPercent
        /// - GCHeapHardLimitSOH
        /// - GCHeapHardLimitLOH
        /// - GCHeapHardLimitPOH
        /// - GCHeapHardLimitSOHPercent
        /// - GCHeapHardLimitLOHPercent
        /// - GCHeapHardLimitPOHPercent
        ///
        /// Instead of updating the environment variable (which will not be read), these are overridden setting a ulong value in the AppContext.
        ///
        /// For example, you can use AppContext.SetData("GCHeapHardLimit", (ulong) 100 * 1024 * 1024) to override the GCHeapHardLimit to a 100M.
        ///
        /// As a limitation, for 32-bit systems (e.g. x86, ARM) or OSX, if we do not already have a heap hard limit set, we will not establish a new heap hard limit.
        ///
        /// As of now, this API is feature preview only and subject to changes as necessary.
        ///
        /// </summary>
        ///
        /// <returns> A status code to indicate the operation's result:
        /// 0 - The refresh operation succeed
        /// 1 - The operation failed because the new memory limit is too low
        /// 2 - The new hard limit configuration settings is invalid
        ///
        /// We reserve the rights to add more status code as we evolve the implementation.
        ///
        /// </returns>
        [System.Runtime.Versioning.RequiresPreviewFeaturesAttribute("RefreshMemoryLimit is in preview.")]
        public static int RefreshMemoryLimit()
}

API Usage

GC.RefreshMemoryLimit();

Alternative Designs

An alternative design is to have some system mechanism that do that automatically when the container memory limit change happens. This will involve

1. Non-trivial work with various containers that we support, and
2. No way to communicate if the container size reduction is infeasible (*).

The API is meant to be used by people who host services. Therefore it makes sense to make sure this API is CLR Hosting API friendly.

Risks

(*) We reserve the right to refuse the container size reduction when it is infeasible (it might be too tight, leaving the GC no room to maneuver).

But the operator might have already reduced the container size - hmm - we need to do something about that.

Certain things can't easily change - it is easy to say that we wanted to reduce the number of cores as well, but it would be hard to implement changing the number of heaps.

Here is a proof-of-concept prototype, feel free to try it out and leave us feedback.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

In a cloud service scenario, demand comes and goes. To be cost-effective, it is important that our services can scale up and scale down on resource consumption as the demand fluctuates.

While the GC is capable of de-commit unused memory (using aggressive GC manually or waiting until the automatic aging de-commit kicks in), there is no guarantee that the application is not going to allocate more as requests come back. It is much safer if we allow the container to shrink its memory limit after the scale down happened so that the operator can put more instances on the box without worrying about running out of memory just because the GC heuristics believe they are more memory than it is.

As of now, we can change the container limit as we wish, but the GC won't notice. Here I propose an API to notify the GC when that happens.

API Proposal

namespace System;
public static class GC
{
  // ...
  bool RefreshMemoryLimit();
}

API Usage

GC.RefreshMemoryLimit();

Alternative Designs

An alternative design is to have some system mechanism that do that automatically when the container memory limit change happens. This will involve

1. Non-trivial work with various containers that we support, and
2. No way to communicate if the container size reduction is infeasible (*).

Risks

(*) We reserve the right to refuse the container size reduction when it is infeasible (it might be too tight, leaving the GC no room to maneuver).

But the operator might have already reduced the container size - hmm - we need to do something about that.

Certain things can't easily change - it is easy to say that we wanted to reduce the number of cores as well, but it would be hard to implement changing the number of heaps.

Here is a proof-of-concept prototype, feel free to try it out and leave us feedback.

Author:	cshung
Assignees:	-
Labels:	api-suggestion, area-GC-coreclr
Milestone:	Future

[zackliu]

Reloading memory is awesome. I support the manual API instead of auto refreshing.

[davidni]

Related: Environment.ProcessorCount suffers from a similar issue. The result is cached on first access and the runtime assumes the number of processors cannot change, which is false. Relevant source (I think): util.cpp:GetCurrentProcessCpuCount().

See also: Environment.ProcessorCount behavior on Windows

[normj]

This API would be important for us at AWS. For .NET AWS Lambda functions the .NET runtime does not understand the memory limits put in place by the Lambda runtime. If we had this method available to us we could as part of our .NET Lambda startup code take the value from the AWS_LAMBDA_FUNCTION_MEMORY_SIZE environment variable set by the Lambda service into a value for DOTNET_GCHeapHardLimit and then call GC.RefreshMemoryLimit();

[cshung]

@normj, as I tried to implement the logic to read the DOTNET_GCHeapHardLimit during RefreshMemoryLimit, I am hitting issues with CLRConfig. In particular, the current config subsystem is optimized for loading environment variables only once during process startup, fixing that will require ugly hacks and impact the whole subsystem.

After discussing with @AaronRobinsonMSFT, we believe the better approach is to communicate these values through AppContext.SetData API here, will that work for you?

[normj]

@cshung If I understand correctly I could detect what the actual memory limit is based on the Lambda runtime environment and then call something like the following

long lambdaMemoryLimit = determineLambdaMemory();


AppContext.SetData("GCHeapHardLimit", lambdaMemoryLimit);

That seems cleaner then setting an in-process environment variable and then pinging the .NET runtime to go take another look at the environment variables.

[normj]

Out of curiosity assuming the memory value being set is still significantly greater then what is currently being consumed will there be any performance cost setting a new memory limit?

[cshung]

Out of curiosity assuming the memory value being set is still significantly greater than what is currently being consumed will there be any performance cost in setting a new memory limit?

In the short term - the act of setting the limit itself won't cost much, all it does is walk the heap data structures and set some integer fields.

In the long term - obviously - how the application performs after switching the limits depends on what the application does.

One way to think about the problem is to think of this API as switching gears. How fast the car runs depends on a lot of stuff, how much gas you give (i.e. allocations), whether are you running against a slope (i.e. survivals), and what gears you are on. Your car probably won't slow down right away when you tune down the gears, but hopefully, eventually, it will as the car moved longer and the pedals are off.

I'd love to see that after refreshing the memory limit, after a reasonable amount of time to let the current situation change, the application should behave as if the memory limit was specified during startup.

While that's an ideal situation I would like to see, the reality may or may not be what we wanted it to be. The best way to tell is to actually experiment with it.