Memory leak in blazor server application .net 7

[alanma75]

Description

as planned .net 7 upgrade on our existing blazor projects, we noticed memory usage on performance test increased drastically, with dockerized container we can drive the application to OOM exception.
reproduced and profiled locally, it shows the top locking objects are ConcurrentDictionary+Tables<JsonSerializerOptions, WeakReference<JsonSerializerOptions+CachingContext>>
#76548 - this issue was already reported in November 2022.

this is blocking us from upgrading an important application in production to .net 7. I have attached the sample project to demonstrate the OOM exception, with GCHeapHardLimit set to 1GB, 10+ webpage reload of blazor application will get OOM exception.

Reproduction Steps

Launch blazor server application attached, open http://localhost:5000 to load the sample application, refresh the page and observe the memory increasing with Generation2 stack and not be able to free after GC.

using the HeapHardLimit value in the sample (1GB), observe OOM exception in about 10 reloads

Expected behavior

Memory properly GCed

Actual behavior

OOM exception occurs

Regression?

this is new in .net 7 with the C

Known Workarounds

no workarounds

Configuration

.net 7
it happens in all OS
telerikgridhorscrollingapp.zip

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[alanma75]

from dotMemory profiling

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

as planned .net 7 upgrade on our existing blazor projects, we noticed memory usage on performance test increased drastically, with dockerized container we can drive the application to OOM exception.
reproduced and profiled locally, it shows the top locking objects are ConcurrentDictionary+Tables<JsonSerializerOptions, WeakReference<JsonSerializerOptions+CachingContext>>
#76548 - this issue was already reported in November 2022.

this is blocking us from upgrading an important application in production to .net 7. I have attached the sample project to demonstrate the OOM exception, with GCHeapHardLimit set to 1GB, 10+ webpage reload of blazor application will get OOM exception.

Reproduction Steps

Launch blazor server application attached, open http://localhost:5000 to load the sample application, refresh the page and observe the memory increasing with Generation2 stack and not be able to free after GC.

using the HeapHardLimit value in the sample (1GB), observe OOM exception in about 10 reloads

Expected behavior

Memory properly GCed

Actual behavior

OOM exception occurs

Regression?

this is new in .net 7 with the C

Known Workarounds

no workarounds

Configuration

.net 7
it happens in all OS
telerikgridhorscrollingapp.zip

Other information

No response

Author:	alanma75
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[jeffhandley]

I was able to install the 30-day trial of the Telerik UI for Blazor and update the project to use Telerik.UI.for.Blazor.Trial version 3.7.0 to reproduce the issue. Note to @dotnet/area-system-text-json folks that I also tried the repro against a net8.0 build that incorporates #76607, and memory still climbs with each reload. Also note that the repro app produces other errors when (re)loading the page, but that does not interfere with the reported issue.

There's nothing in this repro app itself that uses System.Text.Json directly, which means the actual repro here is down in the Telerik layer's use of JSON serialization.

[alanma75]

the main change in the fix is not having the ConcurrentDictionary with key JsonSerializerOptions, that hold a strong reference of each instance in blazor circuit (session). and the size of that JsonSerializerOptions can get big in our test.

the new Array of weakReference will not have such issue, so under normal GC, the heap will be able to reduce, from my read of the code changes.

[eiriktsarpalis]

I haven't checked the repro yet, but this sounds somewhat different to the report in #76548. The original issue reported increased memory consumption when the application remained idle, however the dangling cache entries would get immediately evicted on the first subsequent serialization operation. This case sounds like an unbounded memory leak, which is a problem of a whole new magnitude.

[jeffhandley]

I used the repro to capture memory dumps. I've shared them offline.

[eiriktsarpalis]

I ran the repro locally using both .NET 7 and .NET 8 and can confirm that the shared JsonSerializerOptions cache is the primary contributor to memory consumption in .NET 7:

Even though memory consumption also climbs in .NET 8, it climbs at a slower rate and is not caused by JsonSerializerOptions:

Instead it appears to be caused by MemoryCache instances (which are eventually evicted):

So it seems that the reproducing app might be impacted by two different types of memory leaks.