Different Heap behavior between 5.0 and 6.0. Maybe memory leak?

[lenew]

After upgrade my website from 5.0 to 6.0, memory consumption is about 180% as before.
Is this normal?

---

dotnet-counters on 5.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                               49,432    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Fragmentation (%)                                           1.23 
    GC Heap Size (MB)                                             30    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                24    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                         1,293,576    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        15,028,960    
    IL Bytes Jitted (B)                                    1,524,780    
    LOH Size (B)                                          10,806,240    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       23    
    Number of Assemblies Loaded                                  144    
    Number of Methods Jitted                                  21,915    
    POH (Pinned Object Heap) Size (B)                        375,464    
    ThreadPool Completed Work Item Count (Count / 1 sec)           1    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        4    
    Working Set (MB)                                             240 

---

dotnet-counters on 6.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                                8,168    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Committed Bytes (MB)                                      295    
    GC Fragmentation (%)                                           1.172
    GC Heap Size (MB)                                             88    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                48    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                        25,512,768    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        10,452,680    
    IL Bytes Jitted (B)                                    1,395,898    
    LOH Size (B)                                           7,419,152    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       16    
    Number of Assemblies Loaded                                  141    
    Number of Methods Jitted                                  20,317    
    POH (Pinned Object Heap) Size (B)                        260,872    
    ThreadPool Completed Work Item Count (Count / 1 sec)           2    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        6    
    Time spent in JIT (ms / 1 sec)                                 0    
    Working Set (MB)                                             436 

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[hez2010]

We can't tell the actual problem without a gc root dump. You can try using dotnet-collect to collect a dump and analyze the gc heap following https://docs.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak (See Generate memory dump and Analyze the core dump sections).

[lenew]

Already tried that. I spent a lot time on that but get nothing, due to the enormous amount of objects. Any other method ?
Here is some eeheap on 5.0 and 6.0. 6.0 has 2 heaps and 'Commited Size' is much bigger.

Number of GC Heaps: 1
generation 0 starts at 0x00007F6AE9275F78
generation 1 starts at 0x00007F6AE8CB62E8
generation 2 starts at 0x00007F6AE7FFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007F6AE7FFE000  00007F6AE7FFF000  00007F6AEA0637F8  00007F6AEA9CD000  0x20647f8(33966072)  0x29ce000(43835392)
Large object heap starts at 0x00007F6AF7FFF000
         segment             begin         allocated         committed    allocated size    committed size
00007F6AF7FFE000  00007F6AF7FFF000  00007F6AF8982F50  00007F6AF8983000  0x983f50(9977680)  0x984000(9977856)
Pinned object heap starts at 0x00007F6AFFFFF000
00007F6AFFFFE000  00007F6AFFFFF000  00007F6B0005AAA8  00007F6B00063000  0x5baa8(375464)  0x64000(409600)
Total Allocated Size:              Size: 0x2a441f0 (44319216) bytes.
Total Committed Size:              Size: 0x3352000 (53813248) bytes.
------------------------------
GC Allocated Heap Size:    Size: 0x2a441f0 (44319216) bytes.
GC Committed Heap Size:    Size: 0x3352000 (53813248) bytes.

Number of GC Heaps: 2
------------------------------
Heap 0 (0000564D12C7BDD0)
generation 0 starts at 0x00007FAAA041DB28
generation 1 starts at 0x00007FAAA028DFA8
generation 2 starts at 0x00007FAA9FFFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007FAA9FFFE000  00007FAA9FFFF000  00007FAAA4AE4D38  00007FAAA7B30000  0x4ae5d38(78535992)  0x7b31000(129175552)
Large object heap starts at 0x00007FAC9FFFF000
         segment             begin         allocated         committed    allocated size    committed size
00007FAC9FFFE000  00007FAC9FFFF000  00007FACA03949E8  00007FACA0395000  0x3959e8(3758568)  0x396000(3760128)
Pinned object heap starts at 0x00007FACBFFFF000
00007FACBFFFE000  00007FACBFFFF000  00007FACC000D060  00007FACC0010000  0xe060(57440)  0x11000(69632)
Allocated Heap Size:       Size: 0x4e89780 (82352000) bytes.
Committed Heap Size:       Size: 0x7ec7000 (132935680) bytes.
------------------------------
Heap 1 (0000564D12C82060)
generation 0 starts at 0x00007FABA09D0A18
generation 1 starts at 0x00007FABA08D2888
generation 2 starts at 0x00007FAB9FFFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007FAB9FFFE000  00007FAB9FFFF000  00007FABA4DCFC20  00007FABA817D000  0x4dd0c20(81595424)  0x817e000(135782400)
Large object heap starts at 0x00007FACAFFFF000
         segment             begin         allocated         committed    allocated size    committed size
00007FACAFFFE000  00007FACAFFFF000  00007FACB055B088  00007FACB055C000  0x55c088(5619848)  0x55d000(5623808)
Pinned object heap starts at 0x00007FACCFFFF000
00007FACCFFFE000  00007FACCFFFF000  00007FACD0030AA8  00007FACD0040000  0x31aa8(203432)  0x41000(266240)
Allocated Heap Size:       Size: 0x535e750 (87418704) bytes.
Committed Heap Size:       Size: 0x86db000 (141406208) bytes.
------------------------------
GC Allocated Heap Size:    Size: 0xa1e7ed0 (169770704) bytes.
GC Committed Heap Size:    Size: 0x105a2000 (274341888) bytes.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

After upgrade my website from 5.0 to 6.0, memory consumption is about 180% as before.
Is this normal?

---

dotnet-counters on 5.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                               49,432    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Fragmentation (%)                                           1.23 
    GC Heap Size (MB)                                             30    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                24    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                         1,293,576    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        15,028,960    
    IL Bytes Jitted (B)                                    1,524,780    
    LOH Size (B)                                          10,806,240    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       23    
    Number of Assemblies Loaded                                  144    
    Number of Methods Jitted                                  21,915    
    POH (Pinned Object Heap) Size (B)                        375,464    
    ThreadPool Completed Work Item Count (Count / 1 sec)           1    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        4    
    Working Set (MB)                                             240 

---

dotnet-counters on 6.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                                8,168    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Committed Bytes (MB)                                      295    
    GC Fragmentation (%)                                           1.172
    GC Heap Size (MB)                                             88    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                48    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                        25,512,768    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        10,452,680    
    IL Bytes Jitted (B)                                    1,395,898    
    LOH Size (B)                                           7,419,152    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       16    
    Number of Assemblies Loaded                                  141    
    Number of Methods Jitted                                  20,317    
    POH (Pinned Object Heap) Size (B)                        260,872    
    ThreadPool Completed Work Item Count (Count / 1 sec)           2    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        6    
    Time spent in JIT (ms / 1 sec)                                 0    
    Working Set (MB)                                             436 

Author:	lenew
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[mangod9]

is your app using workstation or server gc? Since the number of heaps has changed (assuming without any code/config changes on your side) wonder if you are hitting: #61674.

[lenew]

Yes, workstation mode in container. The same problem in #61674.
Mem usage back to normal after change "System.GC.Server": false in runtimesconfig.json.

[mangod9]

Thanks @lenew for validating, we hope to backport fix for #61674 to 6.0.