Implement pluggable memory management

[antonfirsov]

Problem

• In some use cases the default ArrayPool behavior does not fit user needs (see Encoding an animated image requires all frames to be loaded into memory, can lead to OutOfMemoryExceptions #222). Users should be able to configure + reset + disable pools.
• Some users may want to back Image<T> by memory resources other than managed memory. (Eg. native buffers).

Solution

Implement a pluggable MemoryManager, make the default pooling memory manager configurable.

Update:
feature/memory-manager is the WIP branch for this refactor. It's having #431 merged. If anyone wishes to contribute, please file your PR-s against that branch. The tasks below are being updated according to the progress on that branch.

Tasks for beta-3

• Refactor all raw ArrayPool and PixelDataPool usages to use Buffer and Buffer2D instead.
• Buffer2D<T> should compose Buffer<T> (as IBuffer<T>) instead of inheriting it
• Move the responsibility of providing Buffer<T> instances to a generic MemoryManager class. Refactor all new Buffer<T> and new Buffer2D<T> instantiations to call factory method on MemoryManager instead.
• We need to cover all the encoders/decoders with proper regression tests before continuing the refactor work. (Working on this - would be filed as a separate PR)
• Extensive test coverage for ArrayPoolMemoryManager
• Allocations in DefaultPixelBlenders<TPixel> should use MemoryManager taken from the outside
• Replace the legacy PixelArea<TPixel> class with BufferArea<T> + extensions methods working on BufferArea<TPIxel> and/or IBuffer2D<TPixel> using PixelOperations<TPixel>
• Replace all most PixelAccessor<TPixel> usages with Buffer2D<T>. Drop the IBuffer2D<T> interface. - the full removal could be done later.
• MemoryManager should construct IBuffer<T> and IManagedByteBuffer instead of Buffer<T>, codecs should use IManagedByteBuffer.Array
• Based on user feedback, there also might be some strange, unexpected leaking when images are created/destroyed continously. Doing some load testing, identifying+understanding this behavior is inevitable!

Probably 1.0

• Passing custom MemoryManager to codecs

Most likely post- 1.0:

• Passing custom MemoryManager to Mutate() and Clone() (---> Allow passing custom Configuration instances to processors #650)
• Investigate new System.Memory API-s: Memory<T> and
  OwnedMemory<T>. Find a way to integrate these classes into ImageSharp.Memory.
• Image.Wrap<T>(Memory<T> memory) or similar
• Implement an MMFMemoryManager which uses Memory Mapped Files to reduce the pressure on system memory when processing large images.

[vpenades]

About processing large images, there's a common issue in .Net which is that very large memory chunks are not moved during a GC, which can result in out of memoty exceptions even when there's enough memory available.

A solution for this I found in the past was to use a low level bitmap primitive that split images into chunks of 64x64 pixels, all wrapped with a generic IBitmap interface. That way, relatively small images, or images that would require very fast procesing (requeted upon image creation) would use an IBitmap that simply wraps an Array. Large images would use instead an IBitmap implementation that stores multiple chunks.

[Toxantron]

@vpenades the reason the CLR does not move objects larger than ~85kB is the negative performance impact. Objects that big are stored in the LOH. Splitting the images into movable chunks disables that optimization and will negatively influence performance.

A large image still reserves the some amount of memory if split into chunks. Using the power of streams and the filesystem seems to be the better solution here.

[vpenades]

@Toxantron Not everybody is looking for performance; some years ago I was tasked to process images on the size of about 20.000 x 15.000 pixels, which allocated around 1gb RAM, in a single chunk. We also discovered that when compiling for x86, even if theoretically you can allocate 4gb, the Net Framework runtime was actually limited to 2gb only, essentially limiting the load of a single image of such size.

In practice, it was nearly impossible to load such image, because any time we had any object fragmenting the LOH, it was impossible to allocate more than 800 or 900mb in a single chunk, and ended with out of memory exceptions.

We looked for every single image library available at the time (+5 years), and we found all libraries of the time allocated a single chunk.

We ended developing our own custom image library which splitted images in chunks of less than 64kb, to allow the runtime to defragment memory, at the cost of performance.

It is true that time has passed, memory is not that scarce as it was 5 years ago and x64 is commonplace these days... but don't assume people is going to use image libraries the same way, or for the same reasons as you use them. For example, there's lots of areas related to scientific analysis, or design artists working with large form factors, which work with extremely large images.

[dlemstra]

@vpenades Did you take a look a ImageMagick back then? This can handle extremely large images.

p.s. I am one of the authors.

[antonfirsov]

@vpenades An MMF-based memory manager could achieve the same tradoff (running slower, but using less memory) in a more simple way.

BTW there is a new property GCSettings.LargeObjectCompactionMode since 4.5.1 to address the LOH fragmentation issue.

[Toxantron]

@vpenades thank you for the clarification.

@antonfirsov I fand that value as well. It comes with performance impacts, but at least we can leave it to the GC then and not implement our own version.

[vpenades]

@dlemstra I think we tried, but if I recall correctly, we had std:boost compiling issues with other third party libraries.

@antonfirsov that's great to know the GC also has the compact large objects feature, I wish I had it a while back.

@Toxantron 😄

[JulianRooze]

Some feedback on this: we currently use ImageSharp for all our image resizing needs in a cloud application and the optimization of speed over memory usage seems to run counter to the idea of horizontal scaling, because of how it holds onto memory. We'd like to provision many containers with a 2 GB memory limit, but currently if we do that, we'd end up with all those containers using 2 GB all of the time. Currently, we run only a few containers limited to 4 GB and they frequently get OOM killed by Docker, even though I already limited image resizing concurrency. This high memory usage seems to be mostly caused by a few outliers in image sizes, most images we process are in the 500 KB range, but some are 10-20 MB, which then consumes 500+ MB of memory just to decode that gets cached indefinitely. Do that concurrently and memory quickly starts to climb to 2-4 GB which never gets released.

From this issue, I can tell you're working on making it configurable and pluggable, but it sounds like a lot of very fundamental work and somewhat long-term. Is there any chance you would accept a more short term quick-fix? Correct me if I'm wrong here, but from what I can tell, most of our memory usage comes from the caching of Block8x8 which is done by the Jpeg decoder, and the memory is provided by the PixelDataPool. If you were to add properties to Configuration that allowed you to set the maxArrayLength / maxArraysPerBucker that get passed to ArrayPool.Create, you could turn off/tweak array pooling for a common scenario.

Something like:

Configuration.Default.Memory.PixelDataPool.MaxArrayLength = 0;
Configuration.Default.Memory.PixelDataPool.MaxArrays = 0;

I understand that this would expose some internal implementation details, but I think (haven't tried it yet, might later) it would also solve some real issues.

[antonfirsov]

@JulianRooze I plan to realize my roadmap in February, so it will be part of our 1.0 release or maybe even some beta. Is this late for you?
The only thing I can propose as a quickfix is is to add a temporal configuration API to control the behavior of PixelDataPool for beta-3:

[Obsolete("This is a temporal API, use it for your own risk!")]
public static class PoolConfiguration
{
    [Obsolete("This is a temporal API, use it for your own risk!")]
    public static int MaximumPooledBufferSizeInBytes { get; set; }
}

@JimBobSquarePants what do you think?