Proposal for minimalistic imaging platform

[dajuric]

Proposal for minimalistic imaging platform

.NET does not provide a truly portable image format nor portable image IO operations.
For example System.Drawing.Bitmap nor WPF's BitmapSource and does not provide generic interface which imposes compile-time constraints. EmguCV's generic image provides the generic interface but is heavily coupled with OpenCV.

All mentioned image types do not provide a unified interoperability needed when mixing image-processing algorithms from different libraries.

Goals:

• to create a portable module for image reading/writing
• to create a portable, slim image format such as DotImaging has (native array Bgr<byte>[,])
• to create system drawing primitives - such as in DotImaging.Primitives2D
• optionally to create portable video reading and web-camera access module

Proposal

To leverage existing structure and components of DotImaging framework, but to replace the backing OpenCV part needed for image IO.
The first step would be to create a portable image reading/writing module with the most simplistic API (e.g. ImageIO.LoadColor(), LoadGray()...).
Options:

• a wrapper arround LibGD (@ Proposal for Cross-plat Server-side Image Manipulation Library #14706)
• trim down and reuse the existing platform specific module of: Eto.Forms - includes drawing capabilities as well, or Splat.

Related issues

Proposal for Cross-plat Server-side Image Manipulation Library - proposal of much broader scale - this proposal resolves the core issue and levaes out separates image processing part managed by other libraries Accord.NET, Accord.NET Extensions, EmguCV.

.NET core should have primitive drawing types - already done in DotImaging.Primitives2D

[shmuelie]

Just a note about storing the image data: Don't do it in managed memory. I've been doing lots of image processing and writing my own "Bitmap" types. I learnt this leading the hard way.

I'll have to see if I can show any of my work or it's stuck under NDA

[dajuric]

Can you elaborate why not to store the image data in the managed memory ?

I have also been doing a lot of image processing and created two frameworks (Accord.NET Extensions and DotImaging. After many iterations I have concluded that it is the easiest to store the data into managed memory and if you need speed to temporary lock the array using "unmanaged image" (DotImaging-generic image).

The reasons why I am using managed container are the following:

• it leverages .NET native array structure
• more easier to debug
• does not introduce any new classes
• can be temporary locked if you need speed (DotImaging-generic image)
• portable (and can be serialized as ordinary array)
• can be leveraged by managed OpenCL implementation (Cudafy)

I am just putting some arguments why I chose this approach (just want to have a constructive conversation); I am curious why did you choose unmanaged container ? (omitting maybe speed and IO)

Thanks for reply and the effort tho show your work.

[shmuelie]

Sorry about the bluntness and stating what really is a "in my case" as an "always". Replying to GitHub issues at 3 AM is never a good idea.

First a quick overview of my architecture:
UnmanagedMemory: This class is in charge of managing unmanaged memory. It handles allocation, deallocation, tracking usage, tracking total usage, preventing leaks, and provides friendly exceptions.
BitmapPlus: Pretty much a raw image in memory. It uses UnmanagedMemory internally to store the data. Provides methods for manipulating the data.
WriteableBitmapPlus: A custom implementation of WriteableBitmap, using UnmanagedMemory internally.

One big reason I use unmanaged memory is that, at least in my use cases, images work against the GC. Images are large and long lived objects. At the same time when I don't need them any more I can't have their memory just lying around, otherwise I hit OOM very quickly. If they are put in the standard heap they'll probably make it to Gen 2. Which means they won't go away for a while and when they do the user feels it. If they are put on the LOH they will almost never be reclaimed and/or cause massive fragmentation that will speed up the time to OOM.

Another reason, though I admit this may be very particular to my use case, is that that I repeatedly find myself passing the images to unmanaged land. The constant need to fix the backing array's location also (once again) messes with the GC and causes issues.

Those are the top issues I've run into that have caused me to switch to unmanaged memory.

As to the points you bring up:

it leverages .NET native array structure

I admit the native "unsafe" code is more complex to understand but it's not impossible.

more easier to debug

I admit seeing the actual values of the memory is harder (though VS has the lovely memory windows). However, in general when debugging I'm more concerned with the algorithm than the actual bytes in the image.

does not introduce any new classes

not sure if this is really an advantage in my opinion

can be temporary locked if you need speed

Flip side:unmanaged can copy into managed code if need to (Which WriteableBitmapPlus does in fact)

portable

You got me there.

can be leveraged by managed OpenCL implementation

And unmanaged can be leveraged by SharpDX (a managed DirectX implementation)

Hope this explains my thinking!

[dajuric]

Hi Samuel,

thanks for the reply. No need to apologize, I just wanted to show you the reasons why I chose the managed approach :)

garbage collection

That bothered me too, and that is why my first implementation was unmanaged image Image<TColor>. I do not know much about GC design, but I can tell you what I have learned from experience.

The algorithms I write or use are related to video processing where each frame is consumed e.g. Accord.NET Extensions . Yes, the image objects are big and they do go into 2nd generation, but a new design of GC (from .NET 4.5.1 Article, Options) handles those objects more efficiently, and it restores the consumed memory. When processing video, I need only one image allocation is needed (for frame), so you do not experience any lag or performance issues (the memory allocation remains constant).

Moreover, you can call GC.Collect() to restore the memory if you need to (Options) (I do not need that).

Plus and minuses regarding unamanged representation (for me):

• Pluses:
  • they can be deallocated when needed
  • many algortihms and IO operations are unmanaged so this is more convinient
• Minuses:
  • allocation and deallocation of many unamanged images is slower than managed counterpart (tested using AForge.NET)
  • you need to take responibility to deallocate such objects which seems unatural in C#
  • the backend behind is more complex (your framework has a unmanaged image class, a manager and a wrapper); and more xomplex designs (I am not saying yours) is potentionally more error prone. I had simmilar architecture (before), but I found the design was complex and buggy.
  • you can not return to a managed world without actual data copy, but.. (next statement)
  • you can overcome the managed-unamanged interop by temporary locking an object (what problems does it cause if an object is temporary locked ?):

var image = new Bgr<byte>[480, 640]; using(var uImg = image.Lock()) { unmanagedFunc(uImg.ImageData, uImg.Stride, ...); }

So, to conclude: I see unmanaged / unsafe C# features more like extensions not as fully integrated feature as the interpoerability is not two-way, meaning that you can not cast unamanaged data into managed objects + pointers and generic do not come along.

In order to leverage the best from both sides: the simplicity of managed world and speed of pointer processing, I have created the framework where managed native array is the central element (similar as in MATLAB) and in the same time fast image processing and interop with other libraries is enabled through slim unamanged generic image class.

I am not saying that this approach is better, but I find it more user-friendly and less error-prone.
I find it interesting, useful and fun to have such conversation, so please do not hesitate to comment; I am willing to learn.

[shmuelie]

I do agree that features like unmanaged memory and unsafe code are less user friendly than "normal" C#. However, for base framework code they're not uncommon. The BCL is full of interop and that's fine because the average user is not messing with the interop code. They're using the nice managed APIs that are above it.

The .NET 4.5.1 improvements sound great! Sadly at work we're still on 4.0, though we are looking into moving to 4.5.

allocation and deallocation of many unamanged images is slower than managed counterpart (tested using AForge.NET)

I wouldn't mind seeing those tests since in my testing unmanaged allocation and deallocation was much faster.

what problems does it cause if an object is temporary locked ?

When you lock an object in place (whether using the fixed keyword or using GCHandle) the GC has to work around the object when compacting/defragmenting the heap. This both makes it harder for the GC and can make the job of compacting/defragmenting less efficient.

you need to take responibility to deallocate such objects which seems unatural in C#

It is on the weirder side to do so in .NET but IDisposable usage is common and calling GC.Collect() is really the same thing.

the backend behind is more complex (your framework has a unmanaged image class, a manager and a wrapper); and more xomplex designs (I am not saying yours) is potentionally more error prone. I had simmilar architecture (before), but I found the design was complex and buggy.

It is true that the greater the complexity the higher the bug count. Most simple designs work though because someone else is doing the complex work.

I'm loving this conversation :)

[shmuelie]

@terrajobst think you could rope in a GC person? For all the time I've spent in .NET I'm really not a GC specialist.

[dajuric]

Hi Samuel,

thanks for the response.

Mind the tests.

If I currently remember, I allocated several hundred times anUnmanagedImage from AForge.NET and native .NET array. After each call I disposed an unmanaged image (I do not recall what I did with GC.Collect()).

One application example which benefited from managed heap is Fast template matching - fast object detection. It creates several images from each frame, and each time those images are deallocated. One of the first versions was written with managed wrapper that called (malloc ?) and than it freed the memory.
When I rewrote the code using managed arrays and pinning the performance actually increased (I was also afraid that the performance will drop).

Temporary object locking

Yes I am aware of this, that is why I stressed temporary (you only lock the object when some operation should be performed). I have not measured the impact on such scenarios, but from my experience, I would say it is not much.

Simple design

I wanted to tell clean, nice (as in MATLAB) where operations on matrices are fast, but the user is not exposed to pointers, etc. It is interesting to see that if you put a little bit more stress on design vs. performance, the whole perspective changes; another things are optimized :)

I am glad we could exchange opinions. Looking forward for more.

[shmuelie]

Allocations
To help my tests were:

Unmanaged:

using (UnmanagedMemory mem = new UnmanagedMemory(short.MaxValue))
{
}

Managed:

byte[] b = new byte[short.MaxValue];
b[0] = 1;
b = null;
GC.Collect(0);

Each of those tests were timed using the Stopwatch class, run 1000 times, and then the average was taken. The results were:

Unmanaged: 00:00:00.0000017
Managed: 00:00:00.0000149

On one hand unmanaged is faster, on the other we're talking about such tiny amounts of time...

Code Design

I admit what is "clean" to me is not clean to others. For example I find Regex and IL perfectly readable. However, I know most don't and so I'm slowly replacing code using Reflection.Emit to use Linq.Expressions.

[dajuric]

Thanks for the reply and tests.

UnmanagedMemory is your class or ?
Have you tried with bigger array sizes (this size corresponds to 256x256 image) ?

[shmuelie]

UnmanagedMemory is the class I mentioned above. It's really just a wrapper around LocalAlloc and Marshal.FreeHGlobal.

Ran some more tests:

Size: 1073741824
Unmanaged: 00:00:00.0005554
Managed:      00:00:00.0001652

I then modified the test to do two allocation and then deallocate. Had to do this since .NET was giving me issues with large arrays.

Size (per allocation): int.MaxValue / 2
Unmanaged: 00:00:00.0009885
Managed:      00:00:00.0012521

So very interestingly it seems very dependent on what size you're allocating on if Unmanaged or Managed is faster...

[kinchungwong]

Hi all,

I would like to share some notes I have collected on the issue of imaging platform on .NET.
Most of the points in my notes are trivial or common knowledge. Feel free to skim my notes and just look for points which you find interesting.

1. https://gist.github.com/kinchungwong/155c9046ba3f1fecf49a
2. http://programmers.stackexchange.com/a/210845

The list of requirements for an imaging platform is very long. Many of the requirements are conflicting, which means they cannot be all satisfied. Thus, by choosing to satisfy some requirements and discarding others, we effectively decide who will be the subset of users (audience) of the imaging platform.

I think it is important to decide if certain goals should be included or excluded early-on.

That said, from my notes you will find that there are two interoperability techniques that we can always rely on as a fallback:

• Deep copy, with or without conversion.
• Slice copy, with or without conversion.
  • Example: System.Windows.Media.Imaging.BitmapSource.CopyPixels method

Performance is important but so are many other considerations. Also, performance depends on a lot of factors such as OS and CPU. If we made performance choices too early, we may find that the choices we made were sub-optimal whenever the execution environment or use-cases is changed.

[phillip-haydon]

I don't believe imaging belongs in corefx. Imaging is hugely complex and no minimalist library in core will ever satisfy the requirements of imaging.

my 2c

[dajuric]

It depends what you define by minimalistic. The original proposal seems pretty viable.
The thing is there are no portable, common image format, nor IO operations that support it. I would be happy if that is resolved (see the first post).

[JimBobSquarePants]

I've been working very hard to provide both with the new ImageSharp

https://github.com/JimBobSquarePants/ImageSharp

It's all managed code running on CoreFX. There's still a lot to do but I've established a pretty useful set of classes so far.

Update Switched the urls to the new repo.