Best way to convert an Image buffer Uint8List to cv.Mat

[gau-nernst]

Typically we can access raw image data from other library as an Uint8List object. Assuming we know the format of the image (e.g. RGB uint8), what is the best (fast and convenient) way to convert it to OpenCV Mat? Some of the approaches I am thinking of:

• Create an empty Mat with the required size. Iterate over Uint8List and copy data over. For some formats like YUV_420sp, it might be a bit tricky to copy the data correctly.
• Encode in a lossless format (e.g. PNG) and use cv.imdecode() -> seems to be the most convenient, but there is overhead in compression-decompression.
• Allocate C memory with Dart FFI, copy data over, create Mat from C pointer. Need to release the memory after use.

[rainyl]

UPDATE 2024.8.30:

Since v1.2.0, the recommended ways are Mat.fromVec and Mat.fromList

Original:

If I'm not mistaken, the raw image data contains image header infos, just like the data read from file directly, if so, I'm afraid the only way is cv.imdecode(), Mat object in c++ opencv need to be constructed by constructors or methods like cv::imread, which is different in python (on python level, all Mat object are actually numpy.ndarray, so can be passed to opencv functions directly, but even so, it most likely a new Mat object will be created in native codes, I didn't dive into it, so just "most likely").

But if the raw image data are raw pixel values, it is possible to construct from values, for now the construction from UInt8List is not implemented, but the following procedure is ok:

// 2D  or 3D raw pix values
final data = ...;
final int rows = ...;
final int cols = ...;
final m = cv.Mat.create(rows, cols, type);
for (int i = 0; i < rows; i++){
  for (int j = 0; j < rows; j++){
    m.setValue<int>(i, j, data[i][j]);
  }
}

So, I think the best way is cv.imdecode(), please point it out if I'm wrong.

[gau-nernst]

No problem at all. Indeed copy is unavoidable. Right now I have best experience with the following code

cv.Mat imageCv = cv.Mat.create(...)  // set correct width, height, and number of channels
imageCv.data.setAll(0, buffer)  // where buffer is Uint8List

[rainyl]

Okay.

BTW, I am confused why you set data with buffer, as far as I know, data is a getter of Mat and will get data from native pointer directly, so change the elements inside it won't change the data of mat.

[gau-nernst]

It works for me. You can try on your own to see. cv.Mat.data is Uint8List in Dart, which allows me to use .setAll() (or .setRange()) to copy the data to OpenCv Mat directly. This is significantly faster than using cv.Mat.setValue(), probably because .setValue() is an FFI call which has a lot of overhead.

[rainyl]

okay. I read the doc of astypedlist and noticed that it will return a reference, so it makes sense.😂

[TheoNanu]

Hello, I am struggling to apply a gaussian blur on an image from the camera. Can you please provide a full example on how to get from the image as a CameraImage object to a opencv Mat object and convert it to BGR format afterwards? This is the code I am using right now and it seems that the number of channels for converting from YUV to BGR is wrong(the image format passed to the camera controller is yuv420).

`

Result processImage(CameraImage image) {

    final Uint8List yPlane = image.planes[0].bytes;
    final Uint8List uPlane = image.planes[1].bytes;
    final Uint8List vPlane = image.planes[2].bytes;

    int height = image.height;
    int width = image.width;

    final Uint8List combinedPlanes =
        Uint8List(yPlane.length + uPlane.length + vPlane.length);
    combinedPlanes.setRange(0, yPlane.length, yPlane);
    combinedPlanes.setRange(
        yPlane.length, yPlane.length + uPlane.length, uPlane);
    combinedPlanes.setRange(
        yPlane.length + uPlane.length, combinedPlanes.length, vPlane);

    Mat matImage = Mat.create(rows: height, cols: width, type: MatType.CV_8UC3);
    matImage.data.setAll(0, combinedPlanes);

    Mat bgrBytes = cvtColor(matImage, COLOR_YUV2BGR_NV21);

    Mat blurredImage = gaussianBlur(bgrBytes, (size, size), sigma);

    return Result(processesImage: image, data: {});
}

`

[rainyl]

@TheoNanu Sorry for the late reply, I am kind of busy recently.

I havn't used the camera plugin so not familiar with it, but Mat.data actually stores pixel values, ensure image.planes[0].bytes; are raw pixel values too.

I will try to provide a camera interaction example in awesome-opencv_dart when having some time, you can also open a PR to https://github.com/rainyl/awesome-opencv_dart if you solved it.

Also, although not recommended, you can use the videoio module from opencv too, which alows opening a VideoCapture from a camera, but in this way, you need to use Texture if showing the frames is needed.

[TheoNanu]

Ok, I have found a solution for converting the CameraImage image object in YUV420 format to Mat:

 int width = image.width;
 int height = image.height;

 // Plane data
 final yPlane = image.planes[0];
 final uPlane = image.planes[1];
 final vPlane = image.planes[2];

 // Calculate stride and copy Y data
 Uint8List yBuffer = Uint8List(width * height);
 for (int row = 0; row < height; row++) {
   yBuffer.setRange(
     row * width,
     (row + 1) * width,
     yPlane.bytes
         .sublist(row * yPlane.bytesPerRow, row * yPlane.bytesPerRow + width),
   );
 }

 // Calculate stride and copy UV data
 int uvWidth = width ~/ 2;
 int uvHeight = height ~/ 2;

 Uint8List uBuffer = Uint8List(uvWidth * uvHeight);
 Uint8List vBuffer = Uint8List(uvWidth * uvHeight);

// Struture of bytes in plane: uvHeight * (width + padding)
 // bytesPerRow = width + padding
 // bytesPerPixel = distance between two consecutive chroma values
 // For YUV420_888 the first byte represents the chroma value and the second byte represents the padding (only used for memory consistency)

 // Bytes associated with last row do not have the row padding and also the last chroma value does not have padding

 for (int row = 0; row < uvHeight; row++) {
   for (int col = 0; col < uvWidth; col++) {
     int sourceIndex =
         row * uPlane.bytesPerRow + col * (uPlane.bytesPerPixel ?? 1);
     uBuffer[row * uvWidth + col] =
         uPlane.bytes[sourceIndex]; // Use the first byte
     vBuffer[row * uvWidth + col] =
         vPlane.bytes[sourceIndex]; // Use the first byte
   }
 }

Uint8List yuvBuffer =
     Uint8List(yBuffer.length + uBuffer.length + vBuffer.length);
 yuvBuffer.setRange(0, yBuffer.length, yBuffer);
 yuvBuffer.setRange(yBuffer.length, yBuffer.length + uBuffer.length, uBuffer);
 yuvBuffer.setRange(
     yBuffer.length + uBuffer.length, yuvBuffer.length, vBuffer);

Mat matImage = Mat.create(
     rows: height + height ~/ 2, cols: width, type: MatType.CV_8UC1);
 matImage.data.setAll(0, yuvBuffer);

 Mat bgrImage = cvtColor(matImage, COLOR_YUV2BGR_IYUV);

For some reason the image is rotated to a degree which depends on the current orientation of the device, if you need to correct that you can use rotate function from the library with the degree of rotation parameter appropriate to the current orientation

[rainyl]

Thanks for your solution~
Actually I have managed to make it work, refer to
rainyl/awesome-opencv_dart#9
But some bugs still exists, so it is not merged.

[eramirez3]

Is there any way to make YUV420 to RGB more efficient ?

[angelru]

static imglib.Image _convertToImage(CameraImage cameraImage) {
    try {
      if (Platform.isAndroid) {
        final int width = cameraImage.width;
        final int height = cameraImage.height;
        final int uvRowStride = cameraImage.planes[1].bytesPerRow;
        final int uvPixelStride = cameraImage.planes[1].bytesPerPixel ?? 1;

        var img = imglib.Image(width: width, height: height);

        for (int y = 0; y < height; y++) {
          final int yRow = y * cameraImage.planes[0].bytesPerRow;

          final int uvRow = (y ~/ 2) * uvRowStride;

          for (int x = 0; x < width; x++) {
            final int yIndex = yRow + x;

            final int uvIndex = uvRow + (x ~/ 2) * uvPixelStride;

            final int yp = cameraImage.planes[0].bytes[yIndex];
            final int up = cameraImage.planes[1].bytes[uvIndex];
            final int vp = cameraImage.planes[2].bytes[uvIndex];

            int r = (yp + 1.402 * (vp - 128)).round().clamp(0, 255);
            int g = (yp - 0.344136 * (up - 128) - 0.714136 * (vp - 128))
                .round()
                .clamp(0, 255);
            int b = (yp + 1.772 * (up - 128)).round().clamp(0, 255);

            img.setPixelRgba(x, y, r, g, b, 255);
          }
        }

        return imglib.copyRotate(img, angle: 90);
      } else {
        return imglib.Image.fromBytes(
          width: cameraImage.width,
          height: cameraImage.height,
          bytes: cameraImage.planes[0].bytes.buffer,
          order: imglib.ChannelOrder.bgra,
          format: imglib.Format.uint8,
        );
      }
    } catch (e) {
      throw Exception('Error en conversión de imagen: $e');
    }
  }

Camera:

      _cameraController = CameraController(
        camera,
        ResolutionPreset.medium,
        enableAudio: false,
        imageFormatGroup:
            Platform.isAndroid
                ? ImageFormatGroup.yuv420
                : ImageFormatGroup.bgra8888,
      );

Usage:

var imgC = _convertToImage(cameraImage);
img = await cv.imdecodeAsync(imglib.encodePng(imgC), cv.IMREAD_GRAYSCALE);

This always works for me, but I'm not sure it's the "most efficient" way.

Any ideas?

[angelru]

The conversion in the example is faster than the one I was using, so I've switched to the one you posted.