Video temporal upsampling

[HReynaud]

Hi,

Is there a way with the current library to do temporal super-resolution ? From what I can see, only the spatial super resolution is currently possible. I would like to have one UNet that generates videos with dimensions 64x64x16 and a super-resolution UNet that would upsample them to 128x128x32.
Please let me know if I am missing something !

[lucidrains]

@HReynaud not yet, i'm planning to build that by month's end, or by early next month

[HReynaud]

Excellent news ! Thank you for your hard work !

[lucidrains]

@HReynaud yea no problem, it will take some thought rather than just blindly throwing in a 3d conv with strides

the reason is because i want to maintain the ability to pretrain on images before video, so it'll have to follow the existing scheme of being agnostic to image / video inputs

[lucidrains]

@HReynaud should be able to get the temporal upsampling / interpolation finished tomorrow morning!

[lucidrains]

@HReynaud pray tell, what dataset are you training on?

[HReynaud]

Thank you so much ! I was considering giving it a try, but was not sure where to start. I am training on the Echonet-Dynamic dataset to do cardiac ultrasound generation: https://echonet.github.io/dynamic/ .
The goal is to explore video generation with precise control over the embeddings. I am using auto-encoders and other techniques to encode specific information into a latent space that I use as the "text embeddings" for the conditional generation. I will give it a try as soon as it's live !

[lucidrains]

@HReynaud oh that's so cool! ok i'll def make sure to build this right 😃 this-mitral-valve-prolapse-does-not-exist.com lmao

[lucidrains]

@HReynaud was able to get it working, although inpainting may still not be functional (will look at that tomorrow morning) 44da9be

[lucidrains]

@HReynaud to use it, when instantiating Imagen, just pass in temporal_downsample_factor

it will be a tuple of integers, specifying at each stage of the series of unets, how much to divide the frame size by

[HReynaud]

Awesome ! I will let you know how it goes !

[lucidrains]

@HReynaud how did it go?

[HReynaud]

The code seems to work flawlessly and temporal_downsample_factor is very straight forward to set up, thank you for that !

(Top is sampled, bottom is ground truth)

This is the output of the upsampling model, I am going from 64x64x16 to 112x112x64. The top row is sampled from the SR model, conditioned on the low resolution ground truth videos. I have to investigate if tuning some parameters in the ElucidatedImagen can make the speckle noise more consistent.

[lucidrains]

@HReynaud ohh beautiful, it captured the valves and septal motion so well! looking forward to reading your paper on generated cardiac pathologies 😄 ❤️

[lucidrains]

@HReynaud make sure you show that to some cardiologists, go blow their minds!