1/3 of all the images in the data set are blank

[rsokl]

I was looking through this dataset and noticed that many of the images are completely blank (all pixels are 0).
Using your mybatch_generator_prediction I loaded all the images and was surprised to find that roughly 1/3 of all the images – train and test alike – consist only of 0s

this is almost certainly inflating the kaggle statistics and is likely diluting the training process. Do you have any insight into what might be going on here?

For context: I downloaded the data from kaggle, in case that is relevant.

[SorourMo]

Hi @rsokl,
That is correct (has been mentioned here) and it is due to the fact that complete Landsat 8 images have kind of a parallelogram shape (one example is shown here). This leads to having four empty (pixel values = zero) triangles around the informative (nonzero) area of the images. Since small 384*384 patches are extracted from complete Landsat 8 images, some of those patches are completely empty.
The names of informative patches (with more than 80% nonzero pixels) in 38-Cloud and 95-Cloud datasets are stored in the attached csv files. Feel free to use only these informative patches in your trainings.
38-cloud_95-cloud_training_informative_patches.zip

Please note that for reconstructing a complete Landsat 8 cloud mask at the output for a test image, the empty patches are required.
Thank you for your point. I have updated 38-Cloud dataset to include the csv file of informative patches.

[rsokl]

Great! Thank you for clarifying and for the useful zip file.

Congrats on your CloudNet+ model; it's a nice piece of work! :)

[Lewisbarbara]

Hi, I tried to use the dataset (file name 38-Cloud_cloud_detection_dataset.zip), but apparently this .zip file is corrupted. I have downloaded it several times, but I always get the same error when extracting the data.

[sorourm-bmg]

Hi @Lewisbarbara
Which URL did you use? I downloaded 38-cloud dataset from kaggle with no problems (https://www.kaggle.com/datasets/sorour/38cloud-cloud-segmentation-in-satellite-images?resource=download then press Download (13GB) button on the top). Make sure the downloaded file zip size is ~12.7 GB. Sometimes download stops because of the Internet problems without any errors which leads to a corrupted file.

[Lewisbarbara]

Hello, @SorourMo
Thank you very much for your response. I was able to download the link you pointed out (https://www.kaggle.com/datasets/sorour/38cloud-cloud-segmentation-in-satellite-images?resource=download), but I encountered an error when trying to extract the .zip file. So, I downloaded the new dataset from https://www.kaggle.com/datasets/sorour/95cloud-cloud-segmentation-on-satellite-images. All downloads from this new link were successful, and I was able to extract the zip files. Now, I am only unsure about how to combine the data as explained here: https://github.com/SorourMo/95-Cloud-An-Extension-to-38-Cloud-Dataset?tab=readme-ov-file
“Because of space limitation, the dataset has been split into two parts:
57 scenes of 95-Cloud training + (the rest of the training scenes (38 scenes) and 20 test scenes).
Click here for downloading 57 scenes of "95-Cloud training set".
Click here for downloading the "rest of the dataset".
After downloading the data, the user should combine them together.