Awesome

pytorch-unet-segmentation

Members : <a href="https://github.com/PyeongKim">PyeongEun Kim</a>, <a href="https://github.com/juhlee">JuHyung Lee</a>, <a href="https://github.com/mijeongl"> MiJeong Lee </a>

Supervisors : <a href="https://github.com/utkuozbulak">Utku Ozbulak</a>, Wesley De Neve

Description

This project aims to implement biomedical image segmentation with the use of U-Net model. The below image briefly explains the output we want:

The dataset we used is Transmission Electron Microscopy (ssTEM) data set of the Drosophila first instar larva ventral nerve cord (VNC), which is dowloaded from ISBI Challenge: Segmentation of of neural structures in EM stacks

The dataset contains 30 images (.png) of size 512x512 for each train, train-labels and test.

Table of Content

• Dataset

• Preprocessing

• Model

• Loss function

• Post-processing

• Results

• Dependency

• References

Dataset <a name="dataset"></a>

class SEMDataTrain(Dataset):

    def __init__(self, image_path, mask_path, in_size=572, out_size=388):
        """
        Args:
            image_path (str): the path where the image is located
            mask_path (str): the path where the mask is located
            option (str): decide which dataset to import
        """
        # All file names
	# Lists of image path and list of labels
        # Calculate len
        # Calculate mean and stdev

    def __getitem__(self, index):
        """Get specific data corresponding to the index
        Args:
            index (int): index of the data

        Returns:
            Tensor: specific data on index which is converted to Tensor
        """
        """
        # GET IMAGE
        """
        #Augmentation on image
          # Flip 
          # Gaussian_noise
          # Uniform_noise
          # Brightness
          # Elastic distort {0: distort, 1:no distort}
          # Crop the image
          # Pad the image
          # Sanity Check for Cropped image
          # Normalize the image

          # Add additional dimension
          # Convert numpy array to tensor
        
        #Augmentation on mask
          # Flip same way with image
          # Elastic distort same way with image
          # Crop the same part that was cropped on image
          # Sanity Check
          # Normalize the mask to 0 and 1
      
        # Add additional dimension
        # Convert numpy array to tensor

        return (img_as_tensor, msk_as_tensor)

    def __len__(self):
        """
        Returns:
            length (int): length of the data
        """

Preprocessing <a name="preprocessing"></a>

We preprocessed the images for data augmentation. Following preprocessing are :

• Flip
• Gaussian noise
• Uniform noise
• Brightness
• Elastic deformation
• Crop
• Pad

Image Augmentation

Crop and Pad

Padding process is compulsory after the cropping process as the image has to fit the input size of the U-Net model.

In terms of the padding method, symmetric padding was done in which the pad is the reflection of the vector mirrored along the edge of the array. We selected the symmetric padding over several other padding options because it reduces the loss the most.

To help with observation, a 'yellow border' is added around the original image: outside the border indicates symmetric padding whereas inside indicates the original image.

Model <a name="model"></a>

Architecture

We have same structure as U-Net Model architecture but we made a small modification to make the model smaller.

Loss function <a name="lossfunction"></a>

We used a loss function where pixel-wise softmax is combined with cross entropy.

Softmax

Cross entropy

Post-processing <a name="postprocessing"></a>

In attempt of reducing the loss, we did a post-processing on the prediction results. We applied the concept of watershed segmentation in order to point out the certain foreground regions and remove regions in the prediction image which seem to be noises.

The numbered images in the figure above indicates the stpes we took in the post-processing. To name those steps in slightly more detail:

* 1. Convertion into grayscale
* 2. Conversion into binary image
* 3. Morphological transformation: Closing
* 4. Determination of the certain background
* 5. Calculation of the distance
* 6. Determination of the certain foreground
* 7. Determination of the unknown region
* 8. Application of watershed
* 9. Determination of the final result

Conversion into grayscale

The first step is there just in case the input image has more than 1 color channel (e.g. RGB image has 3 channels)

Conversion into binary image

Convert the gray-scale image into binary image by processing the image with a threshold value: pixels equal to or lower than 127 will be pushed down to 0 and greater will be pushed up to 255. Such process is compulsory as later transformation processes takes in binary images.

Morphological transformation: Closing.

We used morphologyEX() function in cv2 module which removes black noises (background) within white regions (foreground).

Determination of the certain background

We used dilate() function in cv2 module which emphasizes/increases the white region (foreground). By doing so, we connect detached white regions together - for example, connecting detached cell membranes together - to make ensure the background region.

Caculation of the distance

This step labels the foreground with a color code: red color indicates farthest from the background while blue color indicates closest to the background.

Determination of the foreground

Now that we have an idea of how far the foreground is from the background, we apply a threshold value to decide which part could surely be the foreground.

The threshold value is the maximum distance (calculated from the previous step) multiplied by a hyper-parameter that we have to manually tune. The greater the hyper-parameter value, the greater the threshold value, and therefore we will get less area of certain foreground.

Determination of the unknown region

From previous steps, we determined sure foreground and background regions. The rest will be classified as 'unknown' regions.

Label the foreground: markers

We applied connectedComponents() function from the cv2 module on the foreground to label the foreground regions with color to distinguish different foreground objects. We named it as a 'marker'.

Application of watershed and Determination of the final result

After applying watershed() function from cv2 module on the marker, we obtained an array of -1, 1, and many others.

* -1 = Border region that distinguishes foreground and background
*  1 = Background region

To see the result, we created a clean white page of the same size with the input image. then we copied all the values from the watershed result to the white page except 1, which means that we excluded the background.

Results <a name="results"></a>

We chose the best learning rate that fits the optimizer based on how fast the model converges to the lowest error. In other word, the learning rate should make model to reach optimal solution in shortest epoch repeated. However, the intersting fact was that the epochs of lowest loss and highest accuracy were not corresponding. This might be due to the nature of loss function (Loss function is log scale, thus an extreme deviation might occur). For example, if the softmax probability of one pixel is 0.001, then the -log(0.001) would be 1000 which is a huge value that contributes to loss. For consistency, we chose to focus on accuracy as our criterion of correctness of model.

Model Downloads

Model trained with SGD can be downloaded via dropbox: https://www.dropbox.com/s/ge9654nhgv1namr/model_epoch_2290.pwf?dl=0

Model trained with RMS prop can be downloaded via dropbox: https://www.dropbox.com/s/cdwltzhbs3tiiwb/model_epoch_440.pwf?dl=0

Model trained with Adam can be downloaded via dropbox: https://www.dropbox.com/s/tpch6u41jrdgswk/model_epoch_100.pwf?dl=0

Example

Dependency <a name="dependency"></a>

Following modules are used in the project:

* python >= 3.6
* numpy >= 1.14.5
* torch >= 0.4.0
* PIL >= 5.2.0
* scipy >= 1.1.0
* matplotlib >= 2.2.2

References <a name="references"></a> :

[1] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, http://arxiv.org/pdf/1505.04597.pdf

[2] P.Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, http://cognitivemedium.com/assets/rmnist/Simard.pdf