Home

Awesome

DOI

Deep Neural Networks for Active Wave Breaking Classification

This repository contains code and data to reproduce the results of the paper Deep Neural Networks for Active Wave Breaking Classification published by Scientific Reports.

Contents

Deep Neural Networks for Active Wave Breaking Classification

1. Dependencies

Using conda:

conda env create -f environment_win.yml
conda env create -f environment_linux.yml

Manually

<details> <summary> Package list </summary>
# install GIT
conda install git

# create a new environment
conda create --name tf python=3

# activate your new environment
conda activate tf

# If you have a nvidia GPU installed and properly configured
pip install --upgrade pip
pip install tensorflow

# extras
pip install -q git+https://github.com/tensorflow/examples.git
pip install tensorflow_addons

# conda packages
conda install -y numba natsort pandas scikit-learn scikit-image natsort  matplotlib seaborn netCDF4 xarray ipython tqdm
pip install h5netcdf

# Extra thresholding methods
pip install pythreshold

# fitting circles to data
pip install miniball

# parallel computations
pip install pebble
</details>

2. Data

2.1. Manual Creation

2.2. Production Ready

ModelLinkAlternative link
Train (10k)-
Train (20k)-
Test (1k)-
Black Sea (200k)Upcoming-
La Jument 2019 (10k)Upcoming-

3. Training

Open In Colab | Jupyter Notebook

Note: The training dataset used here is a smaller version (10k) of the published dataset so it can run on Google Colab. The 20K dataset takes over 6 hours to train and Google will disconnect your session.

The data needs to be in a folder which has sub-folders "0" and "1"

For example:

train
    ├───0
    ├───1

There are 5 backbones implemented: VGG16, ResNet50V2, InceptionResNetV2, MobileNetV2 and EfficientNet

Note that the weights from these pre-trained models will be reset and updated from the scratch here. These models have no knowledge of the present data and, consequently, transferred learning does not work well.

Example

python train.py --data "train/" --backbone "VGG16" --model "vgg_test" --logdir "logs/" --random-state 11 --validation-size 0.2 --learning-rate 0.00001 --epochs 200 --batch-size 64 --dropout 0.5 --input-size 256 256
<details> <summary> Arguments: </summary> </details> <br/>

The neural network looks something like this:

Pre-trained Models

Please use the links below to download pre-trained models:

Scientific Reports (20K dataset)

ModelLinkAlternative link
VGG16-
ResNet50V2-
InceptionResNetV2-
MobileNet-
EfficientNet-
<!-- **Black Sea (200k dataset)** | Model | Link | Alternative link | | --------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------- | | **VGG16** | [![](badges/google_drive_badge.svg)](https://drive.google.com/file/d/1Oy_4q4SJpg0TVE8LWXR7L57S0zD8MP7T/view?usp=sharing) | - | | **ResNet50V2** | [![](badges/google_drive_badge.svg)](https://drive.google.com/file/d/1Exs1_bpE9Rqk9aPDlzfH4HDgF2wygR4c/view?usp=sharing) | - | | **InceptionResNetV2** | [![](badges/google_drive_badge.svg)](https://drive.google.com/file/d/1tKIEWKr6G8RtwsTk1vqg9UJB8cx2M8J5/view?usp=sharing) | - | | **MobileNet** | [![](badges/google_drive_badge.svg)](https://drive.google.com/file/d/15Roa1tjtoXbn2w-H0OAcfWVNXMKgr_fV/view?usp=sharing) | - | | **EfficientNet** | [![](badges/google_drive_badge.svg)](https://drive.google.com/file/d/1_JEAJYfD3-QUSEReJOI7aI2QBr4LlafP/view?usp=sharing) | - | **Note**: These models were trained from the scratch with data processed by Pedro Guimarães. Overall, they have accuracy scores higher than 95% but the training data is less general than the 20K dataset seen in the paper. -->

La Jument (10K dataset)

ModelLinkAlternative link
VGG16Upcoming-
ResNet50V2Upcoming-
InceptionResNetV2Upcoming-
MobileNetUpcoming-
EfficientNetUpcoming-

Note: Work in progress.

4. Model Performance

4.1. Evaluating

Open In Colab | Jupyter Notebook

To evaluate a pre-trained model on test data, use the test script.

Example:

python test.py --data "path/to/test/data/" --model "VGG16.h5" --threshold 0.5 -- output "path/to/results.csv"
<details> <summary> Arguments: </summary>

The classification report with be printed on the screen. For example:

              precision    recall  f1-score   support

         0.0       0.88      0.99      0.94      1025
         1.0       0.87      0.23      0.36       175

    accuracy                           0.88      1200
   macro avg       0.88      0.61      0.65      1200
weighted avg       0.88      0.88      0.85      1200

To summarize the model metrics do:

python metrics.py --data "path/to/data/" --model "VGG16.h5" --threshold 0.5 -- output "path/to/metrics.csv"

The arguments are the same as above.

The results look something like this:

VGG16Binary_AccuracyTrue_PositivesFalse_PositivesTrue_NegativesFalse_NegativesPrecisionRecallAUC
Train0.89771.00248.005680.00521.000.760.600.92
Validation0.87100.0019.001463.00222.000.840.310.90
Test0.8840.006.001019.00135.000.870.230.82

To plot the training curves and a confusion matrix, do:

python plot_history_and_confusion_matrix.py --history "path/to/history.csv" --results "path/to/results.csv" --output "figure.png"
<details> <summary> Arguments: </summary>

The results look like this:

4.2. Results

The table below summarizes the results presented in the paper. Results are sorted by AUC.

Train

ModelAccuracyTPFPTNFNPrecisionRecallAUC
ResNetV2500.971414198139782800.8770.8350.989
VGG160.93855273139118310.7580.5070.943
InceptionResnetV20.927886359138238020.7120.5250.932
EfficientNet0.77214033346109202970.2950.8250.874
MobileNet0.9044362681391612500.6190.2590.848

Validation

ModelAccuracyTPFPTNFNPrecisionRecallAUC
VGG160.9322216534782040.7730.520.946
ResNetV2500.9191979734502240.670.4680.873
InceptionResnetV20.9211908134662310.7010.4510.93
EfficientNet0.8093536872856720.3390.8310.897
MobileNet0.9081236434793020.6580.2890.878

Test

ModelAccuracyTPFPTNFNPrecisionRecallAUC
VGG160.87610680945690.570.6060.855
ResNetV2500.8819563962800.6010.5430.843
InceptionResnetV20.8829157968840.6150.520.839
EfficientNet0.8738865960870.5750.5030.827
MobileNet0.87530510201450.8570.1710.768

5. Using a Pre-trained Neural Network

5.1 Predicting on New Data

Create a dataset either manually or with the provided tools then use the predict script. The data structure is as follows:

pred
    ├───images
        ├───img_00001.png
        ├───img_00002.png
        ├───...
        ├───img_0000X.png

Example:

python predict.py --data "pred/" --model "VGG16.h5" --threshold 0.5 --output "results.csv"
<details> <summary> Arguments: </summary>

5.2. Predicting from the Results of the Naïve Detector

Use predict from naïve candidates and the results from the naive wave breaking detector and a pre-trained neural network to obtain only active wave breaking instances. This script runs on CPU but can be much faster on GPU.

Example:

python predict_from_naive_candidates.py --debug --input "naive_results.csv" --model "path/to/model.h5" --frames "path/to/frames/folder/"  --region-of-interest "region_of_interest.csv" --output "robust_results.csv" --temporary-path "tmp" --frames-to-plot 1000 --threshold 0.5
<details> <summary> Arguments: </summary>

The output of this script is a comma-separated value (csv) file. It looks like exactly like the output of naive wave breaking detector but adding a extra column with the results of the classification.

5.3. Plotting Wave Breaking Detection Results

Plot the results of the wave breaking detection algorithms. Can handle outputs of any algorithm, as long as the input data is correct. Ideally the results from cluster.py are used as input.

Example:

python plot_wave_breaking_detection_results.py --input "clustered_events.csv" --output "path/to/output/" --frames "path/to/frames/" --region-of-interest "path/to/roi.csv" --frames-to-plot 1000
<details> <summary> Arguments: </summary>

Note: The input data must have at least the following entries: ic, jc, ir, frame, wave_breaking_event.

</details>

6. Model Interpretation

Open In Colab | Jupyter Notebook

Use interpret.py to apply Grad-CAM to data samples. Organize your data as follows:

gradcam
    ├───images
        ├───img_00001.png
        ├───img_00002.png
        ├───...
        ├───img_0000X.png

Example:

python interpret.py --data "path/to/gradcam" --model "VGG16.h5" -o "path/to/output"
<details> <summary> Arguments: </summary>

Note: This script will only work with VVG16 models.

</details> <br/>

7. Active Wave Breaking Segmentation

The neural networks developed here can also be used for image segmentation. Please refer to Image Segmentation.

8. Wave Tracking

Please refer to Wave Tracking.

9. Wave Breaking Statistics

Please refer to Wave Breaking Statistics.

Gallery

La Jument:

Black Sea:

Aqua Alta:

<!-- **La Jument (2019):** **Note:** This is a production example. The model has not seen this data before. ![](docs/production.gif) -->

Grad-CAM (Black Sea):

Image Segmentation (La Jument):

Standard Variable Names

The following variables are standard across this repository and scripts that output these quantities should use these names. If a given script has extra output variables, these are documented in each script.

<details> <summary> Variables </summary>
VariableDescription
xx-coordinate in metric coordinates.
yy-coordinate in metric coordinates.
zz-coordinate in metric coordinates.
timedate and time. Use a format that pandas.to_datetime() can understand.
framesequential number.
ipixel coordinate in pixel units. Use Matplotlib coordinate system.
jpixel coordinate in pixel units. Use Matplotlib coordinate system.
iccenter of a circle or ellipse in pixel coordinates.
jccenter of a circle or ellipse in pixel coordinates.
xccenter of a circle or ellipse in metric coordinates.
yccenter of a circle or ellipse in metric coordinates.
irradius in the i-direction.
jrradius in the j-direction.
xrradius in the x-direction.
yrradius in the y-direction.
theta_ijangle of rotation of an ellipse with respect to the x-axis counter-clockwise.
theta_xyangle of rotation of an ellipse with respect to the x-axis counter-clockwise.
wave_breaking_eventunique wave breaking event id.
vxvelocity in the x-direction in m/s.
vyvelocity in the y-direction in m/s.
vivelocity in the x-direction in pixels/frame.
vjvelocity in the y-direction in pixels/frame.
</details> <br/>

Disclaimer

There is no warranty for the program, to the extent permitted by applicable law except when otherwise stated in writing the copyright holders and/or other parties provide the program “as is” without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. the entire risk as to the quality and performance of the program is with you. should the program prove defective, you assume the cost of all necessary servicing, repair or correction.