Home

Awesome

CoastSat

Last Commit GitHub issues GitHub release License: GPL v3 GitHub commit activity DOI Join the chat at https://gitter.im/CoastSat/community

CoastSat is an open-source software toolkit written in Python that enables users to obtain time-series of shoreline position at any coastline worldwide from 40 years (and growing) of publicly available satellite imagery (Landsat and Sentinel-2).

Alt text

Finding CoastSat useful? Show your support with a Github star — it’s a simple click that helps others discover it ⭐️

:point_right: Visit the CoastSat website to explore and download existing satellite-derived shoreline datasets generated with CoastSat in the Pacific and Atlantic basins.

<details> <summary><strong>Latest updates</strong></summary>

:arrow_forward: (2024/10/02) CoastSat v3.0: integration with FES2022 global tide model to perform tidal correction and beach slope estimation within CoastSat.

:arrow_forward: (2024/08/29) CoastSat v2.7: reverse compatibility for file downloads (pre v2.6) and removed Collection 1 (deprecated, throws an error)

:arrow_forward: (2024/05/07) CoastSat v2.6: added the tilename at the end of each image filename when downloading so that images can be separated by tiles if needed. Also a number of bug fixes on matplotlib and numpy from @2320sharon and @thekester.

:arrow_forward: (2024/04/26) CoastSat v2.5: contributions from @2320sharon and @DanieTheron to improve the download updates and cloud masking for Landsat.

</details> <details> <summary><strong>Additional toolkits and documentation</strong></summary>

:point_right: Other repositories and extensions related to CoastSat:

:point_right: Publications describing the CoastSat satellite-derived shorelines:

</details> <details> <summary><strong>Project description</strong></summary>

Satellite remote sensing can provide low-cost long-term shoreline data capable of resolving the temporal scales of interest to coastal scientists and engineers at sites where no in-situ field measurements are available. CoastSat enables the non-expert user to extract shorelines from Landsat 5, Landsat 7, Landsat 8, Landsat 9 and Sentinel-2 images. The shoreline detection algorithm implemented in CoastSat is optimised for sandy beach coastlines. It combines a sub-pixel border segmentation and an image classification component, which refines the segmentation into four distinct categories such that the shoreline detection is specific to the sand/water interface.

The toolbox has the following functionalities:

  1. easy retrieval of satellite imagery spanning the user-defined region of interest and time period from Google Earth Engine, including state-of-the-art pre-processing steps (re-projecting the different bands, pansharpening, advanced cloud masking).
  2. automated extraction of shorelines from all the selected images using a sub-pixel resolution technique and options for quality-control.
  3. intersection of the 2D shorelines with user-defined shore-normal transects.
  4. tidal correction using tide/water levels and an estimate of the beach slope.
  5. post-processing of the shoreline time-series, despiking and seasonal averaging.
  6. Beach slope estimation using satellite-derived shorelines and predicted tides
</details>

Table of Contents

1. Installation<a name="introduction"></a>

<details> <summary><strong>1.1 Create an environment with Mamba/Anaconda:</strong></summary>

To run the toolbox you first need to install the required Python packages in an environment.You can do this with Anaconda or with Mamba. It is highly recommended to use Mamba. Go to https://github.com/conda-forge/miniforge and download Miniforge for your operating system.

Once you have it installed on your PC, open the Miniforge Prompt (in Mac and Linux, open a terminal window) and run the following commands to install the coastsat environment:

mamba create -n coastsat
mamba activate coastsat
mamba install python=3.11 geopandas gdal -y
mamba install earthengine-api scikit-image matplotlib astropy notebook -y
pip install pyqt5 imageio-ffmpeg
mamba install pyfes -y

All the required packages have now been installed and are self-contained in an environment called coastsat. Always make sure that the environment is activated with:

mamba activate coastsat

To confirm that you have successfully activated CoastSat, your terminal command line prompt should now start with (coastsat).

:warning: In case errors are raised :warning:: clean things up with the following command before attempting to install coastsat again:

mamba clean --all
mamba update conda

If you are more used to Anaconda you can run the same commands in the Anaconda Prompt by replacing mamba with conda.

If any problems with the installation, raise an issue.

</details> <details> <summary><strong>1.2 Activate Google Earth Engine Python API:</strong></summary>

First, create a Google Earth Engine project at https://signup.earthengine.google.com/. Then, go to https://cloud.google.com/sdk/docs/install and install the gcloud CLI. After you have installed it will automatically launch and let you authenticate with your GEE account (or personal gmail).

:warning: if you're finding that you're always asked to authenticate, open the gloud CLI and run this command: gcloud auth application-default login to set a default authentication on your machine.

</details>

:white_check_mark: If you completed those two steps you are ready to start using CoastSat!

2. Usage<a name="usage"></a>

An example of how to run the software in a Jupyter Notebook is provided in example_jupyter.ipynb.

If you prefer to use Spyder or other integrated development environments (IDEs), a Python script named example.py is also included in the repository. If using Spyder, make sure that the Graphics Backend is set to Automatic and not Inline (as this mode doesn't allow to interact with the figures). To change this setting go under Preferences>IPython console>Graphics.

<details> <summary><strong>How to run Jupyter Notebooks</strong></summary> To run a Jupyter Notebook, open the Miniforge prompt, activate your `coastsat` environment with `mamba activate coastsat`, and then run type `jupyter lab`.

A web browser will open with the JupyterLab interface. Browse to the directory where you downloaded CoastSat and click on the file example_jupyter.ipynb. The notebook will open. A Jupyter Notebook combines formatted text and code. To run the code, place your cursor inside one of the code sections and click on the run cell button (or press Shift + Enter) and progress forward.

image

</details> <br> The following sections show an example of how to run the full CoastSat workflow at Narrabeen-Collaroy beach (Australia).

2.1 Retrieval of the satellite images<a name="retrieval"></a>

To retrieve from the GEE server the available satellite images cropped around the user-defined region of coastline for the particular time period of interest, the following variables are required:

The call metadata = SDS_download.retrieve_images(inputs) will launch the retrieval of the images and store them as .TIF files (under /filepath/sitename). The metadata contains the exact time of acquisition (in UTC time) of each image, its projection and its geometric accuracy. If the images have already been downloaded previously and the user only wants to run the shoreline detection, the metadata can be loaded directly by running metadata = SDS_download.get_metadata(inputs).

The inputs below will download all the images of Narrabeen acquired by since the start of 2024 by Landsat and Sentinel-2.

<details> <summary><strong>Inputs download example</strong></summary> 
# region of interest (longitude, latitude)
polygon = [[[151.2957545, -33.7012561],
            [151.297557, -33.7388075],
            [151.312234, -33.7390216],
            [151.311204, -33.701399],
            [151.2957545, -33.7012561]]]
# date range
dates = ['2024-01-01', '2025-01-01']
# satellite missions ['L5','L7','L8','L9','S2']
sat_list = ['L8','L9','S2']
# name of the site
sitename = 'NARRA'
# directory where the data will be stored
filepath = os.path.join(os.getcwd(), 'data')
# put all the inputs into a dictionnary
inputs = {'polygon': polygon, 'dates': dates, 'sat_list': sat_list,
          'sitename': sitename, 'filepath':filepath}
# download images
metadata = SDS_download.retrieve_images(inputs)
</details>

2.2 Shoreline detection<a name="detection"></a>

Once the images have been downloaded, the shorelines can be mapped. The following user-defined settings are needed:

There are additional parameters (min_beach_size, min_length_sl, cloud_mask_issue, sand_color, pan_off, s2cloudless_prob) that can be fine-tuned to optimise the detection (for Advanced users). For the moment leave these parameters on their default values, we will see later how they can be edited and what they do.

An example of settings for Narrabeen beach is provided below.

<details> <summary><strong>Settings example</strong></summary> 
settings = {
    # general parameters:
    'cloud_thresh': 0.5,        # threshold on maximum cloud cover
    'dist_clouds': 300,         # ditance around clouds where shoreline is not mapped
    'output_epsg': 28356,       # epsg code of spatial reference system for the output
    # quality control:
    'check_detection': True,    # if True, shows each shoreline detection to the user for validation
    'adjust_detection': False,  # if True, allows user to adjust the postion of each shoreline by changing the threhold
    'save_figure': True,        # if True, saves a figure showing the mapped shoreline for each image
    # [ONLY FOR ADVANCED USERS] advanced detection parameters:
    'min_beach_area': 1000,     # minimum area (in metres^2) for an object to be labelled as a beach
    'min_length_sl': 500,       # minimum length (in metres) of shoreline perimeter to be valid
    'cloud_mask_issue': False,  # switch this parameter to True if sand pixels are masked as clouds
    'sand_color': 'default',    # 'default', 'latest', 'dark' (for grey/black sand beaches) or 'bright' (for white beaches)
    'pan_off': False,           # True to switch pansharpening off for Landsat 7/8/9 imagery
    's2cloudless_prob': 60,     # probability to mask cloudy pixels in s2cloudless

    'inputs': inputs, # add the inputs defined previously
}
</details>

Before mapping the shorelines, it is HIGHLY RECOMMENDED to digitize a reference shoreline in order to improve the detection. This can be done by calling settings['reference_shoreline'] = SDS_preprocess.get_reference_sl_manual(metadata, settings), which allows the user to manually digitize the reference shoreline on a cloud-free image. Then you can set the maximum distance from the reference shoreline where shoreline points can be detection using settings['max_dist_ref']. This reference shoreline helps to reject outliers and false detections when mapping shorelines. See below how to accurately add this reference shoreline.

<details> <summary><strong>Digitize reference shoreline</strong></summary>

The user can manually digitize one or several reference shorelines on one of the images by calling:

settings['reference_shoreline'] = SDS_preprocess.get_reference_sl_manual(metadata, settings)
settings['max_dist_ref'] = 100 # max distance (in meters) allowed from the reference shoreline

This function allows the user to click points along the shoreline on cloud-free satellite images, as shown in the animation below.

ref_shoreline

The maximum distance (in metres) allowed from the reference shoreline is defined by the parameter max_dist_ref. This parameter is set to a default value of 100 m. If you think that 100 m buffer from the reference shoreline will not capture the shoreline variability at your site, increase the value of this parameter. This may be the case for large nourishments or eroding/accreting coastlines.

</details>

Additionally, there is the option to visualise the images and create a timelapse (MP4) using the code below:

<details> <summary><strong>Timelapse animation</strong></summary>

You can save JPEGs of each satellite image using the function SDS_preprocess.save_jpg(metadata, settings) and create a timelapse using SDS_tools.make_animation_mp4(fp_images, fps, fn_animation) (also shown in the notebook). This can be handy to visualise your data before processing the shorelines and make sure cloud masking and preprocessing are fine.

# preprocess images and save as jpg
SDS_preprocess.save_jpg(metadata, settings, use_matplotlib=True)
# create MP4 timelapse animation
fn_animation = os.path.join(inputs['filepath'],inputs['sitename'], '%s_animation_RGB.mp4'%inputs['sitename'])
fp_images = os.path.join(inputs['filepath'], inputs['sitename'], 'jpg_files', 'preprocessed')
fps = 4 # frames per second in animation
SDS_tools.make_animation_mp4(fp_images, fps, fn_animation)
</details>

Run the shoreline detection

Once all the settings have been defined, the batch shoreline detection can be launched by calling:

output = SDS_shoreline.extract_shorelines(metadata, settings)

To quality-control the detections manually, set check_detection to True, and a figure like the one below will pop up and let the user manually accept/reject each detection by pressing on the keyboard the right arrow (⇨) to keep the shoreline or left arrow (⇦) to skip the mapped shoreline. The user can break the loop at any time by pressing escape (nothing will be saved though).

map_shorelines

For further control on the detections (especially in meso/macrotidal coastal environments), the user can set adjust_detection to True. In that case, the threshold used to define the shoreline can be manually adjusted on each image. See the animation below that shows how the shoreline position can be adjusted in a gentle-sloping macrotidal beach in France (Truc Vert).

<details> <summary><strong>Adjust thresholds (Truc Vert, France)</strong></summary>

If adjust_detection was set to True in settings, a figure like the one below appears and let the user adjust the position of the shoreline by clicking on the histogram of MNDWI pixel intensities. Once the threshold has been adjusted, press Enter and then accept/reject the image with the keyboard arrows. This is very time-consuming but allows more control on the mapped shorelines and is faster that digitising in a GIS environment.

Alt text

</details>

Once all the shorelines have been mapped, the output is saved in two different formats (under /filepath/data/SITENAME):

The GeoJSON shorelines can be opened in a GIS software (QGIS) as shown below.

<p align="center"> <img width="500" height="300" src="https://user-images.githubusercontent.com/7217258/49361401-15bd0480-f730-11e8-88a8-a127f87ca64a.jpeg"> </p>

While the default settings work in most cases, there are instances where you will need to adjust the Advanced Parameters described below.

<details> <summary><strong> Advanced shoreline detection parameters</strong></summary>

There are some additional parameters that can be modified to optimise the shoreline detection:

</details>

Finally, the provided classifiers may not be able to detect sand accurately at certain beaches so you have the option to re-train your own classifier in a separate notebook.

<details> <summary><strong> Re-training the classifier</strong></summary>

CoastSat's shoreline mapping alogorithm uses an image classification scheme to label each pixel into 4 classes: sand, water, white-water and other land features. While this classifier has been trained using a wide range of different beaches, it may be that it does not perform very well at specific sites that it has never seen before. You can try the different classifiers already available in the /classification folder by changing the settings['sand_color'] parameter to latest, dark or bright, but if none of those fit your data you can label images of your own site and uses those labels to train a new classifier specifically for your site. This process is implemented in a separate Jupyter notebook re-train CoastSat classifier located in the /classification folder.

Once you have trained your new classifier, a new .pkl file will be created in /classification/models. In order to use it for shoreline mapping, you will need to open the script SDS_shorelines located in /coastsat and go to lines 132-134 and change the filename to match your new classifier .pkl file.

</details>

2.3 Shoreline intersections with transects<a name="analysis"></a>

This section shows how to process the satellite-derived shoreline to obtain time-series along shore-normal transects.

3 options are provided to define the coordinates of the transects:

  1. Interactively draw shore-normal transects along the mapped shorelines:
transects = SDS_transects.draw_transects(output, settings)
  1. Load the transect coordinates from a .geojson file:
transects = SDS_tools.transects_from_geojson(path_to_geojson_file)
  1. Create the transects by manually providing the coordinates of two points:
transects = dict([])
transects['Transect 1'] = np.array([[342836, ,6269215], [343315, 6269071]])
transects['Transect 2'] = np.array([[342482, 6268466], [342958, 6268310]])
transects['Transect 3'] = np.array([[342185, 6267650], [342685, 6267641]])

:warning: each transect is defined by two points, its origin and a second point that defines its length and orientation. The origin is always defined first and located landwards, the second point is located seawards.

:warning: if you choose options 2 or 3, make sure that the points that you are providing are in the spatial reference system defined by settings['output_epsg'], otherwise they won't match the shorelines.

An example of how to draw the transects is shown below. transects

To calculate the intersections between the shorelines and the transects, there are 2 modes: simple and quality-controlled.

<details> <summary><strong> Simple intersections</strong></summary>

This simple intersections are computed by the following function:

settings['along_dist'] = 25
cross_distance = SDS_transects.compute_intersection(output, transects, settings)

The parameter along_dist defines the along-shore distance around the transect over which shoreline points are selected to compute the intersection. The default value is 25 m, which means that the intersection is computed as the median of the points located within 25 m of the transect (50 m alongshore-median). This helps to smooth out localised water levels in the swash zone.

</details> <details> <summary><strong> Quality-controlled intersections (RECOMMENDED)</strong></summary>

There is also a more advanced function to compute the intersections SDS_transects.compute_intersection_QA(), which provides more quality-control and can deal with small loops, multiple intersections, false detections etc. It is recommended to use this function as it can provide cleaner shoreline time-series. An example of parameter values is provided below, the default parameters should work in most cases (leave as it is if unsure).

settings_transects = { # parameters for computing intersections
                      'along_dist':          25,        # along-shore distance to use for computing the intersection
                      'min_points':          3,         # minimum number of shoreline points to calculate an intersection
                      'max_std':             15,        # max std for points around transect
                      'max_range':           30,        # max range for points around transect
                      'min_chainage':        -100,      # largest negative value along transect (landwards of transect origin)
                      'multiple_inter':      'auto',    # mode for removing outliers ('auto', 'nan', 'max')
                      'auto_prc':            0.1,       # percentage of the time that multiple intersects are present to use the max
                     }
cross_distance = SDS_transects.compute_intersection_QC(output, transects, settings_transects)

The multiple_inter setting helps to deal with multiple shoreline intersections along the same transect. This is quite common, for example when there is a lagoon behind the beach and the transect crosses two water bodies. The function will try to identify this cases and the user can choose whether to:

</details>

2.4 Tidal Correction and Beach slope estimation<a name="correction"></a>

Each satellite image is captured at a different stage of the tide, therefore a tidal correction is necessary to remove the apparent shoreline changes cause by tidal fluctuations.

In order to tidally-correct the time-series of shoreline change, two inputs are needed: 1) tide level time-series and 2) an estimate of the beach slope.

In the notebook, the user has two options for the tide level time-series:

example_tides

Once you have the tide levels, you need an estimate of the beach slope. You can provide this manually (e.g., 0.1 for all transects) or you can also estimate it using the satellite-derived shorelines and tide levels inside the notebook, see the Beach Slope estimation scetion. This parts uses the CoastSat.slope repository, for more details on the methodology see Vos et al. 2020 (preprint available here).

2.5 Post-processing (seasonal averages and linear trends)<a name="postprocessing"></a>

The tidally-corrected time-series can be post-processed to remove outliers with a despiking algorithm SDS_transects.reject_outliers(). This function was developed to remove obvious outliers in the time-series by removing the points that do not make physical sense in a shoreline change setting. For example, the shoreline can experience rapid erosion after a large storm, but it will then take time to recover and return to its previous state. Therefore, if the shoreline erodes/accretes suddenly of a significant amount (max_cross_change) and then immediately returns to its previous state, this spike does not make any physical sense and can be considered an outlier.

settings_outliers = {'max_cross_change':   40,             # maximum cross-shore change observable between consecutive timesteps
                     'otsu_threshold':     [-.5,0],        # min and max intensity threshold use for contouring the shoreline
                     'plot_fig':           True,           # whether to plot the intermediate steps
                    }
cross_distance = SDS_transects.reject_outliers(cross_distance,output,settings_outliers)

Additionally, this function also checks that the Otsu thresholds used to map the shoreline are within the typical range defined by otsu_threshold, with values outside this range (typically -0.5 to 0) identified as outliers.

otsu_threhsolds

Additionally, a set of functions to compute seasonal averages, monthly averages and linear trends on the shoreline time-series are provided.

SDS_transects.seasonal_averages()

NA1_seasonally

SDS_transects.monthly_averages()

NA1_monthly

:warning: given that the shoreline time-series are not uniformly sampled and there is more density of datapoints towards the end of the record (more satellite in orbit), it is best to estimate the long-term trends on the seasonally-averaged shoreline time-series as the trend estimated on the raw time-series may be biased towards the end of the record.

2.6 Validation against survey data<a name="validation"></a>

This section provides code to compare the satellite-derived shorelines against the survey data for Narrabeen, available at http://narrabeen.wrl.unsw.edu.au/.

comparison_transect_PF1

Contributing and Issues<a name="issues"></a>

Having a problem? Post an issue in the Issues page (please do not email).

If you are willing to contribute, check out our todo list in the Projects page.

  1. Fork the repository (https://github.com/kvos/coastsat/fork). A fork is a copy on which you can make your changes.
  2. Create a new branch on your fork
  3. Commit your changes and push them to your branch
  4. When the branch is ready to be merged, create a Pull Request (how to make a clean pull request explained here)

References and Datasets<a name="references"></a>

This section provides a list of references that use the CoastSat toolbox as well as existing shoreline datasets extracted with CoastSat.

Publications

Datasets