Awesome

Image Classification in MATLAB Using TensorFlow

This example shows how to call a TensorFlow™ model from MATLAB® using co-execution with Python®.

There are different options for accessing deep learning models within MATLAB. These include:

Using models created in MATLAB using the Deep Learning Toolbox
Converting models from other frameworks into MATLAB
Co-executing models from other frameworks with MATLAB

This example provides an overview of how to perform 3. Co-execution. For a further example on co-execution see Hyperparameter Tuning in MATLAB using Experiment Manager & TensorFlow.

For reference, the following table provides a comparison for all options.

What is co-execution?

Co-execution between MATLAB and TensorFlow is when both frameworks are used together, in executing a single application. Co-execution can achieve this by passing data to and from each framework. It requires MATLAB and Tensorflow to be installed on the same machine.

Requirements

The example requires the following to be installed:

MATLAB R2021b or later
MATLAB Live Editor task for Python - Note: This must be installed prior to running the examples in this repo.
Tensorflow 2.8
Python 3.8+

For more information about installing Python, see Install Supported Python Implementation.

Why perform co-execution?

There are many benefits to co-execution. These include:

You can leverage the features and capabilities of both frameworks together in a single application. For example, MATLAB performs the pre-and post-processing of data when training a model in TensorFlow.
Teams working in multiple frameworks can build applications together. This reduces the need to:
- Rely on a single language
- Manually re-code code from one language to another.
- Force some team members to work in their non-preferred environment.

What are the limitations of co-execution?

Performance - Due to data passed between frameworks, there is a latency introduced. In most situations, the latency overhead is negligible, e.g., when performing model training. In other situations, e.g., performing inference at high throughputs, the latency impact is higher.
Automatic code generation to embedded-devices. MATLAB can automatically generate code for the following: C/C++ using MATLAB Coder, CUDA using GPU Coder and, VHDL using Deep Learning HDL Toolbox, including the deep learning model and pre and post-processing steps. This deployment option is not available when co-execution with TensorFlow.
- Note: For versions R2022a or newer, MATLAB supports integration with TensorFlow Lite (TFLite) pretrained models. This enables the simulation of TensorFlow Lite models from both MATLAB and Simulink. For code generation, MATLAB generates code for pre and/or post-processing and generates a call to the TensorFlow Lite interpreter on a supported target.
Datatype conversion and data reformatting - Only select data types in both frameworks are supported for co-execution. This is because not all data types from one framework can be mapped and translated into an equivalent type in the other framework. For a list of data types that can be used, see MATLAB to Python Data Type Mapping,

How can co-execution be performed?

In this repo, 2 workflows for performing co-execution are presented.

1. MATLAB calling a TensorFlow model using a Live Editor task

2. MATLAB calling a TensorFlow model using MATLAB commands

MATLAB calling a TensorFlow model using Live Editor tasks <a name="matlabcallingtflivetask"/>

Using the MATLAB Live Editor task for Python enables users to write or copy & paste Python code into MATLAB.

Steps to use the MATLAB Live Editor task for Python are:

Write or copy & paste Python code into the Live Editor task
Define MATLAB input variables to be called in Python
Define Python output variables to be called in MATLAB

Example code available here: MATLAB calling TensorFlow model for Image Classification using a Live Editor task.mlx

Read in image

imgOriginal = imread("./Images/banana.png");
imshow(imgOriginal)

Each pretrained model in tensorflow.keras.applications takes input Images of different sizes. Therefore the image being classified needs to be resized.

imageHWSize = 480;
img = imresize(imgOriginal, [imageHWSize, imageHWSize]);

TensorFlow orients image data in a different format to MATLAB. This requires conversion (HWCN TO NHWC)

imgforTF = permute(img, [4 1 2 3]); 
batch_size = int32(1); % Tensorflow require inputs to be converted to int32.

Import TensorFlow model using the Live Task for Python

MATLAB calling a TensorFlow model using MATLAB commands <a name="matlabcallingtf"/>

Example code available here: ImageClassificationinMATLABusingTensorFlow.m

Configuring python setup:

The script checkPythonSetup contains commands to help set up the python environment. You don't need to run these commands, unless the default Python configuration causes errors.

checkPythonSetup

For more information on setting up or troubleshooting the Python Environment in MATLAB see Calling Python from MATLAB

Loading an image:

imgOriginal = imread("./Images/banana.png");
imshow(imgOriginal)

Preparing inputs:

imageHWSize = 480;
img = imresize(imgOriginal, [imageHWSize, imageHWSize]);

% TensorFlow orients image data in a different format to MATLAB. This 
% requires conversion (HWCN TO NHWC)
imgforTF = permute(img, [4 1 2 3]);

batch_size = int32(1); % Tensorflow require inputs to be converted to int32.

Importing model directly into MATLAB:

model = py.tensorflow.keras.applications.efficientnet_v2.EfficientNetV2L();

Calling a TensorFlow pretrained model for image classification:

% converting input from MATLAB array into Python array.
X = py.numpy.asarray(imgforTF);

% call preprocessing function that is required for the image input in Keras.
X = py.tensorflow.keras.applications.efficientnet_v2.preprocess_input(X); 

% classify image 
Y = model.predict(X, batch_size); 

% label of classification output
label = py.tensorflow.keras.applications.efficientnet_v2.decode_predictions(Y);

Note that many pretrained models are available for use directly in MATLAB without the need for co-execution.

Gathering and displaying the classification result in MATLAB:

label = label{1}{1}{2}; % The label is stored in a nested cell. In the file layer of the cell there is a tuple (id, class, probability) - The predicted class label is the 2nd element of the tuple
labelStr = string(label); 
imshow(imgOriginal);
title(labelStr,Interpreter="none");

Comparison of Models accessible in MATLAB <a name="comparison-table"/>

Capability	Models created using the Deep Learning Toolbox	Models Converted from other Frameworks	Co-execution
Integrates with pre and post processing with MATLAB
Requires installation of MATLAB products only
Supports debugging from MATLAB
Offers best inference performance in MATLAB and Simulink
Comes with many MATLAB application examples
Requires no datatype conversion and data reformatting
Provides largest coverage for embedded code generation with MATLAB Coder, GPU Coder & Deep Learning HDL Toolbox
Requires no additional libraries for standalone deployment with MATLAB Compiler
Accesses popular models in a single line of code
Access to models from TensorFlow and PyTorch

Key:

Most support and / or low effort Some support and / or some effort Little to no support and / or high effort