Home

Awesome

tflite-react-native

A React Native library for accessing TensorFlow Lite API. Supports Classification, Object Detection, Deeplab and PoseNet on both iOS and Android.

Table of Contents

Installation

$ npm install tflite-react-native --save

iOS (only)

TensorFlow Lite is installed using CocoaPods:

  1. Initialize Pod:

    cd ios
    pod init
    
  2. Open Podfile and add:

    target '[your project's name]' do
    	pod 'TensorFlowLite', '1.12.0'
    end
    
  3. Install:

    pod install
    

Automatic link

$ react-native link tflite-react-native

Manual link

iOS

  1. In XCode, in the project navigator, right click LibrariesAdd Files to [your project's name]
  2. Go to node_modulestflite-react-native and add TfliteReactNative.xcodeproj
  3. In XCode, in the project navigator, select your project. Add libTfliteReactNative.a to your project's Build PhasesLink Binary With Libraries
  4. Run your project (Cmd+R)<

Android

  1. Open up android/app/src/main/java/[...]/MainApplication.java
  1. Append the following lines to android/settings.gradle:
    include ':tflite-react-native'
    project(':tflite-react-native').projectDir = new File(rootProject.projectDir,   '../node_modules/tflite-react-native/android')
    
  2. Insert the following lines inside the dependencies block in android/app/build.gradle:
      compile project(':tflite-react-native')
    

Add models to the project

iOS

In XCode, right click on the project folder, click Add Files to "xxx"..., select the model and label files.

Android

  1. In Android Studio (1.0 & above), right-click on the app folder and go to New > Folder > Assets Folder. Click Finish to create the assets folder.

  2. Place the model and label files at app/src/main/assets.

  3. In android/app/build.gradle, add the following setting in android block.

    aaptOptions {
        noCompress 'tflite'
    }

Usage

import Tflite from 'tflite-react-native';

let tflite = new Tflite();

Load model:

tflite.loadModel({
  model: 'models/mobilenet_v1_1.0_224.tflite',// required
  labels: 'models/mobilenet_v1_1.0_224.txt',  // required
  numThreads: 1,                              // defaults to 1  
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});

Image classification:

tflite.runModelOnImage({
  path: imagePath,  // required
  imageMean: 128.0, // defaults to 127.5
  imageStd: 128.0,  // defaults to 127.5
  numResults: 3,    // defaults to 5
  threshold: 0.05   // defaults to 0.1
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});
{
  index: 0,
  label: "person",
  confidence: 0.629
}

Object detection:

SSD MobileNet

tflite.detectObjectOnImage({
  path: imagePath,
  model: 'SSDMobileNet',
  imageMean: 127.5,
  imageStd: 127.5,
  threshold: 0.3,       // defaults to 0.1
  numResultsPerClass: 2,// defaults to 5
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});

Tiny YOLOv2

tflite.detectObjectOnImage({
  path: imagePath,
  model: 'YOLO',
  imageMean: 0.0,
  imageStd: 255.0,
  threshold: 0.3,        // defaults to 0.1
  numResultsPerClass: 2, // defaults to 5
  anchors: [...],        // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828]
  blockSize: 32,         // defaults to 32 
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});

x, y, w, h are between [0, 1]. You can scale x, w by the width and y, h by the height of the image.

{
  detectedClass: "hot dog",
  confidenceInClass: 0.123,
  rect: {
    x: 0.15,
    y: 0.33,
    w: 0.80,
    h: 0.27
  }
}

Deeplab

tflite.runSegmentationOnImage({
  path: imagePath,
  imageMean: 127.5,      // defaults to 127.5
  imageStd: 127.5,       // defaults to 127.5
  labelColors: [...],    // defaults to https://github.com/shaqian/tflite-react-native/blob/master/index.js#L59
  outputType: "png",     // defaults to "png"
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});

PoseNet

Model is from StackOverflow thread.

tflite.runPoseNetOnImage({
  path: imagePath,
  imageMean: 127.5,      // defaults to 127.5
  imageStd: 127.5,       // defaults to 127.5
  numResults: 3,         // defaults to 5
  threshold: 0.8,        // defaults to 0.5
  nmsRadius: 20,         // defaults to 20 
},
(err, res) => {
  if(err)
    console.log(err);
  else
    console.log(res);
});

x, y are between [0, 1]. You can scale x by the width and y by the height of the image.

[ // array of poses/persons
  { // pose #1
    score: 0.6324902,
    keypoints: {
      0: {
        x: 0.250,
        y: 0.125,
        part: nose,
        score: 0.9971070
      },
      1: {
        x: 0.230,
        y: 0.105,
        part: leftEye,
        score: 0.9978438
      }
      ......
    }
  },
  { // pose #2
    score: 0.32534285,
    keypoints: {
      0: {
        x: 0.402,
        y: 0.538,
        part: nose,
        score: 0.8798978
      },
      1: {
        x: 0.380,
        y: 0.513,
        part: leftEye,
        score: 0.7090239
      }
      ......
    }
  },
  ......
]

Release resources:

tflite.close();

Example

Refer to the example.