Home

Awesome

<div align="center"> <img src="logo.png" width="400" alt="Mobile AI Bench" /> </div>

License pipeline status

FAQ | 中文

In recent years, the on-device deep learning applications are getting more and more popular on mobile phones or IoT devices. It's a challenging task for the developers to deploy their deep learning models in their mobile applications or IoT devices.

They need to optionally choose a cost-effective hardware solution (i.e. chips and boards), then a proper inference framework, optionally utilizing quantization or compression techniques regarding the precision-performance trade-off, and finally run the model on one or more of heterogeneous computing devices. How to make an appropriate decision among these choices is a tedious and time-consuming task.

Mobile AI Benchmark (i.e. MobileAIBench) is an end-to-end benchmark tool which covers different chips and inference frameworks, with results include both speed and model accuracy, which will give insights for developers.

Daily Benchmark Results

Please check benchmark step in daily CI pipeline page, due to the lack of test devices, the CI result may not cover all hardwares and frameworks.

FAQ

Q: Why are benchmark results not stable on my device?

A: Due to power save considerations, some SoCs have aggressive and advanced power control scheduling to reduce power consumption which make performance quite unstable (especially CPU). Benchmark results highly depend on states of devices, e.g., running processes, temperature, power control policy. It is recommended to disable power control policy (as shown in tools/power.sh) if possible (e.g., rooted phone). Otherwise, keep your device at idle state with low temperature, and benchmark one model on one framework each time.

Q: Why do some devices run faster (or slower) than expected in the CI benchmark result?

A: Some devices is rooted and has some specialized performance tuning while some others is not rooted and failed to make such tuning (see the code for more details).

Q: Why is ncnn initialization time much less than others?

A: ncnn benchmark uses fake model parameters and skips loading weights from filesystem.

Q: Does benchmark use all available cores of devices?

A: Most modern Android phones use ARM big.LITTLE architecture which can lead to significant variance between different runs of the benchmark, we use only available big cores to reduce this variance by taskset command for MACE/NCNN/TFLITE benchmark. Moreover, there are no well-defined APIs for SNPE to bind to big cores and set thread count. Thread count can be set by adding --num_threads to tools/benchmark.sh command.

Environment requirement

MobileAIBench supports several deep learning frameworks (called executor in this project, i.e., MACE, SNPE, ncnn, TensorFlow Lite and HIAI) currently, which may require the following dependencies:

SoftwareInstallation commandTested version
Python2.7
ADBapt-get install android-tools-adbRequired by Android run, >= 1.0.32
Android NDKNDK installation guideRequired by Android build, r15c
Bazelbazel installation guide0.13.0
CMakeapt-get install cmake>= 3.11.3
FileLockpip install -I filelock==3.0.0Required by Android run
PyYamlpip install -I pyyaml==3.123.12.0
shpip install -I sh==1.12.141.12.14
SNPE (optional)download and uncompress1.18.0

Note 1: SNPE has strict license that disallows redistribution, so the default link in the Bazel WORKSPACE file is only accessible by the CI server. To benchmark SNPE in your local system (i.e. set --executors with all or SNPE explicitly), you need to download the SDK here, uncompress it, copy libgnustl_shared.so and modify WORKSPACE as the following:

#new_http_archive(
#    name = "snpe",
#    build_file = "third_party/snpe/snpe.BUILD",
#    sha256 = "8f2b92b236aa7492e4acd217a96259b0ddc1a656cbc3201c7d1c843e1f957e77",
#    strip_prefix = "snpe-1.22.2.233",
#    urls = [
#        "https://cnbj1-fds.api.xiaomi.net/aibench/third_party/snpe-1.22.2_with_libgnustl_shared.so.zip",
#    ],
#)

new_local_repository(
    name = "snpe",
    build_file = "third_party/snpe/snpe.BUILD",
    path = "/path/to/snpe",
)

Note 2: HIAI has strict license that disallows redistribution, so the default link in the Bazel WORKSPACE file is only accessible by the CI server. To benchmark HIAI in your local system (i.e. set --executors with all or HIAI explicitly), you need to login and download the SDK here, uncompress it and get the HiAI_DDK_100.200.010.011.zip file, uncompress it and modify WORKSPACE as the following:

#new_http_archive(
#    name = "hiai",
#    build_file = "third_party/hiai/hiai.BUILD",
#    sha256 = "8da8305617573bc495df8f4509fcb1655ffb073d790d9c0b6ca32ba4a4e41055",
#    strip_prefix = "HiAI_DDK_100.200.010.011",
#    type = "zip",
#    urls = [
#        "http://cnbj1.fds.api.xiaomi.com/aibench/third_party/HiAI_DDK_100.200.010.011_LITE.zip",
#    ],
#)

new_local_repository(
    name = "hiai",
    build_file = "third_party/hiai/hiai.BUILD",
    path = "/path/to/hiai",
)

Architecture

+-----------------+         +------------------+      +---------------+
|   Benchmark     |         |   BaseExecutor   | <--- | MaceExecutor  |
+-----------------+         +------------------+      +---------------+
| - executor      |-------> | - executor       |
| - model_name    |         | - device_type    |      +---------------+
| - quantize      |         |                  | <--- | SnpeExecutor  |
| - input_names   |         +------------------+      +---------------+
| - input_shapes  |         | + Init()         |
| - output_names  |         | + Prepare()      |      +---------------+
| - output_shapes |         | + Run()          | <--- | NcnnExecutor  |
| - run_interval  |         | + Finish()       |      +---------------+
| - num_threads   |         |                  |
+-----------------+         |                  |      +---------------+
| - Run()         |         |                  | <--- | TfLiteExecutor|
+-----------------+         |                  |      +---------------+
        ^     ^             |                  |
        |     |             |                  |      +---------------+
        |     |             |                  | <--- | HiaiExecutor  |
        |     |             +------------------+      +---------------+
        |     |
        |     |             +--------------------+
        |     |             |PerformanceBenchmark|
        |     --------------+--------------------+
        |                   | - Run()            |
        |                   +--------------------+
        |
        |                   +---------------+      +---------------------+                           
+--------------------+ ---> |PreProcessor   | <--- |ImageNetPreProcessor |
| PrecisionBenchmark |      +---------------+      +---------------------+
+--------------------+
| - pre_processor    |      +---------------+      +---------------------+
| - post_processor   | ---> |PostProcessor  | <--- |ImageNetPostProcessor|
| - metric_evaluator |      +---------------+      +---------------------+
+--------------------+
| - Run()            |      +---------------+
+--------------------+ ---> |MetricEvaluator|
                            +---------------+

How To Use

Benchmark Performance of all models on all executors

bash tools/benchmark.sh --benchmark_option=Performance \
                        --target_abis=armeabi-v7a,arm64-v8a,aarch64,armhf

The whole benchmark may take a few time, and continuous benchmarking may heat the device very quickly, so you may set the following arguments according to your interests. Only MACE supports precision benchmark right now.

optiontypedefaultexplanation
--benchmark_optionstrPerformanceBenchmark options, Performance/Precision.
--output_dirstroutputBenchmark output directory.
--executorsstrallExecutors(MACE/SNPE/NCNN/TFLITE/HIAI), comma separated list or all.
--device_typesstrallDeviceTypes(CPU/GPU/DSP/NPU), comma separated list or all.
--target_abisstrarmeabi-v7aTarget ABIs(armeabi-v7a,arm64-v8a,aarch64,armhf), comma separated list.
--model_namesstrallModel names(InceptionV3,MobileNetV1...), comma separated list or all.
--run_intervalint10Run interval between benchmarks, seconds.
--num_threadsint4The number of threads.
--input_dirstr""Input data directory for precision benchmark.

Configure ssh devices

For embedded ARM-Linux devices whose abi is aarch64 or armhf, ssh connection is supported. Configure ssh devices in generic-mobile-devices/devices_for_ai_bench.yml, for example:

devices:
  nanopi:
    target_abis: [aarch64, armhf]
    target_socs: RK3333
    models: Nanopi M4
    address: 10.231.46.118
    username: pi

Adding a model to run on existing executor

Adding a new AI executor

License

Apache License 2.0.

Notice

For third party dependencies, please refer to their licenses.