Home

Awesome

how-to-optimize-gemm

English | 简体中文

News

2023/08 aarch64 add cmake and mperf, try -DMPERF_ENABLE=ON !

Introduction

row-major matmul optimization tutorial

backendarmv7aarch64aarch64-int8cudacuda-int4vulkanx86
support✔️✔️✔️✔️-✔️

All backends and corresponding tutorials

backendtutorial
aarch64GEMM 入门
aarch64GEMM caching
aarch64-int8-
armv7ARMv7 4x4kernel 懒人优化小实践
cudacuda 入门的正确姿势:how-to-optimize-gemm
cuda-int4 WIPint4 炼丹要术
vulkan如何火急火燎地上手 Vulkan

Build and run

Usage is similar for all backends:

  1. Open the backend directory to be used, and change the OLD and NEW of makefile to the same implementation for the first run, for example
$ cd aarch64
$ cat makefile
OLD    := MMult_4x4_10
NEW   := MMult_4x4_10
..
  1. makefilewill compile and run the implementation whichNEWpoint at, and copyoutput_MMult_4x4_10.mtooutput_new.m`
$ make run
$ cat output_new.m
  1. It may not be intuitive to look at the numbers directly, so draw a line chart
$ python3 -m pip install -r ../requirements.txt
$ python3 plot.py

Differences between backends

Specific to each hardware, there are subtle differences:

1. armv7 and aarch64

A. Prepare armv7/aarch64 linux development environment, Raspberry Pi/rk3399/aws arm server are all fine.

B. By default ARCH := native, build and run directly

$ cd armv8 && make run

2. aarch64 int8

chgemm is an int8 gemm library.

Compared to the code in this tutorial, the differences are:

  1. Dealing with the boundary problem, unlike the tutorial where only multiples of 4 are considered;
  2. Int8 reaches a maximum of 18.6 gflops (relative to the theoretical limit of fp32 is only 14.3 on RK3399, gemmlowp is about 12-14gflops);
  3. Based on symmetric quantization, input value range must be in [-127, +127], and -128 cannot appear;
  4. Built-in small example about how to integrate into android studio

chgemm has been merged into ncnn INT8 convolution implementation.

3. x86 original

flame referenced by x86 is the original implementation, with some differences from this repo:

  1. The original is column-major x86 SSE version
  2. Both are tutorials, and the MMult_4x4_17.c written now can reach 70% of the armv8.1 CPU peak
  3. The boundary problem is not dealt with now, only the case where MNK is a multiple of 4 is considered; sub_kernel also only writes the simplest kind of assembly. Practical needs a simple adjustment;
  4. In terms of drawing, octave was discarded (it is too troublesome to configure the environment once for embedded devices), and python was used instead.

4. CUDA

This version is faster than NVIDIA cuBLAS

  1. Need to install cuda driver and nvcc by yourself
  2. CPU OpenBLAS is required to be the baseline
$ apt install libopenblas-dev

5. Vulkan

  1. vulkan build depends on kompute API packaging, see vulkan build documentation for details

  2. More about how to learn compute shader

6. CUDA int4

WIP

Some Tools

License

GPLv3