Awesome

Pytorch element-wise operation optimization benchmark

1. Abstract

Providing a benchmark for element-wise operation performance evaluation on CPU.

Tested CPU：

Tested operations:

Conclusions:

• OpenMP threshold which is set to 100k in official version is too high for contiguous tensors of small and medium size to benefit from OpenMP parallelism.
• Discontiguous tensors' operations can be boosted significantly by Intel Pytorch .
• The optimal OpenMP threshold is dependent on the operation type and CPU type.
  • OpenMP threshold becomes smaller for more complex operations.
  • OpenMP threshold of discontiguous tensor is usually lower than that of contiguous tensor.

annotation:
OpenMP threshold -- If the size of a tensor is larger than the value, the operations run in parallel, otherwise in serial.

This benchmark also gives a rough estimation of optimal OpenMP threshold of copy, add, div, exp, sin, sum and prod operation on different types of CPU.

For contiguous tensor operation:

For discontiguous tensor operation:

2. Major work

• Optimal OpenMP threshold is identified to fully exploit performance potentiality on CPU
  The OpenMP threshold of official Pytorch is set to 100K. However, the evidence gained by benchmarking copy, add, div, exp, sin operation in both contiguous and discontiguous cases on different CPU types shows that the value is too high. A rough estimation of optimal OpenMP threshold is also proposed for those operations.

• Discontiguous tensors' operation parallelization with OpenMP
  Slice operation of tensor is very common in science computation. Using slice operation will generate discontiguous tensor. Meanwhile, Official Pytorch does not support parallelism of discontiguous tensor at the moment. Our main work is trying to fill this blank. Code available at dev-omp and upstreaming is in progress.

3. Installation and test

3.1 Installation

Official Pytorch

Please refer to official link

Intel Pytorch

Download Intel pytorch source code.

git clone --recursive -b dev-omp2 https://github.com/intel/pytorch.git

Before installing, you should set the CMAKE_PREFIX_PATH.

export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" # [anaconda root directory]

Install intel Pytorch

python setup.py install

3.2 Test

python benchmark.py <CONTIGUITY> <OPERATION> [OUTPUT FILENAME] 

Positional arguments:
CONTIUITY—— operands' contiguity, ontiguous/discontiguous
OPERATION—— operation, copy/add/div/sin/exp/sum/prod

Optional arguments:
o output filename——output filename, output.log is in default

4. The benchmark result

4.1 Contiguous Tensor Operation OpenMP Threshold Tuning

Add, exp operation for contiguous tensors whose sizes range from 1K to 100K are listed here as test cases. We compiled two versions of official Pytorch by setting two different OpenMP threshold. The threshold of one version is set to 100K to make all of the test case runs in series. Meanwhile the threshold of the other one is set to 800 to make all of the test case in parallel.

Platform: Platinum 8180
Operation: add
Tensor Continuity: contiguous
Unit: microsecond

Time cost result is below:

Conclusion: Setting the threshold to 80K is good for add operation of contiguous tensors.

Platform: Platinum 8180
Operation: exp
Tensor Continuity: contiguous
Unit: microsecond

Time cost result is below:

Conclusion: Setting the threshold to 1K is good for exponential operation of contiguous tensors.

From above results, it is easy to understand that,

• Different operations have their own optimal OpenMP threshold, but 100K is not suitable.
• OpenMP threshold becomes smaller for more complex operations.

We don't list all the detailed data for div, sin, sum and prod operation but provide a rough estimation of optimal OpenMP threshold for different operations.

4.2 Discontiguous tensor operation parallelization

Add and exp operation performance for discontiguous tensors whose sizes range from 1k to 180k are listed. Official pytorch does not optimize operations for discontiguous tensors with OpenMP but Intel version does. In order to expalin that OpenMP also do good in discontiguous tensor operations and to find a optimal OpenMP threshold, we compiled two versions of Pytorch. One is the Official Pytorch. The other one is the Intel one whose OpenMP threshold is set to 800 to make all test cases run in parallel.

Platform: Platinum 8180
Operation: add
Tensor Continuity: discontiguous
Unit: microsecond

Time cost result is below:

Conclusion: Setting the threshold to 10K is good for add operation of discontiguous tensors.

Platform: Platinum 8180
Operation: exp
Tensor Continuity: discontiguous
Unit: microsecond

Time cost result is below:

Conclusion: Setting the threshold to 1K is good exponential operation of contiguous tensors.

Conclusions:

• Discontiguous operation can be improved a lot by using OpenMP optimization.
• OpenMP threshold of discontiguous tensor is usually lower than that of contiguous tensor because the same operation of discontiguous tensor is more time-consuming than contiguous tensor.

4.3 LSTM benchmark test

To consolidate the performance boost benefiting from the elementwise optimization, we choose the a widely-used RNN unit: LSTM as the model-level benchmark reference. This is because:

1. LSTM related computations involve considerable elementwise operations;
2. PyTorch provides a scalable and flexible Python API to execute LSTM computation.

We retrieve the LSTM benchmark via the script: https://github.com/xhzhao/pytorch-rnn-benchmark , and in which,

1. The Python API torch.nn.LSTM is used as the entry of LSTM computation.
2. We run the benchmarks on 24 selective input shapes utilized by different NLP models,
3. The unit for benchmarks is Sentence Per Second (SPS). [N, T, D, Z] stands for batch size, embedding size, sentence length and hidden size. Specifically, The [64, 50, 500, 500] is used by OpenNMT. The [64, 25, 4096, 4096] is used by Deepbench.

Platform: Platinum-8180
Phase: Inference
Unit: SPS(Scentence per Sencond)

Platform: Platinum-8180
Phase: Training
Unit: SPS(Scentence per Sencond)

Platform: CPU E5-2699 v4
Phase: Inference
Unit: SPS(Scentence per Sencond)

Platform: CPU E5-2699 v4
Phase: Training
Unit: SPS(Scentence per Sencond)

Conclusion:

According to the benchmarks retrieved on Intel Xeon Platforms, On Platinum 8180:

1. For LSTM inference (forward-only), the performance is get boosted from 1.25X to 8.22X.
2. For LSTM training (forward + backward), the performance is get boosted from 1.23X to 1.74X.

On E5-2699 V4:

1. For LSTM inference (forward-only), the performance is get boosted from 1.64X to 7.56X.
2. For LSTM training (forward + backward), the performance is get boosted from 1.01X to 3.03X.

Test results analysis:

1. For inference benchmarks: As the contributions of elementwise operation varies from the different input shapes, it is expected the performance boosts are not uniform with input shape changing.
2. For training benchmarks: Apart from sharing the same reason of inference benchmarks. As the backward computation gains less from the elementwise optimization, it is expected the performance boosts on training benchmarks are not outstanding as inference benchmarks, and not uniform with input shape changing.