Optimize inference performance of ERNIE INT8 on CPU

[bingyanghuang]

Now, paddle ERNIE fp32 inference on CPU performance is ass below:
single thread： 251.464 m
20 threads：29.8818 ms
Our goal is to prove that with INT8 real kernel, ERNIE can get the performance gain.
@Sand3r- @wojtuss Please update your benchmark progress here.
@wzzju @luotao1 Please track the status here.

[wojtuss]

We are currently working with the following two models: ernie_quant.zip and fp32_model.tar.gz.
Results gathered with the optimized fp32_model model will be labeled FP32.
Results gathered with the fake quantized FP32 ernie_quant model will be labeled QAT FP32.
Results gathered with an optimized INT8 model obtained from ernie_quant model will be labeled QAT INT8.

Initial performance results using the former (for FP32, 4 inputs, test_ds dataset) and the latest (for QAT FP32/INT8, 2 inputs, 1.8w.bs1 dataset) unit test obtained from Baidu.
(20 threads, bs 1, CLX 6248)
FP32:
Run 5010 samples, average latency: 55.7539 ms per sample.
Run 5009 samples, average latency [exclude 1 warmup steps]: 55.7415 ms per sample.
QAT FP32:
Run 2490 samples, average latency: 92.2313ms per sample.
QAT INT8 (with mul):
Run 2490 samples, average latency: 47.1024ms per sample.

[wojtuss]

I have just noticed the above results were mixed. They are updated now.

[wojtuss]

FP32 results with @GaoWei8's fix (PaddlePaddle/Paddle#20972):
Run 5010 samples, average latency: 32.0582 ms per sample.

[bingyanghuang]

Attached the baseline profile results:

• FP32 with Gaowei's padding patch

Event	Calls	Total	Min.	Max.	Ave.	Ratio.
fc	370666	127408	0.01446	20.14810	0.34373	0.75962
elementwise_add	190342	10271.7	0.03167	4.93884	0.05396	0.06124
transpose2	240432	7704.78	0.02098	3.71560	0.03205	0.04594
matmul	125225	7266.52	0.02186	3.96825	0.05803	0.04332
scale	70126	4823.14	0.00407	4.92560	0.06878	0.02876
layer_norm	125225	4184.69	0.02592	4.72700	0.03342	0.02495
softmax	60108	2565.96	0.03276	2.04443	0.04269	0.01530
reshape2	240432	1383.47	0.00393	2.92184	0.00575	0.00825
lookup_table	15027	1331.17	0.06888	1.23500	0.08859	0.00794
stack	5009	462.643	0.06707	0.30953	0.09236	0.00276
tanh	5009	181.429	0.02849	0.12309	0.03622	0.00108
slice	5009	68.8743	0.01152	0.02409	0.01375	0.00041
fetch	5009	40.985	0.00489	0.01910	0.00818	0.00024
feed	20036	33.6655	0.00066	0.01238	0.00168	0.00020

• Real INT8 ERNIE profile

Event	Calls	Total	Min.	Max.	Ave.	Ratio.
elementwise_add	290080	48707.3	0.00604	7.29221	0.16791	0.39068
mul	191660	41987.3	0.02023	4.70104	0.21907	0.33678
reshape2	124320	9530.43	0.02449	0.37624	0.07666	0.07644
quantize	129500	5801.52	0.00967	2.96898	0.04480	0.04653
transpose2	124320	4886.13	0.01829	3.68880	0.03930	0.03919
matmul	64750	3845.23	0.02139	15.42340	0.05939	0.03084
scale	36260	2754.43	0.00546	0.14959	0.07596	0.02209
relu	31080	2548.26	0.05381	2.56348	0.08199	0.02044
layer_norm	64750	2127.01	0.02338	1.18454	0.03285	0.01706
softmax	31080	1192.7	0.02731	2.38116	0.03838	0.00957
lookup_table	7770	707.691	0.04808	0.23893	0.09108	0.00568
stack	2590	260.936	0.06343	0.27896	0.10075	0.00209
slice	7770	74.5761	0.00523	0.02923	0.00960	0.00060
tanh	2590	57.4421	0.01791	0.08974	0.02218	0.00046
unsqueeze2	5180	45.0293	0.00550	0.02471	0.00869	0.00036
fill_constant	10360	37.9978	0.00214	0.02271	0.00367	0.00030
fetch	2590	23.3609	0.00753	0.01779	0.00902	0.00019
range	2590	23.1647	0.00674	0.01581	0.00894	0.00019
expand	2590	18.4988	0.00649	0.01439	0.00714	0.00015
cast	5180	16.361	0.00223	0.01195	0.00316	0.00013
equal	2590	12.2261	0.00411	0.01321	0.00472	0.00010
feed	5180	11.6075	0.00127	0.00955	0.00224	0.00009
shape	2590	5.52088	0.00176	0.01129	0.00213	0.00004

[bingyanghuang]

1. FP32 ERNIE model and INT8 ERNIE model have slightly different on inputs numbers

2. INT8 optimized latency is slower than FP32 optimized latency because

   • elementwise_add takes a lot of time after quantizing the mul, fc int8 op should be introduced (support for 3 dims fc input) elementwise_add bias/residual ?

   • Too much reorder time, INT8 pipeline for feed-forward part can be further optimizing the model

   • INT8 execution on GEMM with MKL is without padding (UT MKLDNN INT8 GEMM with the same dimension with FP32 GEMM )

[bingyanghuang]

Schedule plan of ERNIE INT8 optimization:

ERNIE INT8 Task	Status	Plan to Finish	Risk	Developer
Accuracy check in UT	WIP	12.06	Low	Asia
MKL-DNN inner product performance benchmark	UT benchmark  done	12.06	Low	Wojtuss
3D support in FC INT8 kernel	WIP	12.13	High	Michal
Fusion: FC INT8	Not started	12.13	Dependency	-
Fusion: FC+scale	WIP	12.13	Dependency	Danqing
Reshape2 + Transpose2  INT8 kernel verification	WIP	12.13	Dependency	Asia & Wojtuss
Overall performance & accuracy tuning	Not started	12.20	Dependency	-

[bingyanghuang]

MKL-DNN 0.20 from paddle
aef88b7c2
20 threads
Dimension	128x768x768	128x3072x768	128x768x3072
u8s8s32s32	0.0545178	0.170797	0.129593
s8s8s32s32	0.102495	0.326905	0.320389
f32	0.135169	0.419238	0.271828
f32 -> s8 increase	24%	22%	-18%
1 thread
Dimension	128x768x768	128x3072x768	128x768x3072
u8s8s32s32	0.21483	0.884561	0.904923
s8s8s32s32	0.638012	2.62521	2.57349
f32	0.848266	3.6114	3.40055
f32 -> s8 increase	25%	27%	24%

[wozna]

@luotao1
I'm working on accuracy test on ernie2 right now .
I've run test on:
ernie_origin_2_inputs.tar.gz fp32 model with 2 inputs
data file - 1.8w.bs1
label file - label.xnli.dev that looks like 2 0 1 2 0 1 0 1 2 1 0 2 ...

I got outputs like below:
--- iteration 204 --- 0.0957182 0.0770019 0.243675 --- iteration 205 --- 0.0957183 0.0770018 0.243675 --- iteration 206 --- 0.0957183 0.0770018 0.243675 --- iteration 207 --- 0.0957182 0.0770018 0.243675 --- iteration 208 --- 0.0957182 0.0770017 0.243675

My question is: am I using the correct file or is there any process for calculating accuracy?

[bingyanghuang]

MKL-DNN master (v 1.1)
1ee831fa6a2f802de1d399fe1de4e6cc629ad855
20 threads
problem descriptor	128x768x768	128x3072x768	128x768x3072
u8s8s32	0.0446907	0.162134	0.132759
s8s8s32	0.0455766	0.15978	0.13624
f32	0.081141	0.283924	0.467588
f32 -> s8 increase	44%	44%	71%
1 thread
problem descriptor	128x768x768	128x3072x768	128x768x3072
u8s8s32	0.216017	0.925034	0.901553
s8s8s32	0.221654	0.947028	0.942317
f32	0.742698	2.96747	6.87472
f32 -> s8 increase	70%	68%	86%

[wojtuss]

Below are our latest performance results for Ernie FP32 and INT8 runs. The tests were run with affinity settings

export KMP_AFFINITY=granularity=fine,compact,1,0
export KMP_BLOCKTIME=1

on CLX 6248. INT8 tests were run with the memory_optimize_pass disabled, due to PaddlePaddle/Paddle#21492.

With the current develop branch (1fd1f06) and the origin FP32 model, the latency was:
FP32, 20 threads: 29.10 ms,
FP32, 1 thread: 180.95 ms.

With additional commits from PRs

• FC INT8 3D support: [DNNL] 3D Fully-Connected Paddle#21746
• FC+dequantize squash: Add fc-dequantize squash in cpu_quantize_squash_pass for ernie model Paddle#21714
• MKL-DNN 1.1.1 update: Update MKL-DNN to 1.1 Paddle#21754

the latency for INT8 (with FC quantized only) was:
INT8, 20 threads: 23.73 ms (18.45% faster than FP32),
INT8, 1 thread: 84.32 ms (53.40% faster than FP32).
After enabling INT8 kernels of reshape2 and transpose2 (support already present in develop branch), the results were:
INT8, 20 threads: 20.47 ms (29.68% faster than FP32),
INT8, 1 thread: 80.64 ms (55.43% faster than FP32).

With additional optimizations possible to implement:

• avoiding reorder nwc->ncw before layer_norm (here imitated by removing the reorder),
• fusing scale ops into fc or into dequantize (here imitated by removing the scale ops),

the latency for INT8 (with FC quantized only) was:
INT8, 20 threads: 20.68 ms (28.93% faster than FP32),
INT8, 1 thread:: 77.84 ms (56.99% faster than FP32).
With INT8 kernels also for reshape2 and transpose2, the latency was:
INT8, 20 threads: 17.39 ms (40.24% faster than FP32),
INT8, 1 thread: 75.66 ms (58.19% faster than FP32).

Other optimizations, like quantization of more operators and more fuses, are also investigated.

[wozna]

I repeat my question.
Im checking accuracy on trained ernie models with 2 inputs. But outputs still are float type.
It looks like that:
--- iteration 0 --- -2.22873 -1.84079 3.08339 ...
And label file - label.xnli.dev consists of integers: 2 0 1 2 0 1 0 1 2 1 0 2 ...

Should output labels have a floating point type?

[Meiyim]

The output float number is the logits of a 3-label classification task. and the label file consists of class_id of the true label.
so you can evaluate the accuracy with python code below:

import sys
import numpy as np

all, acc = 0, 0
for i in sys.stdin:
        a, b, c, d = i.strip().split('\t')
        pred = np.array(map(float, (a, b, c))).argmax()
        if pred == int(d):
                acc += 1
        all += 1
print(float(acc)/all)

run with:

paste score_3_columns label.xnli.dev | python eval.py

[wozna]

@Meiyim you are right. Thank you for the explanation

[wojtuss]

I have updated the results in #275 (comment) above. Results with additionally quantized reshape2 and transpose2 are added.