benchmark openssl with more heavy tools

[Sashan]

The existing perftools we have allow us to monitor for performance regression. They compare openssl with openssl. This is good enough to catch any performance regression early in development. This can be considered as a lightweight performance testing.

We should also do a performance testing using 3rd party tools. For example Apache with mod_ssl nginx with openssl etc. This should allow us to see how openssl performance changes from version to version, but also it should tell us how well/bad openssl is doing when compared with competing libraries.

[Sashan]

I've got two scripts one uses apache the other uses nginx. I think nginx is more viable to move forward as I was able to get it working with openssl-3.x libressl-4.1.0, wolfssl-5.8.2 and boringssl. The only library which I failed to get working is aws-lc. I'd like to investigate HA-proxy which seems to work with every SSL library which is around.

the apache works with openssl and libressl. It builds with wolfssl but I was not able to get it working. the apache receives client helo and replies with nothing the connection just stales.

To obtain some performance data I use siege. The command line argument which runs siege reads as follows:

 siege -f siege_urls.txt \  # list of URL to test
     -t 5M \ # test duration is 5 minutes
     -b 2> results.txt # results are saved to file, -b means benchmark mode

the siege is always linked with libcrypto/libssl from openssl master branch using LD_LIBRARY_PATH.

the siege client is continuosly fetching this set of files:

-rw-r--r-- 1 sashan docker      64 Sep  2 23:45 test.txt
-rw-r--r-- 1 sashan docker     128 Sep  2 23:45 test_1.txt
-rw-r--r-- 1 sashan docker   65536 Sep  2 23:45 test_10.txt
-rw-r--r-- 1 sashan docker  131072 Sep  2 23:45 test_11.txt
-rw-r--r-- 1 sashan docker  262144 Sep  2 23:45 test_12.txt
-rw-r--r-- 1 sashan docker  524288 Sep  2 23:45 test_13.txt
-rw-r--r-- 1 sashan docker 1048576 Sep  2 23:45 test_14.txt
-rw-r--r-- 1 sashan docker 2097152 Sep  2 23:45 test_15.txt
-rw-r--r-- 1 sashan docker 4194304 Sep  2 23:45 test_16.txt
-rw-r--r-- 1 sashan docker     256 Sep  2 23:45 test_2.txt
-rw-r--r-- 1 sashan docker     512 Sep  2 23:45 test_3.txt
-rw-r--r-- 1 sashan docker    1024 Sep  2 23:45 test_4.txt
-rw-r--r-- 1 sashan docker    2048 Sep  2 23:45 test_5.txt
-rw-r--r-- 1 sashan docker    4096 Sep  2 23:45 test_6.txt
-rw-r--r-- 1 sashan docker    8192 Sep  2 23:45 test_7.txt
-rw-r--r-- 1 sashan docker   16384 Sep  2 23:45 test_8.txt
-rw-r--r-- 1 sashan docker   32768 Sep  2 23:45 test_9.txt

I was using ubunti-24 we have on proxmox. the instance has 8GB ram and 16 CPUS if I remember correct

the libraries are built with default settings I just modify --prefix where to install them. refer to script here I'd like to review the data and eventually submit it as PR.

I'm attaching the results.tgz with collected data. Please keep in mind those data are very preliminary. I need to verify if numbers make some sense. perhaps better test environement is needed. I also need to double check I'm bulding wolfssl nginx correctly the wolfssl build process is not straightforward. I could make an error which hampers performance of nginx with wolfssl.

results.tgz

[Sashan]

number of transactions done in 5 minutes.

[Sashan]

throuput in MB/sec

[Sashan]

average response time.

[t8m]

I see some strange things - like almost negligible difference among the openssl versions. I would expect at least 3.0 being much slower. We also need to ensure that the same algorithms are proposed & negotiated - for example it would be unfair if 3.5 was penalized for offering the hybrid keyshare.

[Sashan]

There is not much information in the the state of ssl stacks on how server and client are configured nor information on how SSL is configured, nor how the tests were conducted. With those details missing, it's hard to make up a mind/opinion on this. So I've decided to get my hands on performance tests. The script I have is the first step. Everything is left in default configuration. For example just switching nginx option worker_process 1 to worker_process auto yields completely different picture.

results.tgz

[Sashan]

the chart above shows number of transactions in 5 minutes. the nginx is configured with worker_process auto What's even more confusing here is that eprformance of openssl remains the same as in the 1-worker nginx variant. The other libraries just drop significantly here.I have no explanation for this at the moment.

[bob-beck]

You are certain you are doing a release build and not a debug build for all those projects right? (I would expect major performance differences if, for example, anything was built unoptomized as debug.

[Sashan]

the config/build flags for OpenSSL are as follows:

./Configure --prefix="${INSTALL_ROOT}/${DIRNAME}" \
        --libdir="${INSTALL_ROOT}/${DIRNAME}/lib"

for libressl it's:

./configure --prefix="${INSTALL_ROOT}/${BUILD_DIR}"

for boringssl/aws it's:

cmake -B build -DCMAKE_INSTALL_PREFIX="${INSTALL_ROOT}/${BORING_NAME}" \ 
-DBUILD_SHARED_LIBS=1 \
-DCMAKE_BUILD_TYPE=Release

for wolfssl it's

./configure --prefix="${INSTALL_ROOT}/${DIRNAME}" \ 
--enable-nginx || exit 1

if I understand the build process right for those libs, then I'm building the release build.

[Sashan]

there is a PR with scripts I was using. I think those can be merged so more people can start playing around with this.

openssl/perftools#49