Skip to content

Restructure benchmark scripts #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Sashan merged 3 commits into from
Oct 28, 2025
Merged

Restructure benchmark scripts #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2f421b2
- update README.md
Sashan Oct 23, 2025
d98740a
Restructure
jogme Oct 24, 2025
cd54037
fixup! Restructure
jogme Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions bench-scripts/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,63 @@ Requirements for ubuntu are the following:
- libtool
- g++

## Running scripts

Each benchmark here is implemented by its configuration script which sets up
complete environment (for example `bench_config_apache.sh`) and execution
script which runs the benchmark (for example `bench_run_apache.sh`). All script
which are meant to be executed follow that naming convention:
- `bench_config_*.sh` script builds and configures all tools required to run
the desired benchmark
- `bench_run_*.sh` script executes the benchmarks
There are no command line options all scripts are controlled via environment
variables. The common environment variables which are used by both scripts
are as follows:
- `BENCH_INSTALL_ROOT` - installation prefix for tools and libraries,
`/tmp/bench.binaries` is default
- `BENCH_WORKSPACE_ROOT` - directory used to download and build tools and
libraries, `/tmp/bench.workspace` is default
- `BENCH_MAKE_OPTS` - command line options for `make`, none by default. You may
want to add `-j` option here.
- `BENCH_HTTPS_PORT` - port to use by https server, default is 4430
- `BENCH_HTTP_PORT` - port to use by http server, default is 8080

The `bench_config_*.sh` script further use those two variables here to
control generation of self-signed certificates:
- `BENCH_CERT_SUBJ` - used to generate self-signed certificate, it's
set to `/CN=localhost` by default
- `BENCH_ALT_SUBJ` - it's set to `'subjectAltName=DNS:localhost,IP:127.0.0.1`
by default
Default values are good enough when testing is conveyed over loopback network
interface. You may need to change those values when testing with different
IP addresses/hostnames.

The environment variables which are specific to `bench_run_*.sh` scripts are
as follows:
- `BENCH_RESULT_DIR` directory where to save collected results, it's
`$BENCH_INSTALL_ROOT/results` by default
- `BENCH_HOST` remote server used by test client tools, `localhost` by
default.
- `BENCH_TEST_TIME` duration of each test run, it's 5 minutes by default
`5M` (tools are using siege [6] client tool)

## Contributing

The scripts are written in ksh shell which is widely available on many POSIX
platforms. Those scripts are meant to be portable as much as possible, please
do not introduce platform specific dependencies there.

When it comes time to implement a script which collects data using kernel
counters (or other mean which is specific to platform), then new script should
be implemented (or derived) from existing scripts. The new script should follow
the naming convention introduced here. All new variables which control script
behavior should be prefixed with `BENCH_`. The file should be named as
`bench_run_apache_xyz.sh` where `xyz` should be replaced by platform name (for
example linux).

Similarly when adding a script which will use a xtool to collect performance data
should be named as `bench_run_xtool.sh`

[1]: https://www.openssl.org/

[2]: https://www.wolfssl.com/
Expand Down
304 changes: 2 additions & 302 deletions bench-scripts/apache_bench.sh → bench-scripts/bench_config_apache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,20 +53,17 @@ set -x
#

INSTALL_ROOT=${BENCH_INSTALL_ROOT:-"/tmp/bench.binaries"}
RESULT_DIR=${BENCH_RESULTS:-"${INSTALL_ROOT}/results"}
Comment thread
jogme marked this conversation as resolved.
WORKSPACE_ROOT=${BENCH_WORKSPACE_ROOT:-"/tmp/bench.workspace"}
MAKE_OPTS=${BENCH_MAKE_OPTS}
HTTPS_PORT=${BENCH_HTTPS_PORT:-'4430'}
HTTP_PORT=${BENCH_HTTP_PORT:-'8080'}
CERT_SUBJ=${BENCH_CERT_SUBJ:-'/CN=localhost'}
Comment thread
jogme marked this conversation as resolved.
CERT_ALT_SUBJ=${BENCH_CERT_ALT_SUBJ:-'subjectAltName=DNS:localhost,IP:127.0.0.1'}
TEST_TIME=${BENCH_TEST_TIME:-'5M'}
HOST=${BENCH_HOST:-'127.0.0.1'}
APACHE_VERSION='2.4.65'
HAPROXY='no'
Comment thread
jogme marked this conversation as resolved.

. ./common_util.sh
. ./haproxy_bench.sh
. ./bench_config_haproxy.sh

function install_wolfssl_for_apache {
typeset VERSION=$1
Expand Down Expand Up @@ -744,183 +741,6 @@ function config_apache {
generate_download_files "${INSTALL_ROOT}/${SSL_LIB}/htdocs"
}

function enable_mpm_event {
typeset SSL_LIB=$1
if [[ -z "${SSL_LIB}" ]] ; then
SSL_LIB='openssl-master'
fi
typeset CONF_FILE="${INSTALL_ROOT}/${SSL_LIB}/conf/httpd.conf"

#
# comment out currently loaded mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^LoadModule mpm_.*$\)/#\1/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1

#
# enable event mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^#\)\(LoadModule mpm_event_module .*$\)/\2/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1
}

function enable_mpm_worker {
typeset SSL_LIB=$1
if [[ -z "${SSL_LIB}" ]] ; then
SSL_LIB='openssl-master'
fi
typeset CONF_FILE="${INSTALL_ROOT}/${SSL_LIB}/conf/httpd.conf"

#
# comment out currently loaded mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^LoadModule mpm_.*$\)/#\1/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1

#
# enable worker mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^#\)\(LoadModule mpm_worker_module .*$\)/\2/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1
}

function enable_mpm_prefork {
typeset SSL_LIB=$1
if [[ -z "${SSL_LIB}" ]] ; then
SSL_LIB='openssl-master'
fi
typeset CONF_FILE="${INSTALL_ROOT}/${SSL_LIB}/conf/httpd.conf"

#
# comment out currently loaded mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^LoadModule mpm_.*$\)/#\1/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1

#
# enable pre-fork mpm module
#
cp "${CONF_FILE}" "${CONF_FILE}".wrk
sed -e 's/\(^#\)\(LoadModule mpm_prefork_module .*$\)/\2/g' \
"${CONF_FILE}".wrk > "${CONF_FILE}" || exit 1
}

function run_test {
typeset SSL_LIB=$1
typeset HAPROXY=$2
typeset i=0
typeset PORT=${HTTPS_PORT}
typeset PROTOCOL="https"
if [[ -z "${SSL_LIB}" ]] ; then
SSL_LIB='openssl-master'
fi
if [[ -z "${HAPROXY}" ]] ; then
HAPROXY='no'
fi
typeset RESULTS="${SSL_LIB}".txt
if [[ "${SSL_LIB}" = 'nossl' ]] ; then
SSL_LIB='openssl-master'
RESULTS='nossl.txt'
PORT=${HTTP_PORT}
PROTOCOL="http"
fi
if [[ "${HAPROXY}" != 'no' ]] ; then
RESULTS="haproxy-${SSL_LIB}-${HAPROXY}.txt"
fi
typeset HTDOCS="${INSTALL_ROOT}/${SSL_LIB}"/htdocs
typeset SIEGE="${INSTALL_ROOT}"/openssl-master/bin/siege

#
# we always try to use siege from openssl master by default,
# if not found then we try the one which is installed for
# openssl version we'd like to test.
#
if [[ ! -x "${SIEGE}" ]] ; then
echo "no ${SIEGE}"
exit 1
fi

#
# generate URLs for sewage
#
# The different modes for haproxy are:
# no: client - server
# no-ssl: client -http- haproxy -http- server
# server: client -https- haproxy -http- server
# client: client -http- haproxy -https- server
# both: client -https- haproxy -https- server
#
# Otherwise said, haproxy is a client when it encrypts the outgoing encryption;
# or it's client side.
#
rm -f siege_urls.txt
for i in `ls -1 ${HTDOCS}/*.txt` ; do
if [[ "${HAPROXY}" = "no" ]] ; then
echo "${PROTOCOL}://${HOST}:${PORT}/`basename $i`" >> siege_urls.txt
elif [[ "${HAPROXY}" = "no-ssl" ]] ; then
echo "http://${HOST}:${HAPROXY_NOSSL_PORT}/`basename $i`" >> siege_urls.txt
elif [[ "${HAPROXY}" = "server" ]] ; then
echo "https://${HOST}:${HAPROXY_C2P_PORT}/`basename $i`" >> siege_urls.txt
elif [[ "${HAPROXY}" = "client" ]] ; then
echo "http://${HOST}:${HAPROXY_P2S_PORT}/`basename $i`" >> siege_urls.txt
elif [[ "${HAPROXY}" = "both" ]] ; then
echo "https://${HOST}:${HAPROXY_C2S_PORT}/`basename $i`" >> siege_urls.txt
fi
done

if [[ "${HAPROXY}" = "server" ]] || [[ "${HAPROXY}" = "both" ]] ; then
conf_siege_haproxy_cert $SSL_LIB
fi

#
# start apache httpd server
#
LD_LIBRARY_PATH=${INSTALL_ROOT}/${SSL_LIB}/lib \
${INSTALL_ROOT}/${SSL_LIB}/bin/httpd -k start || exit 1
if [[ $? -ne 0 ]] ; then
echo "could not start ${INSTALL_ROOT}/${SSL_LIB}/bin/httpd"
exit 1
fi
LD_LIBRARY_PATH=${INSTALL_ROOT}/openssl-master/lib "${SIEGE}" -t ${TEST_TIME} -b \
-f siege_urls.txt 2> "${RESULT_DIR}/${RESULTS}"
if [[ $? -ne 0 ]] ; then
echo "${INSTALL_ROOT}/${SSL_LIB} can not run siege"
cat "${RESULT_DIR}/${RESULTS}"
exit 1
fi

LD_LIBRARY_PATH=${INSTALL_ROOT}/${SSL_LIB}/lib \
${INSTALL_ROOT}/${SSL_LIB}/bin/httpd -k stop || exit 1
sleep 1
pgrep httpd
while [[ $? -eq 0 ]] ; do
sleep 1
LD_LIBRARY_PATH=${INSTALL_ROOT}/${SSL_LIB}/lib \
${INSTALL_ROOT}/${SSL_LIB}/bin/httpd -k stop || exit 1
echo "stopping ${INSTALL_ROOT}/${SSL_LIB}/bin/httpd"
pgrep httpd
done

#
# save apache configuration used for testing along the results.
# we do care about httpd.conf and httpd-ssl.conf only as only
# those two were modified.
#
cp ${INSTALL_ROOT}/${SSL_LIB}/conf/httpd.conf \
${RESULT_DIR}/httpd-${SSL_LIB}.conf
cp ${INSTALL_ROOT}/${SSL_LIB}/conf/extra/httpd-ssl.conf \
${RESULT_DIR}/httpd-ssl-${SSL_LIB}.conf

if [[ "${HAPROXY}" = "server" ]] || [[ "${HAPROXY}" = "both" ]] ; then
unconf_siege_haproxy_cert
fi
}

function setup_tests {
typeset i=0
install_openssl master
Expand Down Expand Up @@ -1006,127 +826,7 @@ function setup_tests {
clean_build
}

function run_tests {
typeset SAVE_RESULT_DIR="${RESULT_DIR}"
typeset HAPROXY_OPTIONS=('no' 'client' 'server' 'both')
typeset i=""

for i in event worker pre-fork ; do
mkdir -p ${RESULT_DIR}/$i || exit 1
done

enable_mpm_event
RESULT_DIR="${SAVE_RESULT_DIR}/event"
run_test nossl
run_haproxy
run_test nossl 'no-ssl'
kill_haproxy
for i in 3.0 3.1 3.2 3.3 3.4 3.5 3.6 ; do
enable_mpm_event openssl-${i}
run_haproxy openssl-${i}
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-${i} ${OPTION}
done
kill_haproxy
done
enable_mpm_event openssl-master
run_haproxy openssl-master
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-master ${OPTION}
done
kill_haproxy
enable_mpm_event OpenSSL_1_1_1-stable
run_test OpenSSL_1_1_1-stable
enable_mpm_event libressl-4.1.0
run_test libressl-4.1.0
#enable_mpm_event wolfssl-5.8.2
#run_test wolfssl-5.8.2
enable_mpm_event boringssl
run_test boringssl
enable_mpm_event aws-lc
run_test aws-lc

enable_mpm_worker
RESULT_DIR="${SAVE_RESULT_DIR}/worker"
run_test nossl
run_haproxy
run_test nossl 'no-ssl'
kill_haproxy
for i in 3.0 3.1 3.2 3.3 3.4 3.5 3.6 ; do
enable_mpm_worker openssl-${i}
run_test openssl-${i}
run_haproxy openssl-${i}
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-${i} ${OPTION}
done
kill_haproxy
done
enable_mpm_worker openssl-master
run_haproxy openssl-master
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-master ${OPTION}
done
kill_haproxy
enable_mpm_worker OpenSSL_1_1_1-stable
run_test OpenSSL_1_1_1-stable
enable_mpm_worker libressl-4.1.0
run_test libressl-4.1.0
#enable_mpm_worker wolfssl-5.8.2
#run_test wolfssl-5.8.2
enable_mpm_worker boringssl
run_test boringssl
enable_mpm_worker aws-lc
run_test aws-lc

enable_mpm_prefork
RESULT_DIR="${SAVE_RESULT_DIR}/pre-fork"
run_test nossl
run_haproxy
run_test nossl 'no-ssl'
kill_haproxy
for i in 3.0 3.1 3.2 3.3 3.4 3.5 3.6 ; do
enable_mpm_prefork openssl-${i}
run_test openssl-${i}
run_haproxy openssl-${i}
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-${i} ${OPTION}
done
kill_haproxy
done
enable_mpm_prefork openssl-master
run_haproxy openssl-master
for OPTION in ${HAPROXY_OPTIONS[@]}
do
run_test openssl-master ${OPTION}
done
kill_haproxy
enable_mpm_prefork OpenSSL_1_1_1-stable
run_test OpenSSL_1_1_1-stable
enable_mpm_prefork libressl-4.1.0
run_test libressl-4.1.0
#enable_mpm_prefork wolfssl-5.8.2
#run_test wolfssl-5.8.2
enable_mpm_prefork boringssl
run_test boringssl
enable_mpm_prefork aws-lc
run_test aws-lc

RESULT_DIR=${SAVE_RESULT_DIR}
}

check_env
setup_tests
run_tests
SAVE_RESULT_DIR=${RESULT_DIR}
for i in event worker pre-fork ; do
RESULT_DIR=${SAVE_RESULT_DIR}/$i
plot_results
done
RESULT_DIR=${SAVE_RESULT_DIR}

echo "testing using siege is complete, results can be found ${RESULT_DIR}:"
echo "apache was successfully configured."
Loading