Some details on critical path, alongside initial guides for Score-P and Scalasca

book.bib

@@ -79,6 +79,68 @@ @article{williams2009
 numpages = {12}
 }
 
+% Critical path analysis
+@INPROCEEDINGS{bohme2012,
+  author={Böhme, David and Wolf, Felix and de Supinski, Bronis R. and Schulz, Martin and Geimer, Markus},
+  booktitle={2012 IEEE 26th International Parallel and Distributed Processing Symposium}, 
+  title={Scalable Critical-Path Based Performance Analysis}, 
+  year={2012},
+  volume={},
+  number={},
+  pages={1330-1340},
+  keywords={Runtime;Optimization;Resource management;Synchronization;Performance analysis;Analytical models},
+  doi={10.1109/IPDPS.2012.120}
+}
+
+% OSACA (ref 1)
+@INPROCEEDINGS{laukemann2018,
+author={J. {Laukemann} and J. {Hammer} and J. {Hofmann} and G. {Hager} and G. {Wellein}},
+booktitle={2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)},
+title={Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures},
+year={2018},
+volume={},
+number={},
+pages={121-131}, doi={10.1109/PMBS.2018.8641578}
+}
+
+% OSACA (ref 2)
+@INPROCEEDINGS{laukemann2019,
+author={J. {Laukemann} and J. {Hammer} and G. {Hager} and G. {Wellein}},
+booktitle={2019 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)},
+title={Automatic Throughput and Critical Path Analysis of x86 and ARM Assembly Kernels},
+year={2019},
+volume={},
+number={},
+pages={1-6}, doi={10.1109/PMBS49563.2019.00006}
+}
+
+% original scalasca reference
+@article{geimer2010,
+author = {Geimer, Markus and Wolf, Felix and Wylie, Brian J. N. and Ábrahám, Erika and Becker, Daniel and Mohr, Bernd},
+title = {The Scalasca performance toolset architecture},
+journal = {Concurrency and Computation: Practice and Experience},
+volume = {22},
+number = {6},
+pages = {702-719},
+keywords = {parallel computing, performance analysis, scalability},
+doi = {https://doi.org/10.1002/cpe.1556},
+url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.1556},
+eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpe.1556},
+year = {2010}
+}
+
+
+@ARTICLE{feld2026,
+AUTHOR={Feld, Christian  and Calotoiu, Alexandru  and Corbin, Gregor  and Geimer, Markus  and Hermanns, Marc-André  and Knespel, Maximilian  and Mohr, Bernd  and Reuter, Jan André  and Sander, Maximilian  and Saviankou, Pavel  and Schlütter, Marc  and Schöne, Robert  and Shende, Sameer S. S. and Visser, Anke  and Wesarg, Bert  and Williams, William R.  and Wolf, Felix  and Wylie, Brian J. N. J. and Zarubin, Mikhail },
+TITLE={The Score-P performance tools ecosystem},
+JOURNAL={Frontiers in High Performance Computing},
+VOLUME={Volume 3 - 2025},
+YEAR={2026},
+URL={https://www.frontiersin.org/journals/high-performance-computing/articles/10.3389/fhpcp.2025.1709051},
+DOI={10.3389/fhpcp.2025.1709051},
+ISSN={2813-7337}
+}
+
 % CUBE GUI (original reference)
 @inproceedings{geimer2007,
   title={Scalable Collation and Presentation of Call-Path Profile Data with CUBE.},

main.tex

@@ -65,8 +65,6 @@ \part{Deep Dive}
 \input{workflow/workflow}
 \input{workflow/localisation}
 \input{workflow/navigation}
-\input{workflow/affinity}
-\input{workflow/process-placement}
 
 \input{core/core}
 \input{core/compiler-optimisations}
@@ -97,6 +95,7 @@ \part{Resources}
 \input{tools/aps}
 \input{tools/vtune}
 \input{tools/scorep}
+\input{tools/scalasca}
 \input{tools/map}
 \input{tools/maqao}
 \input{tools/amplifier}

tools/scalasca.tex

@@ -0,0 +1,32 @@
+\section{Scalasca} 
+\label{section:localisation:scalasca}
+
+Building from the infrastructure of Score-P, we now turn to Scalasca to perform
+trace analyses which can gather lots of data and generate further metrics to
+further aid the profiles we can generate with Score-P and visualise with Cube.
+The power of the metrics generated by Scalasca is that they can help the
+analyst dig into the root causes of some performance issues, whilst also
+supporting an analysis with critical path analysis as described in
+Section~\ref{section:workflow:localisation}.
+
+Using the concepts of instrumentation, profiling and filtering from
+Section~\ref{section:localisation:scorep} we can then rerun the instrumented
+executable now through the \texttt{scan} command as below
+\begin{verbatim}
+scan -t mpiexec ./code
+\end{verbatim}
+This then performs a trace analysis which generates further files in our
+experiment directory. As before we can perform some post-processing on this
+directory using \texttt{square}
+\begin{verbatim}
+square -s trace_exp/
+\end{verbatim}
+and finally visualise the \texttt{trace.cubex} file in the CubeGUI.
+
+Finally, we will not go into details on interpreting the results from Scalasca
+in Cube here. As always the documentation is our first recommendation
+\footnote{\url{https://www.scalasca.org/}}. However, we do draw the reader's
+attention to the added metrics our profiles now have which include MPI
+communication issues such as wait states, critical path information including
+imbalances on the critical path. These can be very powerful metrics, but do
+require experience in interpreting.

tools/scorep.tex

@@ -1,2 +1,136 @@
 \section{Score-P} 
 \label{section:localisation:scorep}
+
+We here give an overview of getting started with building profiles of parallel
+applications using Score-P which is a performance infrastructure
+\cite{feld2026}. We here will look briefly at using the Score-P compiler
+wrappers to build instrumented binaries which, when executed, produce profiles
+that can be visualised with the Cube GUI. This section is just a brief
+introduction some of the functionality for Score-P in performance analysis, we
+also recommend the reader to refer to the official documentation for further
+information \footnote{\url{https://score-p.org/}}. It should also be noted that
+Score-P offers a user instrumentation API to target specific kernels and
+functions, but here we focus on using the Score-P compiler wrappers to
+instrument the entire binary.
+
+An individual Score-P install is built with a specific combination of compiler
+and MPI library, meaning on clusters multiple Score-P installs are often
+required. It is best to reach out to cluster support teams for help with
+getting the relevant combinations of compiler and Score-P as it might be useful
+to compare compilers during analysis.
+
+It should be noted that as with many profiling and performance tools, an
+application's runtime can increase (potentially quite significantly). Hence, it
+is always advised to run the application without any tooling to note the
+runtime and also resource usage such as maximum memory footprint.
+
+\subsection{Instrumentation}
+
+We load our installation of Score-P and then check that we have the Score-P
+compiler wrappers typically of the form \texttt{scorep-x} for a given compiler,
+\texttt{x}. If building with a build system like CMake it is best to set the
+\texttt{SCOREP\_WRAPPER=off} environment variable to stop instrumentation during
+configuration.
+
+This compilation will then have produced an instrumented binary which we can
+run in a job script as we before (making sure to load in the Score-P library),
+though there is some good practice to consider first. As with tools like MAQAO,
+Score-P will write output files to an experiment directory which will by
+default be given a long name based on the time and date. Instead, we can be
+more exact and set an experiment directory name which is more meaningful with
+the \texttt{export SCOREP\_EXPERIMENT\_DIRECTORY=...} environment variable.
+
+Once the code is finished, there will be a local experimental directory which
+will contain several files such as the files ending with \texttt{*.cubex} which
+can be visualised in the Cube GUI tool. However, we can also do some further
+post-processing with the \texttt{square -s <experiment-directory>}, and then
+visualise with the CubeGUI. For more information in using the CubeGUI please
+refer to the documentation in Ref~\cite{}. Here we are mainly focused on a
+workflow of generating data in Score-P, we discuss data interpretation
+elsewhere in the document. Next, we touch on several configurations to gather
+more low-level information during the profiling.
+
+\subsubsection{Performance counters}
+
+Many performance tools allow the user to gather information on hardware
+performance counters. With Score-P we can, for example, set the
+\begin{verbatim}
+export SCOREP_METRIC_PAPI=PAPI_TOT_INS,PAPI_FP_OPS
+\end{verbatim}
+environment variable which will gather hardware counters via PAPI, where here
+we have selected the total instructions and floating point operations. The
+availability of PAPI counters can be found by the \texttt{papi\_avail} command. 
+
+We can also use performance metrics from Linux \texttt{perf}, but we refer the
+reader to the Score-P documentation for further information.
+
+These performance counters then appear in the CubeGUI as further metrics which
+we can breakdown across the further dimensions. However, as always it can be
+easy to generate a lot of data; performance counters can be very powerful but
+it useful to have an aim in collecting them, to guide an analysis.
+
+\subsubsection{Memory}
+
+Beyond hardware performance counters, Score-P also supports memory based
+metrics through the
+\begin{verbatim}
+export SCOREP_MEMORY_RECORDING=true
+export SCOREP_MPI_MEMORY_RECORDING=true
+\end{verbatim}
+environment variables which can help detecting memory related issues such as
+memory leaks. The support for these features can vary across languages and, as
+with performance counters, it is useful to keep in mind that gathering this
+data should only be carried out if there is an aim for the analysis.
+
+\subsection{Filtering}
+
+A crucial aspect of using tools like Score-P is filtering. Running instrumented
+binaries can show large increases in runtime, and in the next section we can
+see how to use the Score-P infrastructure to produce trace analyses which can
+generate very large amounts of data. Hence, filtering is a major step in this
+workflow.
+
+To begin with we can use the \texttt{scorep-score} command to generate a
+summary of the experiment directory as below
+\begin{verbatim}
+scorep-score ./profile.cubex
+\end{verbatim}
+Running this command on the Score-P generated profile we get predictions of how
+large a trace analysis would be with the current setup and often these can be
+tens of GigaBytes or even TeraBytes. Crucially we will also see information
+regarding the total memory required by Score-P, which can be set by the
+\texttt{SCOREP\_TOTAL\_MEMORY} environment vaariable.
+
+We can also recover a more detailed
+breakdown by passing the \texttt{-r} flag to \texttt{scorep-score} which then
+shows lots of information including the number of times each function is
+called. This data is very valuable in building our filtering but first we need
+an initial filter file.
+
+We can generate an initial filter file by using the command
+\begin{verbatim}
+score-p -g ./profile.cubex
+\end{verbatim}
+Then we can factor this into our prediction for the trace analysis size with
+the \texttt{-f} flag
+\begin{verbatim}
+scorep-score -f initial-scorep.filter -r ./profile.cubex
+\end{verbatim}
+This should show a reduction in the predicted trace size, although we can
+typically do much better by using the outputs of the \texttt{scorep-score -r}
+command to iteratively add further functions and files to our filtering file.
+
+Performance counter data also adds to the size of any resulting trace analyses,
+and their effects can be included in the size predictions with the \texttt{-c}
+flag followed by the number of performance counters. For example, in this code
+snippet
+\begin{verbatim}
+iscorep-score -f scorep.filter -c 3 -r ./profile.cubex
+\end{verbatim}
+we are predicting to gather 3 performance counters.
+
+Once we decided on a filtering file, it is included with
+\texttt{SCOREP\_FILTERING\_FILE} environment variable. Now that we have built a
+filtering system with Score-P, in the next section we will demonstrate how the
+Score-P infrastructure can be used to build automated trace analyses with
+Scalasca.

workflow/localisation.tex

@@ -31,10 +31,18 @@ \subsection{Tool support}
 
 \subsubsection{Filtering}
 
-To filter data, most toolchains provide marker APIs.
-Alternatively, they come along with precompilers which take your code and add the filter on/off statements to it before it is passed on to the compiler.
-Some very popular tools are sketched in Chapter~\ref{chapter:tools}, including the APIs they offer.
+To filter data, most toolchains provide marker APIs.  Alternatively, they come
+along with precompilers which take your code and add the filter on/off
+statements to it before it is passed on to the compiler.
 
+Filtering becomes a very important topic in trace analyses as initial, and
+somewhat naive, trace analyses can produce incredibly large amounts of data.
+Often tracing all aspects of a code is not necessary, and if an analyst wishes
+to perform a trace analysis of a massively parallel application, then filtering
+is almost a necessity.
+
+Some very popular tools are sketched in Chapter~\ref{chapter:tools}, including
+the APIs they offer.
 
 \subsubsection{Zoom in}
 
@@ -44,6 +52,27 @@ \subsubsection{Zoom in}
  A right click or drawing an observation interval allows you to zoom in.  
 \end{instructions}
 
+\subsubsection{Critical path analysis}
+A critical path defines the longest path through a code's runtime that does not
+include any wait states \cite{bohme2012}. Typically the critical path
+determines the total runtime of the code, and activities on the critical path
+are called critical activities. 
+
+Critical path analysis is a detailed topic which we only touch on briefly here.
+Many profilers can direct users to look at kernels by the proportion of
+runtime, though in many cases this can be misleading for a performance
+assessment [we discuss this further below in
+Section~\ref{section:workflow:localisation:take-away}]. Using critical path
+analysis is another method of directing an analysis as it focuses the analyst
+on routines that are directly corresponding to the total runtime and load
+balance of an application.
+
+Some tools which use critical path analysis include OSACA
+\cite{laukemann2018,laukemann2019} and Scalasca \cite{geimer2010}. For some
+information on getting started with critical path analysis in Scalasca, see
+Sections~\ref{section:localisation:scorep} and
+\ref{section:localisation:scalasca}.
+
  %% this an old VTune box from the 'Preparation' Chapter
 %\begin{instructions}{\vtune}
 %Start a multithreaded run and mark the actual computational phase in the timeline at the bottom.
@@ -121,6 +150,7 @@ \subsection{Related work}
 
 
 \subsection{Take away}
+\label{section:workflow:localisation:take-away}
 
 The most frequent quantity of interest (metric) to developers is typically how much time is spent in routines and which routines or function calls are to be blamed for some behaviour.
 The routines consuming most time are called \emph{hotspots} of a code.