Add file-based probes, horses2probes, and GPU compatibility fixes

[mchavezmodena]

Probes from a file

This PR introduces a new file-based probe monitor that reads probe coordinates from a plain-text file and samples one or more flow variables at each location throughout the simulation.

The monitor extends the existing monitoring framework by allowing users to define an arbitrary number of probes through an external file rather than listing them individually in the input file. Multiple flow variables (e.g. rho, u, v, w, p, ...) can be sampled simultaneously, with the variables selected directly from the monitor definition.

A new input block,

#define probe file
...
#end

has been added to the probes input file to define the probe set. During startup, the solver prints a summary of the loaded probes and the variables being monitored.

To reduce unnecessary I/O, a new probe save timestep option allows probe data to be written only after a user-defined physical-time interval has elapsed. This filtering is performed before entering the O(N) interpolation loop, avoiding unnecessary work when output is skipped.

The monitor supports two output formats. The default ASCII backend generates one .probe file per probe inside a probes/ directory and parallelizes file writing using OpenMP. Alternatively, when compiled with HAS_HDF5 and using output format = HDF5, all probes are stored in a single <solution>.probes.h5 file using extendible datasets containing probe coordinates, time, iteration number, and one dataset for each sampled variable. The HDF5 backend is recommended for large probe sets.

The implementation has also been optimized to support very large numbers of probes. The previous buffering strategy,

values(nVars, BUFFER_SIZE)

could require approximately 8 GB per MPI rank for one million probes. This has been replaced with a size-one write buffer that is flushed every timestep, reducing the memory footprint to roughly 8 MB per MPI rank, independently of the number of probes.

Communication during probe interpolation has also been optimized by replacing N individual MPI_Bcast operations with a single MPI_Allreduce(SUM).

Several robustness issues have also been addressed, including a segmentation fault with inactive probes, an OpenACC bypass for the file-probe interpolation loop, an unterminated #ifdef FLOW, and a bug where readValueInRegion could corrupt the probe file path.

Finally, the new monitor has been integrated into the time-integration loop so that it follows the same execution model as the existing monitoring infrastructure.

Main modified files:

• Solver/src/libs/monitors/Probe.f90
• Solver/src/libs/monitors/Monitors.f90
• Solver/src/libs/timeintegrator/TimeIntegrator.f90

---

horses2probes

This PR also introduces horses2probes, a standalone post-processing utility that converts the output generated by the new probe monitor into Tecplot-compatible files.

The tool supports both the ASCII and HDF5 output formats produced by the solver, transparently handles multiple sampled variables, and is designed to efficiently process very large probe datasets. Since it is built independently from the solver, it can be compiled and executed on any machine containing the simulation outputs without requiring the full HORSES3D build.

The implementation is located under

Solver/src/addons/horses2probes/

and includes all source files together with an independent Makefile.

---

GPU fixes (nvfortran -acc -fopenmp)

The probe monitor relies on OpenMP for parallel ASCII output and MPI reductions. Enabling -fopenmp together with -acc exposed several compilation and runtime issues in nvfortran. Although most of them were unrelated to the new probe monitor itself, they prevented successful GPU builds and therefore have been fixed as part of this PR.

Class A — NVFORTRAN-S-0155 ("DO loop expected after OMP DO")

Current versions of nvfortran reject loops annotated simultaneously with !$omp do and !$acc parallel loop, even when the directives target different execution models. The solution was to guard the directives using compile-time preprocessor conditions so that only one of them is visible to the compiler depending on whether _OPENACC is enabled:

#ifdef _OPENACC
!$acc parallel loop
...
#else
!$omp do
...
#endif

This pattern has been applied to:

• Solver/src/NavierStokesSolver/SpatialDiscretization.f90
• Solver/src/libs/mesh/HexMesh.f90
• Solver/src/libs/mesh/IBMClass.f90
• Solver/src/libs/discretization/EllipticBR1.f90
• Solver/src/libs/discretization/EllipticBR2.f90
• Solver/src/libs/discretization/EllipticIP.f90
• Solver/src/libs/discretization/DGSEMClass.f90
• Solver/src/libs/monitors/VolumeIntegrals.f90
• Solver/src/libs/monitors/SurfaceIntegrals.f90

Class B — NVFORTRAN-F-0000 internal compiler error (mk_mem_ptr_shape)

Another compiler bug was triggered by pointer association inside OpenACC kernels. Code of the form

type(T), pointer :: p
p => array(i)

caused an internal compiler error (mk_mem_ptr_shape) when compiled with -acc.

The affected routine was rewritten using an associate construct instead of pointer association, which avoids the compiler bug while preserving the original behaviour.

Modified file:

• Solver/src/libs/discretization/SCsensorClass.f90

Class C — LLVM IR undefined-value internal compiler error

A separate nvfortran internal compiler error occurred when polymorphic arguments (class(X)) were passed inside OpenMP parallel regions. The LLVM backend failed during code generation with an undefined-value error.

The affected parallel region was restructured so that the polymorphic object is no longer passed into the OpenMP region, eliminating the compiler crash without changing the numerical behaviour.

Modified file:

• Solver/src/libs/timeintegrator/AnalyticalJacobian.f90

GPU correctness fix

While introducing the _OPENACC guards described above, an additional runtime correctness issue was identified.

Although the inner loops had been correctly separated into OpenMP and OpenACC versions, the outer

!$omp parallel

region in ComputeTimeDerivative remained active during GPU execution. Consequently, every OpenMP thread independently launched the same sequence of GPU kernels. Kernels containing operations such as

QDot = QDot + ...

were therefore executed once per OpenMP thread, causing the residual to be accumulated multiple times. In practice, GPU runs diverged immediately, producing residuals on the order of 10^62 at iteration zero.

The solution was to wrap the outer OpenMP region with

#ifndef _OPENACC
!$omp parallel
...
!$omp end parallel
#endif

so that, when compiling for GPU execution, a single CPU thread dispatches all GPU kernels while preserving the original OpenMP behaviour for CPU builds.

Modified file:

• Solver/src/NavierStokesSolver/SpatialDiscretization.f90

---

Documentation

The documentation has been updated to describe all the new functionality introduced in this PR.

The monitor documentation now includes the new #define probe file syntax, the available keywords, the probe input format, the ASCII and HDF5 output formats, and the HDF5 dataset layout.

The PDF user manual has been updated with a new section describing the file-based probe monitor and the horses2probes utility. The main manual has also been renamed from UserManual.tex to HORSES3D_UserManual.tex.

Finally, the post-processing documentation has been extended to include the complete workflow for processing probe outputs using horses2probes.

Updated files:

• doc/pages/user_manual/monitors.md
• doc/HORSES3D_UserManual.tex
• doc/basicManualAndTutorial.tex
• doc/pages/user_manual/s-postprocessing.md

[codecov]

Codecov Report

❌ Patch coverage is 7.01754% with 318 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
Solver/src/libs/monitors/Monitors.f90	12.12%	152 Missing and 22 partials ⚠️
Solver/src/libs/monitors/Probe.f90	0.00%	142 Missing ⚠️
Solver/src/libs/mesh/StorageClass.f90	0.00%	1 Missing ⚠️
Solver/src/libs/sources/ActuatorLine.f90	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!