2.3. Environment variables
pyoomph reads a number of environment variables at runtime (and a few more at build time, see Section 2.2.3). None of these are required – pyoomph works fine without any of them set – but they can be useful to tweak caching, threading, debugging or solver library locations.
2.3.1. Threading
PYOOMPH_OPENBLAS_NUM_THREADS,PYOOMPH_MKL_NUM_THREADSNumber of threads used by OpenBLAS and MKL, respectively. Both default to the
Nof a--omp Ngiven on the command line, and to4without it. On import, pyoomph sets the correspondingOPENBLAS_NUM_THREADS/MKL_NUM_THREADSenvironment variables to these values, unless they are already set in the environment (e.g. by your shell or by a launcher script), in which case the existing value is left untouched. They exist so that a third-party library can be pinned for pyoomph runs alone, without also pinning it for everything else on the machine.They bound the threading inside BLAS/MKL only. They do not make a pyoomph run serial: the element loop takes its thread count from
--omp/set_num_threads()and passes it to its threading backend itself, so with--omp 4the assembly runs on four threads whatever these variables say. Run without--omp(or with--omp 1) for a serial run.PYOOMPH_OMP_NUM_THREADSValue pyoomph sets
OMP_NUM_THREADSto on import, again only if that variable is not already set. It defaults to the--omp Ngiven on the command line, or to1without it – pyoomph’s own threaded element loop takes its thread count from--omp/set_num_threads()and not from the environment, so the pin only keeps a third-party OpenMP runtime in the same process from opening a second pool of threads. See Section 14.1.PYOOMPH_ASSEMBLY_CHUNK_DOUBLESSize, in doubles, of the scratch buffer the threaded element assembly (
--omp N, see Section 14.1) fills before gathering it into the global matrices. Defaults to131072, i.e. 1 MB per matrix, and is divided by the number of matrices being assembled. Chunking is what keeps the threaded summation order identical to the serial one, so this only trades cache behaviour against barrier overhead and never changes the result. The default was measured, and everything from about 16 K upwards lies within a few percent of it – much smaller chunks pay their two barriers for too little work.
2.3.2. JIT code cache
pyoomph compiles the equations of a problem into native code just-in-time. To avoid recompiling identical code again and again, compiled objects are cached on disk, keyed by the content of the generated code. The following variables control this cache:
PYOOMPH_JIT_CACHESet to
0,false,Falseor an empty string to disable the JIT cache for the current process. Defaults to enabled (1). Note that the cache can also be permanently disabled at build time (seePYOOMPH_ENABLE_JIT_CACHEin Section 2.2.3); a build-timeOFFcannot be re-enabled via this variable.PYOOMPH_JIT_CACHE_DIRDirectory used to store cached compiled objects. If unset, a platform-specific default cache directory is used (respecting
XDG_CACHE_HOMEon Linux,LOCALAPPDATAon Windows, or~/Library/Cacheson Mac).PYOOMPH_JIT_CACHE_MAX_MBMaximum size of the JIT cache in megabytes. Defaults to
2048.PYOOMPH_JIT_CACHE_MAX_FINGERPRINTSMaximum number of entries kept in the cache’s Tier-2 fingerprint bookkeeping. Defaults to
100000.PYOOMPH_JIT_CACHE_TIER2Set to
0,false,Falseor an empty string to disable the (more experimental) Tier-2 caching. Defaults to enabled (1), but only has any effect whilePYOOMPH_JIT_CACHEis also enabled.
2.3.3. MPI
These are only read when running under mpirun (cf. Section 14.2).
PYOOMPH_MPI_OUTPUTConsole output mode when several ranks share one terminal:
condensed(the default – only rank 0 prints, while the stderr of every rank still gets through, tagged),all(every rank, each line written in one piece and tagged[rank N]) oroff(no filtering at all, i.e. the raw interleaved output). The same choice is available as--mpi-outputon the command line, which takes precedence; an unrecognized value here is ignored rather than raising, because this is read whilepyoomphis being imported. See Section 14.2.3.PYOOMPH_ALLOW_SERIAL_UNDER_MPIRUNSet to
1to allow a pyoomph built without MPI support to be started by an MPI launcher with more than one rank, which is refused by default (cf. Section 14.2). The ranks are then independent serial runs which know nothing of each other, so each of them must be given its own output directory. This has no effect on a build with MPI support.PYOOMPH_MPI_IDLE_SPINHow many polls a rank waiting for another one does before it starts sleeping between polls. Defaults to
2000. The spin phase is there becausetime.sleep()cannot resolve much better than about 60 microseconds, which would otherwise dominate the many short collectives of a small problem.PYOOMPH_MPI_IDLE_MAX_SLEEPLongest sleep, in seconds, between two polls once a rank has given up spinning. Defaults to
5e-3, which leaves a waiting rank at roughly 200 wake-ups per second, i.e. no measurable CPU load, while bounding the delay with which it notices that a long factorization on another rank has finished.PYOOMPH_MPI_IDLE_WARN_AFTERSeconds a rank may wait before pyoomph prints a one-off notice naming what it is waiting for. Defaults to
600. It is deliberately a notice and not a timeout: raising on the one rank that noticed would create exactly the deadlock it is trying to report.
2.3.4. Compilation and debugging
PYOOMPH_DEBUGSet to
1to compile the just-in-time generated code with debug information/flags instead of the default optimized build. Useful when debugging a crash inside generated code with e.g.gdb.PYOOMPH_FULL_UNIT_ERRORSet to
1to print the offending expression in full when a residual contribution does not come out dimensionless. By default such an expression is truncated, since a single Navier-Stokes contribution prints as tens of thousands of characters and buries the part of the message that says what is actually wrong.
2.3.5. Plotting
PYOOMPH_MPLBACKENDOverrides the matplotlib backend used by pyoomph’s plotting utilities (e.g.
Agg,TkAgg,Qt5Agg). If unset,Aggis used, since these utilities render to files rather than to a window. This does not affect the bifurcation GUI, which embeds its figure in its own tkinter window and therefore selects no backend at all.
2.3.6. Alternative solver libraries
By default, pyoomph loads the MKL/Pardiso solver library from its usual system location. The following variable lets you point at a specific shared library instead, e.g. a custom build or a non-standard install location:
PYOOMPH_PARDISO_LIBFull path to the shared library providing the (MKL) Pardiso solver.
2.3.7. Third-party variables
A few environment variables belonging to third-party dependencies are also relevant when working with pyoomph, most notably PETSC_DIR, PETSC_ARCH and PYTHONPATH for selecting a PETSc/SLEPc installation, see Section 2.6 for details.