2.4. Optional installation of PETSc/SLEPc

If you want to solve for eigenvalue problems, pyoomph by default will invoke scipy’s eigensolver based on ARPACK. However, for unsymmetric matrices which usually arise in complicated problems, SLEPc provides a much more stable alternative, since it also supports nonsymmetric mass matrices. So whenever you want to investigate linear stability, you should consider performing the following steps. In any case, it is advised to occasionally check your eigenvalues by adding report_accuracy=True to calls of solve_eigenproblem().

Also on Mac arm64, we recommend using PETSc with MUMPS as linear solver backend for the linear solves during Newton’s method, since the default MKL Pardiso backend is not yet fully supported on arm64.

We unfortunately do not really know how to install PETSc/SLEPc natively on Windows, so you have to find your own way of installing it (and let us know the steps. A good start can be found here). If you are on Windows, the easiest way to get PETSc/SLEPc working is to install pyoomph inside the Windows Subsystem for Linux (WSL) and follow the normal Linux instructions below – see Section 2.5 for step-by-step instructions.

SLEPc depends on PETSc, so it is advisable to install both of these packages together. Also, since we often will obtain matrices with a zero on a diagonal (mainly due to Lagrange multipliers, incompressibility constraints, etc.) we need a suitable linear solver backend in PETSc which can perform pivoting. We usually use MUMPS for that.

PETSc/SLEPc are compiled for exactly one scalar type – either real or complex – and pyoomph picks up whichever petsc4py/slepc4py happens to be first on PYTHONPATH when the process starts. Which one you need depends on what you do with it:

The real build is used for the linear solves during Newton’s method (with PETSc/MUMPS as linear solver backend) and for conventional, non-normal-mode eigenvalue problems (with SLEPc).
The complex build is mandatory if you want to solve normal mode eigenvalue problems (cf. Section 10.3 and Section 10.4), since these introduce an explicit imaginary prefactor into the equations. If you try to solve such a problem with only a real build available, pyoomph will raise a RuntimeError telling you to switch to a complex installation.

Since most users will want both at some point, we recommend building both a real and a complex arch of PETSc/SLEPc side by side in the same source tree (they can share one PETSC_DIR, just use two different PETSC_ARCH names), and then select the one you currently need via PETSC_ARCH/PYTHONPATH before running a particular script. If you only ever need conventional (real) eigenvalue problems, you can of course skip the complex build and just do the real one.

All packages can be downloaded and installed together. If you have opted for the Rosetta 2 installation route on a Mac arm64 (see previous pages) to also support MKL Pardiso, the following steps must be again done in a Rosetta 2 terminal.

We start by downloading PETSc in a folder of our choice (replace A_FOLDER_OF_YOUR_CHOICE in the following accordingly). If you have installed pyoomph in a python environment, it is advisable to also activate this environment now.

cd A_FOLDER_OF_YOUR_CHOICE
git clone -b release https://gitlab.com/petsc/petsc.git petsc
cd petsc

We now have to export some environment variables. PETSC_DIR is shared between both arches, while PETSC_ARCH selects which one we are currently building:

export PETSC_DIR=$(pwd)
export PETSC_ARCH=pyoomph_petsc_arch_real

Note that the choice of the name pyoomph_petsc_arch_real can be changed arbitrarily.

We then have to make sure that we have flex and Bison. On Ubuntu (and other Linux types analogously), you can install it system-wide via

sudo apt install flex bison

On Mac, you can install it via Homebrew:

brew install flex bison

Alternatively, you can let PETSc download it as well by adding --download-bison at the end of the following configuration command. Note that we download and install further solver packages here, which are usually not needed, but likely will be used in future.

./configure --with-mpi  --with-petsc4py --with-slepc4py --download-mumps=yes --download-hypre=yes --download-parmetis=yes --download-ptscotch=yes --download-slepc=yes --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-metis=yes --download-scalapack --with-scalar-type=real

You can also add optimization or OpenMP support, e.g. --with-debugging=0, -with-openmp, --with-openmp-kernels. For all details, please call ./configure --help.

Note

If you should have issues with cmake on Ubuntu (and potentially other distros), try

Install cmake (updated version, see https://askubuntu.com/questions/355565/how-do-i-install-the-latest-version-of-cmake-from-the-command-line)
add flag --download-fblasapack=1 when configuring

At the end of the configuration process, a make command will be written, which you have to execute as a next step.

Afterwards, PETSc/SLEPc is installed to the folder A_FOLDER_OF_YOUR_CHOICE/petsc/pyoomph_petsc_arch_real. At the end, it will also show a test command, by which you can test the basic functionality of your installation.

If you also need normal mode eigenvalue problems, repeat the same steps for a second, complex arch. Since PETSC_DIR is already set from above, only PETSC_ARCH and the --with-scalar-type flag change:

export PETSC_ARCH=pyoomph_petsc_arch_complex
./configure --with-mpi  --with-petsc4py --with-slepc4py --download-mumps=yes --download-parmetis=yes --download-ptscotch=yes --download-slepc=yes --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-metis=yes --download-scalapack --with-scalar-type=complex

Again, execute the make command that configure prints at the end. This installs the complex build alongside the real one, to A_FOLDER_OF_YOUR_CHOICE/petsc/pyoomph_petsc_arch_complex.

Note that --download-hypre is left out here: HYPRE (and thus its BoomerAMG preconditioner) only supports real-valued matrices, and PETSc’s configure will fail if you request it together with --with-scalar-type=complex. This is not a problem in practice, since MUMPS is used as the actual factorization backend anyway (via use_mumps()) – HYPRE in the real build above is only downloaded for potential future use as an alternative preconditioner.

To use one of the two builds within pyoomph, PETSC_DIR, PETSC_ARCH and PYTHONPATH must be set before you start your Python driver script, e.g. for the real build:

export PETSC_DIR=A_FOLDER_OF_YOUR_CHOICE/petsc
export PETSC_ARCH=pyoomph_petsc_arch_real
export PYTHONPATH=$PYTHONPATH:$PETSC_DIR/$PETSC_ARCH/lib

or, analogously, with pyoomph_petsc_arch_complex for the complex build. Since only one of the two can be active in a given terminal/process, do not put both blocks unconditionally into your .bashrc/.zshrc – the second one would just shadow the first on PYTHONPATH. Instead, either export the variables for the arch you currently need right before running your script, or wrap each block in a shell function (e.g. pyoomph_petsc_real() / pyoomph_petsc_complex()) that you call as needed. If you only installed one arch, it is fine to export it unconditionally in your shell startup file as before.

To use SLEPc with MUMPS as eigensolver, either set it in python during your driver code, e.g.

problem.set_eigensolver("slepc").use_mumps()

or supply the flag --slepc_mumps when calling your driver code:

python my_eigenvalue_simulation.py --slepc_mumps