A.4. Alternative method: wrapper scripts

A.4. Alternative method: wrapper scripts
	Appendix A. Sampling MPI Applications

If you require more complicated substitutions than the substitution mechanisms provide, or if different sampling options should apply depending on the rank, you have the option of providing a wrapper script that implements the required logic. The script can reference environment variables and call other utilities that exist on the host.

You must then invoke mpirun (mpiexec, srun, aprun etc.) on this wrapper script instead of the sample binary.

This technique relies on a few things to be successful:

The cluster nodes must have a suitable script interpreter (e.g., /bin/sh).
The binaries of Freja need to be made available in each cluster node, either by referring to them with absolute paths or proper relative paths or by setting the PATH environment variable properly.
The path to the target application needs to be properly set to locate it in the cluster node (with absolute path name or a proper relative path name).
On some systems (e.g., Cray), the default application staging mechanism has to be disabled. It is not able to identify all the binaries that need to be staged. All applications (Freja, user applications) need to be visible from each cluster node.

In these examples it is assumed the fingerprint files are created in the working directory where mpirun starts the script. The exact capabilities and mechanisms vary with different MPI systems. On some systems you have to write your script to explicitly set the appropriate working directory or use explicit paths.

Example A.1. Sampling Open MPI ranks using a wrapper script

If you are using Open MPI and want to run the MPI application mpi-test, your wrapper script could look like the following example:

sample_mpi_test.sh:

    #!/bin/sh
    sample -o rank$OMPI_COMM_WORLD_RANK.smp -r mpi-test "$@"

Invoke this by:

$ mpirun -np ... sample_mpi_test.sh arg1 arg2

The script samples each rank of the MPI application mpi-test and creates a sample file named rank0.smp for rank 0, rank1.smp for rank 1, and so on

Example A.2. Sampling with a wrapper script

If your MPI implementation does not provide an environment variable that identifies the MPI rank, you can replace the rank in the file name with another unique identifier, for example, the host name and the PID of the script. The following script is an example:

sample_mpi_test2.sh:

    #!/bin/sh
    sample -o rank-$HOSTNAME-$$.smp -r mpi-test "$@"

Invoke this by:

$ mpirun -np 16 sample_mpi_test2.sh arg1 arg2

The script samples each rank of the MPI application mpi-test and creates a sample file named rank-node42-4711.smp for rank 0, rank-node42-4718.smp for rank 1, and so on.

Example A.3. Selectively sampling ranks

If the rank is available through an environment variable, you can also selectively sample ranks. For example, the following script would sample rank 0 and run the other ranks without sampling with Open MPI:

    #!/bin/sh
    if [ $OMPI_COMM_WORLD_RANK == 0 ]; then
      sample -o rank$OMPI_COMM_WORLD_RANK.smp -r mpi-test "$@"
    else
      mpi-test "$@"
    fi