Running a parallel sub-routine within the main routine which is run in series

Hi everybody,

I’m trying to put two different pieces of code into one main routine. One of them (IO_Serial) prepares the I/O in series in XDMF format to be used later on by the 2nd script. The 2nd one (SteadyState_Parallel_Solver) has to be run in parallel using “mpi” otherwise I will run out of memory. Each one of them is working perfectly fine when run separately, but what I want to do now is to make both scripts as sub-routines and call them sequentially from the main routine which is supposed to run in series also.

My question is that how can I run only part of the program in parallel while the main routine is run in series?
I tried something with “Pool.map” from Multiprocessing package but it doesn’t work as the mpirun used to work. Attached below is just a minimal version of the main routine.

from fenics import *
import scipy.io as sio
import numpy as np
import h5py as h5p
import os
from multiprocessing import Pool

#%% Main Routine

User Inputs

TissueFileName = ‘TissueThermalProperties_duke_tongue.mat’
MeshFileName = ‘GeoGmsh_duke_tongue.msh’
PLDFileName = ‘PLD_On_FEM_Nodes_duke_tongue_500MHz.mat’
OUTDIR = (“DukeVisulization”)
Bolus_Temp = Constant(20)
PwrNormFactor = 15

Global Constants used in all sub-routines

Tb = 37 #Arterial blood temperature
Body_Temp = Constant(37)

if not os.path.isdir(‘./SIO’):
IO_Serial(TissueFileName, MeshFileName, PLDFileName, PwrNormFactor, OUTDIR)

THOutputName = OUTDIR + ‘/Temperature_SAROptimized.xdmf’
WritingFlag = True
with Pool(processes=4) as pool:
Temperature = pool.map(SteadyState_Parallel_Solver, THOutputName)

PS: The sub-routines are already good.

Many thanks in advance for your help and suggestions.

You can use the Python package mpi4py to do this.
Say you want to run a script test.py in parallel using 4 mpi ranks by using:
mpirun -n 4 python3 test.py
Then, whenever you want to run a part of the code in serial (on one rank), you can do the following:

from mpi4py import MPI

comm = MPI.COMM_WORLD
if comm.rank == 0:
    function_call_to_serial_code(....)

# run rest in parallel
# ...

Thanks a lot dokken. What you suggested is great, but can’t we do just the opposite?
I mean can’t I run the test.py with the following command:

python3 test.py

then inside it, run only one sub-routine in parallel with four cores? I know this is a rookie question and more related to python than to Fenics, but it will be great if can give me the if condition for a parallel run of only one sub-routine inside the main program because the rest of the program will be run in series and sounds more convenient this way.

Thanks again.

Dolfin relies on MPI, and this is simply not how MPI works.

They key with parallel runs with MPI is that you can vary the number of processes to use, and that is an input argument (supplied through MPI-run). Internally in dolfin the mesh, functionspace and corresponding data is then partitioned (distributed onto different processes, that has to communicate data with each other).

If you want to run several runs of the same code, but with different parameters for each run, you can use: concurrent.futures — Launching parallel tasks — Python 3.12.0 documentation

Thanks for your response and guidance. It’s highly appreciated.

I had a cursory look at the documentation you mentioned, and I just have another quick question in this regard. Actually, what I’m trying to do now is to use my thermal solver (based on Fenices parallel implementation) inside an optimization routine. So there is a cost function to be minimized inside a while loop and in each iteration, I need to call upon the parallel solver to do the function evaluation and then parameters will be updated based on PSO optimization for the next round. Following is a pseudo-code of what I’m trying to explain:

Initialization phase

Starting point and first cost function evaluation

Call upon IO_serial #this is to be run in series
Feed the inputs to the thermal solver and get the temperature distribution back
Based on the distribution, assess the cost function #in series

Minimization of the cost function

While (CostFunction > Threshold )

Update the inputs and call IO_Serial sub-routine #in series
Feed the inputs to the thermal solver and get the new temperature distribution back
Based on the distribution, assess the cost function #in series

Now if you were me, wanting to implement such a code how would you proceed? Do you write them all in parallel and call IO serial part only in series? Or how should someone integrate the thermal solver in a dynamic optimization process in which we don’t know how many iterations or parameters are involved beforehand?

Thanks again!