# How to parallelize code alongside assign()?

**URL:** <https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326>\
**Category:** Errors\
**Created:** [February 2, 2023, 9:39pm UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326 "2023-02-02T21:39:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Biophysman](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Biophysman](https://fenicsproject.discourse.group/u/Biophysman)\
**Post date:** [February 2, 2023, 9:39pm UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326/1 "2023-02-02T21:39:26Z")

</div>

I’m trying to manipulate functions using `assign()` inside of a for-loop, but I want my code to run in parallel, especially when using `solve()` elsewhere inside of a time loop.

As a simple example, consider the following code code as run using `mpirun -np 2`

```auto
# Define submesh mesh and function spaces
mesh = BoxMesh(Point(0, 0), Point(box_length, box_length, 0.5),
                     64, 64, 32)

P1 = FiniteElement('CG', mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, P1)

#Initialize functions
Aexp = Expression('2', degree=1)
A = interpolate(Aexp, W)

Bexp = Expression('2', degree=1)
B = interpolate(Bexp, W)

Cexp = Expression('0', degree=1)
C = interpolate(C, W)

#time loop
for t in range(0,5):
  C.assign(A + B)
  B.assign(C)  
  print(C.vector().get_local())

```

This returns the following error message

```auto
-------------------------------------------------------------------------
*** Error: Unable to successfully call PETSc function 'VecCopy'.
*** Reason: PETSc error code is: 73 (Object is in wrong state).
*** Where: This error was encountered inside /home/conda/feedstock_root/build_artifacts/fenics-pkgs_1668433332839/work/dolfin/dolfin/la/PETScVector.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.1.0
*** Git changeset: 3ea2183fbfe7277de9f16cbe1a9ffaab133ba1fa
*** -------------------------------------------------------------------------

```

This question seems similar to [this question](https://fenicsproject.discourse.group/t/how-to-use-interpolate-and-assign-functions-in-parallel/5693), but that question was never answered and my example is both slightly different in form and returns a different error.

Any help at all would be appreciated, even just an explanation of what is going on with the dofs in parallel, or pointing to the relevant documentation. Clearly, I know that assign itself does not probably need parallelization. Parallelizing other code that might be in the for-loop alongside `assign()` is my real goal. For instance, when `assign()` is used in a time loop to update a source field for some variational form to be solved.

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [February 3, 2023, 6:25am UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326/2 "2023-02-03T06:25:03Z")

</div>

> [@Biophysman](#):
>
> `C = interpolate(C, W)`

Surely this is a typo?

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [February 3, 2023, 7:13am UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326/3 "2023-02-03T07:13:31Z")

</div>

Even with the fix I mentioned above, I cannot reproduce your issue using the lastest version of DOLFIN (on docker), or the 2019.1.0 dolfin quay image:

```python
from dolfin import *
box_length = 1
# Define submesh mesh and function spaces
mesh = BoxMesh(Point(0, 0), Point(box_length, box_length, 0.5),
                     10,10,10)

P1 = FiniteElement('CG', mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, P1)

#Initialize functions
Aexp = Expression('2', degree=1)
A = interpolate(Aexp, W)

Bexp = Expression('2', degree=1)
B = interpolate(Bexp, W)

Cexp = Expression('0', degree=1)
C = interpolate(Cexp, W)

#time loop
for t in range(0,5):
  C.assign(A + B)
  B.assign(C)  
  print(C.vector().get_local())

```

---

<div class="post-metadata">

**Author:** ![Biophysman](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Biophysman](https://fenicsproject.discourse.group/u/Biophysman)\
**Post date:** [February 3, 2023, 3:31pm UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326/4 "2023-02-03T15:31:13Z")

</div>

Thanks for your help @dokken! You are correct about the typo.

I’m not sure why you cannot reproduce the error. I’m running the code from the [conda-forge version](https://anaconda.org/conda-forge/fenics) running on a cluster. Could clashes between the conda installation and the cluster’s handling of multithreading or something be going on?

I was able to “fix” things by doing a hacky workaround. From [this question](https://fenicsproject.discourse.group/t/problem-with-assign-in-parallel/989) I learned that `c.assign(a)` has different behavior than `assign(c, a)` when running under `mpirun`. Thus, the following code has the desired behavior for me:

```auto
# Define submesh mesh and function spaces
mesh = BoxMesh(Point(0, 0), Point(1, 1, 0.5),
                     64, 64, 32)

P1 = FiniteElement('CG', mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, P1)

#Define pointwise addition of functions. Returns a numpy array of the pointwise sym of .get_local() values
def Add(functions):
    garrays = []
    for g in functions:
        garrays.append(g.vector().get_local())
    Sum = garrays[0] #initialize the sum variable with the first array in garrays
    
    #successively add each array in garrays together (-1 in loop +1 in index because sum begins with garrays[0])
    for i in range(len(garrays)-1): 
        Sum = Sum+garrays[i + 1]
    result = Function(W)
    result.vector().set_local(Sum)
    return result

Aexp = Expression('2', degree=1)
A = interpolate(Aexp, W)

Bexp = Expression('2', degree=1)
B = interpolate(Bexp, W)

Cexp = Expression('0', degree=1)
C = interpolate(Cexp, W)

for x in range(0,5):
  assign(C, Add([A,B]))
  assign(B, C)  
  print(C.vector().get_local())

```

While this workaround might “work” for now, I certainly don’t understand what’s going on? Bafflingly, this is the output from the print function using `-np 2`:

```auto
[0. 4. 0. ... 4. 4. 0.]
[0. 6. 0. ... 6. 6. 0.]
[0. 8. 0. ... 8. 8. 0.]
[0. 10. 0. ... 10. 10. 0.]
[0. 12. 0. ... 12. 12. 0.]
[4. 4. 4. ... 4. 4. 4.]
[6. 6. 6. ... 6. 6. 6.]
[8. 8. 8. ... 8. 8. 8.]
[10. 10. 10. ... 10. 10. 10.]
[12. 12. 12. ... 12. 12. 12.]

```

One of the processes seems to get it right, while the other doesn’t?

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [February 3, 2023, 3:35pm UTC](https://fenicsproject.discourse.group/t/how-to-parallelize-code-alongside-assign/10326/5 "2023-02-03T15:35:05Z")

</div>

You error is in PETSc, which might mean there is an issue with the PETSc installation shipped with conda.