Different total number of nodes in serial and parallel runs

1

Dear FEniCSx community, I find sometimes the total number of nodes are different in serial and parallel runs when I am using an unstructured mesh.

The MWE is follows.

import dolfinx.io
from mpi4py import MPI
from dolfinx.fem import VectorFunctionSpace, FunctionSpace, Function
import numpy as np


with dolfinx.io.XDMFFile(MPI.COMM_WORLD, "Mesh.xdmf", "r") as xdmf:
    mesh = xdmf.read_mesh(name="Grid")
with dolfinx.io.XDMFFile(MPI.COMM_SELF, "Mesh.xdmf", "r") as xdmf:
    mesh_serial = xdmf.read_mesh(name="Grid")
V = VectorFunctionSpace(mesh, ("CG", 1))
u = Function(V)

imap = u.function_space.dofmap.index_map  # Index map
num_global_nodes = imap.size_global  # Number of glocal nodes

if MPI.COMM_WORLD.rank == 0:
    V0 = FunctionSpace(mesh_serial, ("CG", 1))
    x_global = V0.tabulate_dof_coordinates()  # Global nodal coordinates 
    print(num_global_nodes, x_global.shape[0])

The code for generating the xdmf file Mesh.xdmf is shown on the next floor separately since it might quite long. When I run the above code in serial, the result is correctly given as

296 296

However, when I run the above code in parallel, num_global_nodes and x_global.shape[0] are unexpectedly different.
Two processes:

298 296

Three processes:

297 296

Four processes

300 296

For some other meshes, I can obtain the same values for num_global_nodes and x_global.shape[0]. I would like to solve this problem since I need to collect all the displacement field from different processes into one serial mesh based on Gather solutions in parallel in FEniCSX - #2 by dokken. So could anyone please tell me how fix this issue?

2

This code is used to generate the above-mentioned mesh.

import numpy as np
import meshio

Nodes = np.array([
    [-2.828427, -2.828427],
    [-2.863161, -0.900200],
    [4.000000, 0.000000],
    [-1.894757, 1.894757],
    [3.272348, 0.346400],
    [-2.661588, 1.941005],
    [-2.478525, -0.499539],
    [-0.000000, -4.000000],
    [-0.853961, 3.450008],
    [-1.168141, 1.168141],
    [-2.459797, 1.064134],
    [1.812381, 1.177565],
    [-2.077605, 0.861631],
    [-0.800000, 0.000000],
    [-0.000000, -1.598080],
    [-2.279022, -0.448518],
    [2.030746, -0.172630],
    [0.499539, -2.478525],
    [-0.499539, -2.478525],
    [1.485258, -0.856149],
    [2.459797, -1.064134],
    [0.800000, 0.000000],
    [-2.030746, 0.172630],
    [2.077605, -0.861631],
    [0.853961, -3.450008],
    [0.435542, -3.581350],
    [1.598080, 0.000000],
    [2.661588, -1.941005],
    [1.894757, -1.894757],
    [0.448518, 2.279022],
    [2.863161, 0.900200],
    [2.828427, 2.828427],
    [-0.000000, -2.000000],
    [-1.064134, -2.459797],
    [-0.496615, -3.969052],
    [2.926971, 1.552104],
    [3.500968, 1.934741],
    [3.581350, 0.435542],
    [-3.206185, -2.391732],
    [-0.299548, -0.741802],
    [1.934741, -3.500968],
    [-0.657399, 3.126895],
    [-2.000000, 0.000000],
    [-0.467987, 2.676036],
    [-0.565685, -0.565685],
    [3.500968, -1.934741],
    [-3.206185, 2.391732],
    [0.861631, 2.077605],
    [2.676036, 0.467987],
    [-1.235922, -3.216867],
    [2.279022, -0.448518],
    [3.206185, -2.391732],
    [3.969052, -0.496615],
    [0.900200, 2.863161],
    [-0.850907, 0.850907],
    [3.206185, 2.391732],
    [2.121559, -0.326775],
    [1.787357, 0.614252],
    [-0.329733, -1.608150],
    [-2.828427, 2.828427],
    [-3.969052, 0.496615],
    [-2.121559, 0.326775],
    [-0.900200, -2.863161],
    [-0.204648, -2.956201],
    [0.850907, -0.850907],
    [3.126895, 0.657399],
    [2.077605, 0.861631],
    [-1.552104, -2.926971],
    [-3.126895, 0.657399],
    [3.216867, 1.235922],
    [0.657399, -3.126895],
    [1.064134, 2.459797],
    [-2.840044, 0.366565],
    [1.189423, 0.000000],
    [-1.476494, 2.173634],
    [-2.926971, -1.552104],
    [-1.608150, -0.329733],
    [-2.030746, -0.172630],
    [1.894757, 1.894757],
    [0.204648, -2.956201],
    [3.712108, -1.490051],
    [0.000000, 2.000000],
    [0.614252, 1.787357],
    [-0.448518, -2.279022],
    [-1.812381, -1.177565],
    [-2.173634, 1.476494],
    [0.741802, -0.299548],
    [0.329733, -1.608150],
    [-0.992870, -3.874817],
    [-1.177565, -1.812381],
    [-3.712108, -1.490051],
    [-1.598080, 0.000000],
    [-2.956201, 0.204648],
    [1.490051, 3.712108],
    [-1.173152, 0.464637],
    [-0.467987, -2.676036],
    [-0.172630, -2.030746],
    [2.676036, -0.467987],
    [3.272348, -0.346400],
    [2.391732, -3.206185],
    [-3.272348, 0.346400],
    [-1.485258, 0.856149],
    [1.168141, -1.168141],
    [0.499539, 2.478525],
    [-0.499539, 2.478525],
    [0.856149, 1.485258],
    [-3.874817, 0.992870],
    [1.168141, 1.168141],
    [2.478525, 0.499539],
    [-1.787357, 0.614252],
    [1.476494, 2.173634],
    [-0.299548, 0.741802],
    [-3.272348, -0.346400],
    [1.064134, -2.459797],
    [2.341494, 2.341493],
    [-2.863161, 0.900200],
    [1.934741, 3.500968],
    [-4.000000, 0.000000],
    [-1.189423, 0.000000],
    [-0.366565, 2.840044],
    [-0.172630, 2.030746],
    [-0.366565, -2.840044],
    [2.341493, -2.341494],
    [-0.326775, 2.121559],
    [-3.874817, -0.992870],
    [2.341493, 2.341494],
    [-1.608150, 0.329733],
    [1.812381, -1.177565],
    [-2.459797, -1.064134],
    [-1.173152, -0.464637],
    [0.435542, 3.581350],
    [3.323808, 0.000000],
    [-0.000000, -3.000000],
    [-2.279022, 0.448518],
    [-3.581350, 0.435542],
    [-0.856149, 1.485258],
    [-3.450008, -0.853961],
    [0.657399, 3.126895],
    [-0.000000, -1.189423],
    [-0.741802, -0.299548],
    [-1.490051, 3.712108],
    [2.478525, -0.499539],
    [-1.941005, 2.661588],
    [-2.661588, -1.941005],
    [0.565685, 0.565685],
    [-2.956201, -0.204648],
    [-3.581350, -0.435542],
    [0.464637, 1.173152],
    [0.326775, 2.121559],
    [-3.000000, 0.000000],
    [3.969052, 0.496615],
    [1.552104, -2.926971],
    [-2.391732, -3.206185],
    [0.861631, -2.077605],
    [-3.216867, 1.235922],
    [-2.121559, -0.326775],
    [0.000000, 3.635053],
    [-1.177565, 1.812381],
    [2.840044, 0.366565],
    [-2.840044, -0.366565],
    [-0.346400, 3.272348],
    [1.177565, -1.812381],
    [3.216867, -1.235922],
    [-0.861631, 2.077605],
    [2.391732, 3.206185],
    [2.279022, 0.448518],
    [-0.435542, 3.581350],
    [0.850907, 0.850907],
    [-3.969052, -0.496615],
    [-1.485258, -0.856149],
    [3.000000, 0.000000],
    [-0.464637, -1.173152],
    [-0.326775, -2.121559],
    [-0.000000, -0.800000],
    [3.450008, -0.853961],
    [2.956201, 0.204648],
    [2.341494, -2.341493],
    [2.661588, 1.941005],
    [-1.235922, 3.216867],
    [1.490051, -3.712108],
    [0.741802, 0.299548],
    [0.000000, 1.598080],
    [1.519290, -1.519290],
    [0.000000, 4.000000],
    [3.635053, 0.000000],
    [-0.853961, -3.450008],
    [0.448518, -2.279022],
    [0.992870, -3.874817],
    [0.346400, 3.272348],
    [1.941005, 2.661588],
    [0.565685, -0.565685],
    [-0.464637, 1.173152],
    [-3.323808, 0.000000],
    [-2.173634, -1.476494],
    [-0.496615, 3.969052],
    [2.459797, 1.064134],
    [0.000000, 3.323808],
    [1.173152, 0.464637],
    [0.614252, -1.787357],
    [-0.614252, 1.787357],
    [-2.341493, -2.341494],
    [3.874817, 0.992870],
    [1.608150, -0.329733],
    [0.900200, -2.863161],
    [0.496615, -3.969052],
    [0.326775, -2.121559],
    [2.173634, 1.476494],
    [-0.900200, 2.863161],
    [0.366565, 2.840044],
    [-2.341493, 2.341494],
    [0.464637, -1.173152],
    [-0.000000, -3.323808],
    [0.366565, -2.840044],
    [0.172630, -2.030746],
    [-0.992870, 3.874817],
    [2.863161, -0.900200],
    [-2.341494, -2.341493],
    [-0.565685, 0.565685],
    [-1.941005, -2.661588],
    [0.853961, 3.450008],
    [-2.926971, 1.552104],
    [-1.064134, 2.459797],
    [-1.476494, -2.173634],
    [-0.346400, -3.272348],
    [-1.519290, -1.519290],
    [-1.168141, -1.168141],
    [3.874817, -0.992870],
    [1.787357, -0.614252],
    [-0.448518, 2.279022],
    [0.299548, -0.741802],
    [1.235922, -3.216867],
    [-3.635053, 0.000000],
    [-1.519290, 1.519290],
    [-2.341494, 2.341493],
    [-2.676036, 0.467987],
    [-2.391732, 3.206185],
    [1.173152, -0.464637],
    [0.467987, 2.676036],
    [0.172630, 2.030746],
    [-1.490051, -3.712108],
    [2.956201, -0.204648],
    [-3.450008, 0.853961],
    [3.712108, 1.490051],
    [1.177565, 1.812381],
    [0.992870, 3.874817],
    [0.000000, 0.800000],
    [1.485258, 0.856149],
    [-0.741802, 0.299548],
    [-0.329733, 1.608150],
    [-0.850907, -0.850907],
    [2.173634, -1.476494],
    [-0.614252, -1.787357],
    [-3.712108, 1.490051],
    [0.346400, -3.272348],
    [-1.894757, -1.894757],
    [2.030746, 0.172630],
    [-0.204648, 2.956201],
    [1.608150, 0.329733],
    [-1.552104, 2.926971],
    [1.476494, -2.173634],
    [2.840044, -0.366565],
    [-0.000000, -3.635053],
    [0.856149, -1.485258],
    [2.121559, 0.326775],
    [-3.216867, -1.235922],
    [3.126895, -0.657399],
    [1.552104, 2.926971],
    [-3.126895, -0.657399],
    [-2.077605, -0.861631],
    [0.204648, 2.956201],
    [1.941005, -2.661588],
    [-2.478525, 0.499539],
    [-1.812381, 1.177565],
    [0.000000, 1.189423],
    [2.828427, -2.828427],
    [-0.657399, -3.126895],
    [1.519290, 1.519290],
    [0.329733, 1.608150],
    [3.450008, 0.853961],
    [-1.787357, -0.614252],
    [2.926971, -1.552104],
    [3.581350, -0.435542],
    [1.235922, 3.216867],
    [-0.861631, -2.077605],
    [-2.676036, -0.467987],
    [0.000000, 3.000000],
    [2.000000, 0.000000],
    [-3.500968, 1.934741],
    [-0.435542, -3.581350],
    [0.467987, -2.676036],
    [0.299548, 0.741802],
    [-1.934741, 3.500968],
    [-0.856149, -1.485258],
    [-3.500968, -1.934741],
    [-1.934741, -3.500968],
    [0.496615, 3.969052],
])
Elements = np.array([
    [116, 93, 282, 266],
    [93, 244, 219, 282],
    [266, 282, 53, 71],
    [282, 219, 137, 53],
    [137, 208, 237, 53],
    [53, 237, 103, 71],
    [103, 29, 47, 71],
    [47, 243, 110, 71],
    [243, 276, 78, 110],
    [78, 114, 189, 110],
    [114, 31, 164, 189],
    [110, 189, 266, 71],
    [189, 164, 116, 266],
    [208, 137, 188, 269],
    [137, 219, 130, 188],
    [219, 244, 295, 130],
    [269, 188, 196, 285],
    [188, 130, 156, 196],
    [130, 295, 183, 156],
    [167, 107, 105, 147],
    [107, 276, 243, 105],
    [243, 47, 82, 105],
    [47, 29, 148, 82],
    [105, 82, 277, 147],
    [82, 148, 238, 277],
    [238, 81, 181, 277],
    [277, 181, 273, 147],
    [273, 245, 290, 147],
    [290, 144, 167, 147],
    [36, 242, 69, 35],
    [242, 201, 278, 69],
    [35, 69, 30, 195],
    [69, 278, 65, 30],
    [65, 158, 48, 30],
    [30, 48, 108, 195],
    [108, 165, 66, 195],
    [66, 11, 206, 195],
    [11, 276, 78, 206],
    [78, 125, 177, 206],
    [125, 31, 55, 177],
    [206, 177, 35, 195],
    [177, 55, 36, 35],
    [158, 65, 4, 175],
    [65, 278, 37, 4],
    [278, 201, 150, 37],
    [175, 4, 131, 170],
    [4, 37, 184, 131],
    [37, 150, 2, 184],
    [167, 107, 246, 197],
    [107, 276, 11, 246],
    [11, 66, 57, 246],
    [66, 165, 263, 57],
    [246, 57, 257, 197],
    [57, 263, 255, 257],
    [255, 286, 26, 257],
    [257, 26, 73, 197],
    [73, 21, 180, 197],
    [180, 144, 167, 197],
    [287, 252, 154, 220],
    [252, 106, 241, 154],
    [220, 154, 115, 10],
    [154, 241, 68, 115],
    [68, 72, 234, 115],
    [115, 234, 271, 10],
    [271, 133, 12, 10],
    [12, 272, 85, 10],
    [272, 232, 3, 85],
    [3, 209, 5, 85],
    [209, 59, 46, 5],
    [85, 5, 220, 10],
    [5, 46, 287, 220],
    [72, 68, 100, 92],
    [68, 241, 134, 100],
    [241, 106, 60, 134],
    [92, 100, 192, 149],
    [100, 134, 231, 192],
    [134, 60, 117, 231],
    [54, 9, 101, 94],
    [9, 232, 272, 101],
    [272, 12, 109, 101],
    [12, 133, 61, 109],
    [101, 109, 126, 94],
    [109, 61, 22, 126],
    [22, 42, 91, 126],
    [126, 91, 118, 94],
    [118, 13, 247, 94],
    [247, 217, 54, 94],
    [291, 140, 178, 258],
    [140, 214, 8, 178],
    [258, 178, 207, 221],
    [178, 8, 41, 207],
    [41, 119, 43, 207],
    [207, 43, 104, 221],
    [104, 228, 163, 221],
    [163, 157, 74, 221],
    [157, 232, 3, 74],
    [3, 233, 142, 74],
    [233, 59, 235, 142],
    [74, 142, 258, 221],
    [142, 235, 291, 258],
    [119, 41, 160, 256],
    [41, 8, 166, 160],
    [8, 214, 194, 166],
    [256, 160, 196, 285],
    [160, 166, 156, 196],
    [166, 194, 183, 156],
    [54, 9, 135, 191],
    [9, 232, 157, 135],
    [157, 163, 199, 135],
    [163, 228, 123, 199],
    [135, 199, 248, 191],
    [199, 123, 120, 248],
    [120, 81, 181, 248],
    [248, 181, 273, 191],
    [273, 245, 111, 191],
    [111, 217, 54, 191],
    [294, 239, 49, 67],
    [239, 88, 185, 49],
    [67, 49, 62, 33],
    [49, 185, 275, 62],
    [275, 121, 95, 62],
    [62, 95, 18, 33],
    [18, 83, 283, 33],
    [283, 89, 222, 33],
    [89, 224, 254, 222],
    [254, 216, 218, 222],
    [216, 0, 152, 218],
    [222, 218, 67, 33],
    [218, 152, 294, 67],
    [121, 275, 223, 63],
    [275, 185, 288, 223],
    [185, 88, 34, 288],
    [63, 223, 211, 132],
    [223, 288, 261, 211],
    [288, 34, 7, 261],
    [249, 225, 292, 171],
    [225, 224, 89, 292],
    [89, 283, 251, 292],
    [283, 83, 172, 251],
    [292, 251, 58, 171],
    [251, 172, 96, 58],
    [96, 32, 14, 58],
    [58, 14, 138, 171],
    [138, 173, 39, 171],
    [39, 44, 249, 171],
    [293, 90, 264, 75],
    [90, 124, 136, 264],
    [75, 264, 1, 128],
    [264, 136, 267, 1],
    [267, 159, 284, 1],
    [1, 284, 6, 128],
    [6, 15, 268, 128],
    [268, 84, 193, 128],
    [84, 224, 254, 193],
    [254, 200, 143, 193],
    [200, 0, 38, 143],
    [193, 143, 75, 128],
    [143, 38, 293, 75],
    [159, 267, 112, 145],
    [267, 136, 146, 112],
    [136, 124, 168, 146],
    [145, 112, 192, 149],
    [112, 146, 231, 192],
    [146, 168, 117, 231],
    [249, 225, 169, 129],
    [225, 224, 84, 169],
    [84, 268, 279, 169],
    [268, 15, 155, 279],
    [169, 279, 76, 129],
    [279, 155, 77, 76],
    [77, 42, 91, 76],
    [76, 91, 118, 129],
    [118, 13, 139, 129],
    [139, 44, 249, 129],
    [45, 80, 162, 280],
    [80, 226, 174, 162],
    [280, 162, 215, 20],
    [162, 174, 265, 215],
    [265, 260, 97, 215],
    [215, 97, 141, 20],
    [141, 50, 23, 20],
    [23, 127, 250, 20],
    [127, 182, 28, 250],
    [28, 122, 27, 250],
    [122, 274, 51, 27],
    [250, 27, 280, 20],
    [27, 51, 45, 280],
    [260, 265, 98, 240],
    [265, 174, 281, 98],
    [174, 226, 52, 281],
    [240, 98, 131, 170],
    [98, 281, 184, 131],
    [281, 52, 2, 184],
    [64, 102, 19, 236],
    [102, 182, 127, 19],
    [127, 23, 227, 19],
    [23, 50, 56, 227],
    [19, 227, 202, 236],
    [227, 56, 16, 202],
    [16, 286, 26, 202],
    [202, 26, 73, 236],
    [73, 21, 86, 236],
    [86, 190, 64, 236],
    [40, 179, 230, 151],
    [179, 187, 24, 230],
    [151, 230, 203, 113],
    [230, 24, 70, 203],
    [70, 212, 289, 203],
    [203, 289, 17, 113],
    [17, 186, 153, 113],
    [153, 161, 259, 113],
    [161, 182, 28, 259],
    [28, 176, 270, 259],
    [176, 274, 99, 270],
    [259, 270, 151, 113],
    [270, 99, 40, 151],
    [212, 70, 253, 79],
    [70, 24, 25, 253],
    [24, 187, 204, 25],
    [79, 253, 211, 132],
    [253, 25, 261, 211],
    [25, 204, 7, 261],
    [64, 102, 262, 210],
    [102, 182, 161, 262],
    [161, 153, 198, 262],
    [153, 186, 205, 198],
    [262, 198, 87, 210],
    [198, 205, 213, 87],
    [213, 32, 14, 87],
    [87, 14, 138, 210],
    [138, 173, 229, 210],
    [229, 190, 64, 210],
])
out_mesh = meshio.Mesh(points=Nodes,
                       cells={"quad": Elements})
meshio.write("Mesh.xdmf", out_mesh)
3

In DOLFINx, we require that cells are connected by facet, i.e. a cell cannot be connected to another cell by vertex (to run properly in parallel). In your mesh, several nodes are connected only by vertices.
You cannot really see this looking at your mesh in Paraview, it looks connected nicely. However, if you carefully inspect your mesh, you will see that you have duplicate vertices at the same coordinates.

You can see that the first node here is at the same point as the last point. There are many such duplications in your mesh.

How did you create your mesh?

4

Thanks for the reply @dokken. Yes, I think this mesh is problematic. Since I would like to have a perfectly rotationally symmetric mesh, I generate one eighth of the domain with Abaqus, and then mirror and rotate it until I have a complete circle. Finally, I remove all the repeated nodes and rebuild the element connectivity. This method might be problematic if we have inputs nodes like [2.341494, 2.341493].

I would like to know if we have better ways to generate such perfectly rotationally symmetric mesh to use in FEniCSx?

5

As far as I’m aware gmsh supports symmetry option and supports removing duplicate nodes: