Understanding the sizes, coordinates and ordering of processes in a 2D MPI grid

Question

I have 3 questions related to MPI (in C). I think the first 2 have the same answer, but I'm not sure.

---

Question 1

When using MPI_Dims_create to create a 2D mesh, is the dimension of the returned dimensions equal to X and which is equal to Y?

For example:

 int numProcs = -1, myProcID = -1; MPI_Comm_rank(MPI_COMM_WORLD, &myProcID); MPI_Comm_size(MPI_COMM_WORLD, &numProcs); int size[2]; size[0] = size[1] = 0; MPI_Dims_create(numProcs, 2, size); int numProcs_y = -1, numProcs_x = 1; 

Should it be:

 numProcs_y = size[0]; numProcs_x = size[1]; 

Or that:

 numProcs_x = size[0]; numProcs_y = size[1]; 

I tried to run this with a number of processes that seemed to give me an answer (e.g. 6). Using 6 processes, MPI_Dims_create should either create a grid with three rows and two columns, or two rows and three columns (if I misunderstood the documentation). For process 3 (out of 6), I found that size[0] = 1 and size[1] = 0 (I believe this corresponds to x = 0, y = 1). This apparently tells me that MPI_Dims_create creates a grid with 3 rows and 2 columns (because it was 2 rows and 3 columns, process 2 (out of 6) should have x = 2, y = 0). Any confirmation that anyone can provide on this subject would be greatly appreciated.

---

Question 2

When using MPI_Cart_coords on a two-dimensional Cartesian grid, from the returned dimensions X and which is Y?

For example:

 int periodic[2]; periodic[0] = periodic[1] = 0; // no wrap around MPI_Comm cart_comm; int coords[2]; // using size from question 1 MPI_Cart_create(MPI_COMM_WORLD, 2, size, periodic, 1, &cart_comm); MPI_Cart_coords(cart_comm, myProcID, 2, coords); 

As in question 1, my question is should it be like this

 myProcID_y = coords[0]; myProcID_x = coords[1]; 

or how is it

 myProcID_x = coords[0]; myProcID_y = coords[1]; 

I looked through the documentation and previous questions here, but I can not find a direct answer to this question.

The documentation seems to indicate that the first of the two approaches is correct, but it does not conclusively indicate it.

---

Question 3

The main question underlying the first two questions is that I am trying to split a 2D grid into rows and columns. However, when I use MPI_Comm_rank to check the row and column identifiers of each process after that, I get the ordered processes in an order that does not match what, in my opinion, answers the above questions.

Based on the above, I expect the processes to be ordered as follows:

 P0 P1 P2 P3 P4 P5 

However, using this code (which appears after the above code in my program, so all of the above codes are available from it: I'm trying to allocate my code to simplify the selection of my questions):

 MPI_Comm row_comm, col_comm; int row_id = -1, col_id = -1; // ** NOTE: My use of myProcID_y and myProcID_x here are based // on my understanding of the previous 2 questions ... if my // understanding to one/both of those is wrong, then obviously the assignments // here are wrong too. // create row and column communicators based on my location in grid MPI_Comm_split(cart_comm, myProcID_y, myProcID_x, &row_comm); MPI_Comm_split(cart_comm, myProcID_x, myProcID_y, &col_comm); // get row and column ID for each process MPI_Comm_rank(row_comm, &row_id); MPI_Comm_rank(col_comm, &col_id); printf("Process: %d\trowID: %d\tcolID: %d\n", myProcID, row_id, col_id); 

I see the following printed:

 Process: 0 rowID: 0 colID: 0 Process: 1 rowID: 1 colID: 0 Process: 2 rowID: 0 colID: 1 Process: 3 rowID: 1 colID: 1 Process: 4 rowID: 0 colID: 2 Process: 5 rowID: 1 colID: 2 

which, apparently, corresponds to the following order of processes:

 P0 P2 P4 P1 P3 P5 

which represents the opposite dimensions of the matrix (2 rows, 3 columns) from what I expected from MPI_Dims_create .

Assuming that my understanding of the first two questions is correct (i.e., that Y is the first dimension returned by these functions), why are the processes (apparently) ordered in a different order at this stage?

Answer 1

Q1 and Q2. There are no dimensions in MPI such as the X dimension and the Y dimension — these are the labels you give to the abstract dimensions used in the standard. MPI works with numbered sizes and follows the row number of C rank rows, i.e. In a 2x3 map of the Cartesian topology, (0,0) maps to rank 0 , (0,1) maps to rank 1 , (0,2) maps to rank 2 , (1,0) maps to rank 3 , etc.

Note that the dimension 0 corresponds to the rightmost element in the coordinate court. This is the opposite of the numbering of elements in C arrays and is often a source of confusion. To create a 2x3 Cartesian topology, the size array must be initialized as:

 int size[2] = { 3, 2 }; 

It is up to you to map abstract numbered dimensions to your problem. You can choose size 0 as X, or you can choose size 1 - it does not matter.

Regarding MPI_DIMS_CREATE , the standard says:

Dimensions are set as close to each other as possible using the appropriate divisibility algorithm.

For dims[i] specified by the call, dims[i] will be ordered in an unordered order.

This operation simply returns an array of dims[i] elements that have the following properties (if the size of one or more dimensions is not set by setting non-zero value (s) to dims[] before calling MPI_DIMS_CREATE )

• dims[0] >= dims[1] >= dims[2] >= ...
• dims[0] * dims[1] * dims[2] == nprocs , where nprocs is the number of processes specified in MPI_DIMS_CREATE .

This means that since MPI_DIMS_CREATE decomposes the MPI_DIMS_CREATE set into a multidimensional grid, it assigns the largest multiplier to a dimension of size 0 , next to size 1 and soon. As 6 factors, such as 2*3 , then MPI_DIMS_CREATE will return { 3, 2 } . If you call MPI_CART_CREATE directly with the result from MPI_DIMS_CREATE , it will create a 2x3 topology with coordinates and ranks:

 (0,0)=0 (0,1)=1 (0,2)=2 (1,0)=3 (1,1)=4 (1,2)=5 

Q3. MPI provides a special procedure for splitting Cartesian topologies - MPI_CART_SUB . It takes an array of logical flags (integers in C) called remain_dims in the standard. Each nonzero remain_dims[i] means that this dimension i should be preserved in the resulting partition, while separate subcommunicators will be created for any possible combination of non-holding measurements. For example, given the topology of 2x3 :

• remain_dims[] = { 1, 0 } save the dimension 0 and lead to 2 non-overlapping 1st communicators with 3 processes each;
• remain_dims[] = { 0, 1 } save dimension 1 will result in 3 non-overlapping 1st communicators with two processes each;
• remain_dims[] = { 0, 0 } will not save either of the two dimensions and will result in 6 non-overlapping zero-dimensional communicators with one process in each.

Which partition you call on the row, and which one you would call column break, is up to you and your labeling of the Cartesian dimensions.

---

It should be noted that MPI codes are often used for systems that consist of multi-core SMP or NUMA nodes. In this case, a suitable 2D mesh would be nodes x cores . If the number of cores is known, then it can be easily fixed by calling MPI_DIMS_CREATE :

 int size[2] = { ncores, 0 }; MPI_Dims_create(numProcs, 2, size); 

This is more convenient than dividing numProcs by ncores and checking for divisibility, since MPI_DIMS_CREATE signals an error if ncores does not divide numProcs . Then a partition preserving dimension 0 , i.e. One with

 int remain_dims[2] = { 1, 0 }; 

will create subcommunicators that contain processes on the same node, and

 int remain_dims[2] = { 0, 1 }; 

will create subcommunicators that do not contain two processes from the same node.

---

Please note that in your code you specified the value 1 (true) in the reorder parameter in MPI_CART_CREATE . This can lead to processes having different ranks in MPI_COMM_WORLD and in the Cartesian communicator. Therefore, there is no guarantee that the following lines of code will do what you expect from them:

 MPI_Cart_coords(cart_comm, myProcID, 2, coords); ^^^^^^^^ ... printf("Process: %d\trowID: %d\tcolID: %d\n", myProcID, row_id, col_id); ^^^^^^^^ 

myProcID was obtained from MPI_COMM_WORLD and may actually differ from the new rank in cart_comm , so it should not be used to obtain process coordinates and perform splits.