Problems with OpenMPI MPI_Barrier

Question

I have some synchronization problems using the OpenMPI implementation of MPI_Barrier:

int rank; int nprocs; int rc = MPI_Init(&argc, &argv); if(rc != MPI_SUCCESS) { fprintf(stderr, "Unable to set up MPI"); MPI_Abort(MPI_COMM_WORLD, rc); } MPI_Comm_size(MPI_COMM_WORLD, &nprocs); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf("P%d\n", rank); fflush(stdout); MPI_Barrier(MPI_COMM_WORLD); printf("P%d again\n", rank); MPI_Finalize(); 

for mpirun -n 2./a.out

the output should be: P0 P1 ...

conclusion sometimes: P0 P0 again P1 P1 again

what's happening?

Answer 1

The order in which print lines are displayed on your terminal is not necessarily the order in which things are printed. A shared resource ( stdout ) is used for this, so there should always be a problem with ordering. (And fflush doesn't help here, stdout always buffered.)

You can try the prefix of your output with a timestamp and save it all in different files, one per MPI process.

Then, to check your log, you can combine the two files and sort them according to the timestamp.

Then your problem should go away.

Answer 2

There is nothing wrong with MPI_Barrier ().

Like Jens mentioned , the reason you don't see the expected result is because stdout is buffered for each process. There is no guarantee that fingerprints from several processes will be displayed in the calling process in order. (If stdout from each process is transferred to the main process for printing in real time, this will lead to a lot of unnecessary communication!)

If you want to convince yourself that the barrier works, you can try writing a file. The presence of several processes writing to one file can lead to additional complications, so you can write each record to a single file, and then after the barrier exchange the files to which they write. For example:

  Proc-0 Proc-1 | | f0.write(..) f1.write(...) | | x ~~ barrier ~~ x | | f1.write(..) f0.write(...) | | END END 

Implementation Example:

 #include "mpi.h" #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { char filename[20]; int rank, size; FILE *fp; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); if (rank < 2) { /* proc 0 and 1 only */ sprintf(filename, "file_%d.out", rank); fp = fopen(filename, "w"); fprintf(fp, "P%d: before Barrier\n", rank); fclose(fp); } MPI_Barrier(MPI_COMM_WORLD); if (rank < 2) { /* proc 0 and 1 only */ sprintf(filename, "file_%d.out", (rank==0)?1:0 ); fp = fopen(filename, "a"); fprintf(fp, "P%d: after Barrier\n", rank); fclose(fp); } MPI_Finalize(); return 0; } 

After running the code, you should get the following results:

 [ me@home ]$ cat file_0.out P0: before Barrier P1: after Barrier [ me@home ]$ cat file_1.out P1: before Barrier P0: after Barrier 

For all files, post-barrier statements will always be displayed later.

Answer 3

In MPI programs, the output order is not guaranteed.

This is not related to MPI_Barrier at all.

Also, I would not spend too much time worrying about the output order using MPI programs.

The most elegant way to achieve this, if you really want to, is to let processes send their messages in one rank, say, to rank 0, and let rank 0 output the result in the order in which it was received or ordered takes.

Again, do not spend too much time trying to order an exit from MPI programs. It is not practical and unsuitable.