MPI Get / Collect Dynamic Vector Length

Question

I have an application that stores vector structs. These structures contain information about each GPU in a system such as memory and giga-flop / s. Each system uses a different number of GPUs.

I have a program that runs on several computers at once, and I need to collect this data. I am very new to MPI, but I can use MPI_Gather() for the most part, however I would like to know how to collect / receive these vectors with dynamic size.

 class MachineData { unsigned long hostMemory; long cpuCores; int cudaDevices; public: std::vector<NviInfo> nviVec; std::vector<AmdInfo> amdVec; ... }; struct AmdInfo { int platformID; int deviceID; cl_device_id device; long gpuMem; float sgflops; double dgflops; }; 

Each machine in the cluster fills its own instance of MachineData . I want to collect each of these instances, but I'm not sure how to approach the collection of nviVec and amdVec , since their length depends on each machine.

Answer 1

You can use MPI_GATHERV in combination with MPI_GATHER to accomplish this. MPI_GATHERV is a variable version of MPI_GATHER and allows the root rank to collect a different number of elements from each sending process. But in order for the root rank to determine these numbers, he must know how many elements each rank occupies. This can be achieved with a simple single MPI_GATHER element. Something like that:

 // To keep things simple: root is fixed to be rank 0 and MPI_COMM_WORLD is used // Number of MPI processes and current rank int size, rank; MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); int *counts = new int[size]; int nelements = (int)vector.size(); // Each process tells the root how many elements it holds MPI_Gather(&nelements, 1, MPI_INT, counts, 1, MPI_INT, 0, MPI_COMM_WORLD); // Displacements in the receive buffer for MPI_GATHERV int *disps = new int[size]; // Displacement for the first chunk of data - 0 for (int i = 0; i < size; i++) disps[i] = (i > 0) ? (disps[i-1] + counts[i-1]) : 0; // Place to hold the gathered data // Allocate at root only type *alldata = NULL; if (rank == 0) // disps[size-1]+counts[size-1] == total number of elements alldata = new int[disps[size-1]+counts[size-1]]; // Collect everything into the root MPI_Gatherv(vectordata, nelements, datatype, alldata, counts, disps, datatype, 0, MPI_COMM_WORLD); 

You must also register the data type created by MPI ( datatype in the code above) for structures (binary send will work, but will not be portable and will not work in heterogeneous settings).