How to run a method in parallel using Julia?

Question

I read Julia's Parallel Computing docs and never did any parallel coding, all I had left was to want a softer introduction. So, I thought about the (possibly) simple problem that I could not figure out how to correctly code Julia's paradigm.

Say I have a / dataframe df matrix from some experiment. Its rows N are variables, and columns M are patterns. I have a pwCorr(..) method that calculates pair correlation of strings. If I needed the NxN matrix of all pair correlations, I would probably run a for loop that would iterate over N*N/2 (the top or bottom triangle of the matrix) and fill in the values; however, this seems like an ideal task for parallelization, since each of pwCorr() calls is independent of the others. (Am I reasoning correctly so that what can be parallelized and what cannot?)

To do this, it seems to me that I need to create a DArray that is populated with a @parallel for loop. And if so, I do not know how this can be achieved in Julia. If this is not the right approach, I think I don’t even know where to start.

Answer 1

This should work, first you need to distribute the top-level variable (data) to all employees:

  for pid in workers() remotecall(pid, x->(global data; data=x; nothing), data) end 

then do the calculation in chunks using the DArray constructor with some weird indexing:

 corrs = DArray((20,20)) do I out=zeros(length(I[1]),length(I[2])) for i=I[1], j=I[2] if i<j out[i-minimum(I[1])+1,j-minimum(I[2])+1]= 0.0 else out[i-minimum(I[1])+1,j-minimum(I[2])+1] = cor(vec(data[i,:]), vec(data[j,:])) end end out end 

In more detail, the DArray constructor takes a function that takes a set of indices and returns a piece of the resulting matrix that matches these index ranges. In the above code, I is a set of ranges with I[1] , which is the first range. You can see this more clearly with:

 julia> DArray((10,10)) do I println(I) return zeros(length(I[1]),length(I[2])) end From worker 2: (1:10,1:5) From worker 3: (1:10,6:10) 

where you can see that he split the array into two parts of the second axis.

The most difficult part of the example was moving from these “global” index ranges to local index ranges by subtracting from the minimum element and then adding 1 back to index on Julia based on 1. Hope this helps!