How to implement MapReduce in C # using PLINQ?

Question

How to implement MapReduce in C # using PLINQ?

Suppose you have 7-8 WebServices for collecting data and for each reception (async), you should put this data in some database tables, in my case it is SQL Server 2008. For example, the data that you receive from each web service:

<employees> <employee> <name>Ramiz</name> </employee> <employee> <name>Aamir</name> </employee> <employee> <name>Zubair</name> </employee> </employees> 

And each time a response is received, this data goes into the table name - Employee:

 Employee === EmployeeID (PK) EmployeeName 

Once the data gets into the table, it should return as json to the client, which is an ASP.NET application (MVC 3) that makes this call using client-side JavaScript (ajax).

Suppose WebServiceEmployee1 is back with data and the remaining 6 are in the queue (still trying to get the data). Then he must go through the registration of the result set in the table instead of waiting for another 6 and return the data of the inserted employee to the client in json. And, keep it connected and performing while others do the same.

Please see, in my toolbelt I have ASP.NET MVC 3 (Razor), SQL SERVER 2008 R2, jQuery.

Thanks.

Answer 1

Without a clear understanding of the processing you want to perform, I suggest reading the following MSDN documentation: http://bit.ly/Ir7Nvk It describes a map / abbreviation with PLINQ with examples, for example:

 public IDMultisetItemList PotentialFriendsPLinq(SubscriberID id, int maxCandidates) { var candidates = subscribers[id].Friends.AsParallel() .SelectMany(friend => subscribers[friend].Friends) .Where(foaf => foaf != id && !(subscribers[id].Friends.Contains(foaf))) .GroupBy(foaf => foaf) .Select(foafGroup => new IDMultisetItem(foafGroup.Key, foafGroup.Count())); return Multiset.MostNumerous(candidates, maxCandidates); } 

The "map", which is Friends.AsParallel , SelectMany and Where , and the "decrease" phase is GroupBy and Select

Answer 2

In this PDF from MS there is an example of "Reduce the map" to the middle / end

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CH0QFjAE&url=http%3A%2F%2Fdownload.microsoft.com%2Fdownload%2F3%2F4%2FD%2F34D13993 2132-4E04-AE48-53D3150057BD% 2FPatterns_of_Parallel_Programming_CSharp.pdf & ei = 4f_VT-ScD-Lg2gWqoeSWDw & usg = AFQjCNGhk_BZL8-5n8DaS_kMTaWRU9Y1Zw & sig2 = ddKl4KuOGUiUb1pIawWeNQ

 public static IEnumerable<TResult> MapReduce<TSource, TMapped, TKey, TResult>( this IEnumerable<TSource> source, Patterns of Parallel Programming Page 76 Func<TSource, IEnumerable<TMapped>> map, Func<TMapped, TKey> keySelector, Func<IGrouping<TKey, TMapped>, IEnumerable<TResult>> reduce) { return source.SelectMany(map) .GroupBy(keySelector) .SelectMany(reduce); }