Comparing Two Large Shared Lists

Question

I can not find a concrete example of this, so I am posting a question. Any help was appreciated.

I have two large general lists that contain more than 300 thousand items.

I look at the first list to discard information and generate a new item for the new list on the fly, but I need to search the second list and return a value based on three matching criteria, if found add it to the list, however, as you can imagine it 300k * 300k times takes time.

Is there a way to do this more efficiently?

My code is:

var reportList = new List<StocksHeldInCustody>(); foreach (var correctDepotHolding in correctDepotHoldings) { var reportLine = new StocksHeldInCustody(); reportLine.ClientNo = correctDepotHolding.ClientNo; reportLine.Value = correctDepotHolding.ValueOfStock; reportLine.Depot = correctDepotHolding.Depot; reportLine.SEDOL = correctDepotHolding.StockCode; reportLine.Units = correctDepotHolding.QuantityHeld; reportLine.Custodian = "Unknown"; reportLine.StockName = correctDepotHolding.StockR1.Trim() + " " + correctDepotHolding.StockR2.Trim(); //Get custodian info foreach (var ccHolding in ccHoldList) { if (correctDepotHolding.ClientNo != ccHolding.ClientNo) continue; if (correctDepotHolding.Depot != ccHolding.Depot) continue; if (correctDepotHolding.StockCode != ccHolding.StockCode) continue; if (correctDepotHolding.QuantityHeld != ccHolding.QuantityHeld) continue; reportLine.Custodian = ccHolding.Custodian; break; } reportList.Add(reportLine); } 

Answer 1

As Pranay says, unification is probably what you need:

 var query = from correct in correctDepotHoldings join ccHolding in ccHoldList on new { correct.ClientNo, correct.Depot, correct.StockCode, correct.QuantityHeld } equals new { ccHolding.ClientNo, ccHolding.Depot, ccHolding.StockCode, ccHolding.QuantityHeld } // TODO: Fill in the properties here based on correct and ccHolding select new StocksHeldInCustody { ... }; var reportList = query.ToList(); 

Answer 2

You can transfer data from the search list to the dictionary, and the key will be a unique hash of the 3 items you are looking for. Then you will get a very quick search and save millions of iterations.

Answer 3

Check out my full post: Linq Join Mutiple columns using anonymous type

Use the Linq inner join that will work for you.

 var list = ( from x in entity join y in entity2 on new { x.field1, x.field2 } equals new { y.field1, y.field2 } select new entity { fields to select}).ToList(); 

Joining linq to multiple fields

 EmployeeDataContext edb= new EmployeeDataContext(); var cust = from c in edb.Customers join d in edb.Distributors on new { CityID = c.CityId, StateID = c.StateId, CountryID = c.CountryId, Id = c.DistributorId } equals new { CityID = d.CityId, StateID = d.StateId, CountryID = d.CountryId, Id = d.DistributorId } select c; 

Answer 4

Use LINQ to join the lists and return it as you like.

eg,

 var list1 = GetMassiveList(); var list2 = GetMassiveList(); var list3 = from a in list1 join b in list2 on new { a.Prop1, a.Prop2 } equals new { b.Prop1, b.Prop2 } select new { a.Prop1, b.Prop2 }; 

To connect to an external word, you can use DefaultIfEmpty () In this example, you set your right side of the connection to the default object (often empty) for cases when the connection has not been made.

eg,

 from a in list1 join b in list2 on new { a.Prop1, a.Prop2 } equals new { b.Prop1, b.Prop2 } into outer from b in outer.DefaultIfEmpty() select new Prop1 = a.Prop1, Prop2 = b != null ? b.Prop2 : "Value for Prop2 if the b join is null" }