Best way to write data tables from a web service in XML?

Question

I am trying to reorganize some slow code that writes XML using nested loops from multiple data. I read that using linq to write xml will be faster. I am not very versed in linq, so I was hoping to get help here.

Some things I should mention are that the current architecture uses a web service that returns data to us in dataTables. Then we go through the datatables (iteratively), and there are several that lead to several nested loops.

Example:

dt1 = Webservice.getStuff(); for each (datarow r1 in dt1.Rows) { dt2 = Webservice.getMoreStuff(r1[col1], r1[col2]); // write out some xml for each (datarow r2 in dt2.Rows) { dt3 = Webservice.getEvenMoreStuff(r2[col1], r2[col2]); // write out more xml for each (datarow r3 in dt3.Rows) { // write out more xml } } } 

As you can see from obvious reasons, this is terribly slow. Is there a way to speed this up with linq? What do you guys suggest as a more effective approach to refactoring? I apologize if the details are unclear ...

I appreciate any help anyone could offer.

Answer 1

Writing XML is not something that slows you down in this case . This speed should be negligible compared to the time spent on all web service calls.

Instead of focusing on writing XML, I would try to figure out how to compress the number of web service calls so that you can immediately get all your data, rather than insert similar nested calls.

Answer 2

I am afraid that you cannot heal your need. Since I'm sure this method slows down, this is not how you write xml, but how you get the data. If there was any improvement in writing xml, it would not be in noticeable proportion. I suggest you reconsider the way you get data. Try to minimize the number of WebService calls.

Answer 3

It looks like you need to profile your application first - perhaps a free trial version of ANTS or similar will work for you.

Answer 4

If you want to use Linq2Xml, you will also need Linq2Datasets.

This way you can change your nested loops to a Linq query and select in XML.

I expect that using XmlWriter may be faster, although it will certainly use less memory. But writing will be a little more.

Answer 5

Using linq for xml and linq for datasets, you can create your own xml as follows:

  static void Main(string[] args) { DataTable t = getStuff("test"); var xml = new XElement("Main", from row in t.AsEnumerable() select new XElement("firstlevel", new XAttribute("a", row["a"]), new XAttribute("b", row["b"]), from row2 in getStuff(row["a"].ToString()).AsEnumerable() select new XElement("secondlevel", new XAttribute("a", row2["a"]), new XAttribute("b", row2["b"]), from row3 in getStuff(row2["a"].ToString()).AsEnumerable() select new XElement("thirdlevel", new XElement("a", row3["a"]), new XElement("b", row3["b"]))))); Console.WriteLine(xml.ToString()); } private static DataTable getStuff(string s) { Random r=new Random(s.GetHashCode()); DataTable t = new DataTable(); t.Columns.Add("a"); t.Columns.Add("b"); for (int i = 0; i < 2; i++) { t.Rows.Add (r.Next().ToString(), r.Next().ToString()); } return t; } 

Answer 6

You will need to change your mind a bit to get rid of all the calls caused by the WS loop. This is a serious performance hit.

Answer 7

You do not say how you write XML or where you write XML.

If you write a line. Stop it and write a stream.

If you are writing a buffer string, look at the size of the buffer. If you write on an ASP.NET page or handler, then call Response.Flush () at regular intervals.

There is balance, since writing to the buffer is almost always faster than writing to the stream. However, writing to the auto-resize buffer will be slower the larger the image size. What else, no matter what XML is processing, cannot take action until it starts to receive some, which does not happen until the first reset.

Thus, there may be room for improvement, although calls to a web service are likely to outweigh what can be achieved. Perhaps this can also be improved if you rewrite the parsing of the response to the web service so that it yields as it is parsed, which means that you can start processing the response before receiving the entire response.