What is the most efficient way to report on a large dataset in memory in .NET?

Question

I am working on a project that sends monthly invoices to thousands (if not millions) of customers. A lot of data needs to be manipulated, with interest and not yet dynamically calculated on the fly through the internal API.

Now I have a process that runs in pieces of 2500 instances at a time and writes an XML file containing all the data when each range ends. Then I use Crystal Reports to report this data, which takes several minutes. I am looking for the best way to report this data and you need some advice.

Is Crystal just slow when parsing XML? I tried to throw Crystal into a dataset in memory, but it seems that after going through more than 500 records, the XML route is actually more efficient. Should I write data to our SQL database and report it this way using index optimization? Or am I missing an alternative approach that will be much faster?

Answer 1

Personally, I found that Crystal Reports is a memory hang, it runs for a long time even for small data sets. We are also developing in VS 2008 for users using Win2K, so there was a version conflict because the CR version that comes with VS 2008 only works on XP or higher.

I have completed the conversion of all our reports to in-memory objects serialized in XML , which are translated via XSL to HTML / CSS and can be printed using WebBrowser Management. Now our reports are loaded as soon as XML is created, instead of waiting for Crystal to appear, which reduced the viewing time from ~ 30 seconds to a second and reduced the MSI size from ~ 27 MB to 1.5 MB.

Answer 2

In the past, I came across the same thing as CR. When I hit this issue, I switched the new development to XML style sheets in HTML. It ran faster, but took longer.

Answer 3

In my experience, Crystal Reports is very slow in very large reporting projects.

I found that some other reporting solutions tend to cope with large reports, especially if they are generated from a dataset in memory. I would recommend trying the Telerik reporting tool for this.

Answer 4

Have you thought about multithreading reports? Or use multiple servers / services to handle subsets of report generation.

We use Crystal Reports XI and find it very slow. In addition, with multi-threaded streams from a single service, Crystal reports have a limit on the number of stream reports you can create. We run our reports (less than your bugs, say, about 30,000 reports) using PdfSharp and stored procedures with Sql Server 2005. And this is a very time-consuming process.

Therefore, we consider several services / servers as an opportunity to generate subsets.

EDIT

as after a thought, I put the Crystal Question in front of X -)